Abstract image of a woman sitting on books and studying on a laptop

Cost-Based Optimizer in Apache Spark 2.2

Databricks, profile picture
Databricks

The document discusses Apache Spark's new cost-based optimizer (CBO) in version 2.2. It describes how the CBO works in two key steps: 1. It collects and propagates statistics about tables and columns to estimate the cardinality of operations like filters, joins and aggregates. 2. It calculates the estimated cost of different execution plans and selects the most optimal plan based on minimizing the estimated cost. This allows it to pick more efficient join orders and join algorithms. The document provides examples of how the CBO improves queries on TPC-DS benchmarks by producing smaller intermediate results and faster execution times compared to the previous rule-based optimizer in Spark 2.1.

Software
1 of 55
Downloaded 451 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
Ron Hu, Zhenhua Wang Huawei Technologies, Inc. Sameer Agarwal, Wenchen Fan Databricks Inc. Cost-Based Optimizer in Apache Spark 2.2
Session 1 Topics • Motivation • Statistics Collection Framework • Cost Based Optimizations • TPC-DS Benchmark and Query Analysis • Demo 2
How Spark Executes a Query? Logical Plan Physical Plan Catalog Optimizer RDDs … SQL Code Generator Data Frames
How Spark Executes a Query? Logical Plan Physical Plan Catalog Optimizer RDDs … SQL Code Generator Data Frames Focus of Today’s Talk
Catalyst Optimizer: An Overview 5 events = sc.read.json(“/logs”) stats = events.join(users) .groupBy(“loc”,“status”) .avg(“duration”) errors = stats.where( stats.status == “ERR”) Query Plan is an internal representation of a user’s program Series of Transformations that convert the initial query plan into an optimized plan SCAN logs JOIN FILTER AGG SCAN users SCAN logsSCAN users JOIN FILTER AGG SCAN users
Catalyst Optimizer: An Overview 6 In Spark, the optimizer’s goal is to minimize end-to-end query response time. Two key ideas: - Prune unnecessary data as early as possible - e.g., filter pushdown, column pruning - Minimize per-operator cost - e.g., broadcast vs shuffle, optimal join order SCAN logsSCAN users JOIN FILTER AGG SCAN users
Rule-based Optimizer in Spark 2.1 • Most of Spark SQL optimizer’s rules are heuristics rules. – PushDownPredicate, ColumnPruning, ConstantFolding,… • Does NOT consider the cost of each operator • Does NOT consider selectivity when estimating join relation size • Therefore: – Join order is mostly decided by its position in the SQL queries – Physical Join implementation is decided based on heuristics 7
An Example (TPC-DS q11 variant) 8 SCAN: store_sales SCAN: customer SCAN: date_dim FILTER JOIN JOIN SELECT customer_id FROM customer, store_sales, date_dim WHERE c_customer_sk = ss_customer_sk AND ss_sold_date_sk = d_date_sk AND c_customer_sk > 1000
An Example (TPC-DS q11 variant) 9 SCAN: store_sales SCAN: customer SCAN: date_dim FILTER JOIN JOIN 3 billion 12 million 2.5 billion 10 million 500 million 0.1 million
An Example (TPC-DS q11 variant) 10 SCAN: store_sales SCAN: customer SCAN: date_dim FILTERJOIN JOIN 3 billion 12 million 2.5 billion 500 million 10 million 500 million 0.1 million 40% faster 80% less data
An Example (TPC-DS q11 variant) 11 SCAN: store_sales SCAN: customer SCAN: date_dim FILTERJOIN JOIN 3 billion 12 million 2.5 billion 500 million 10 million 500 million 0.1 million How do we automatically optimize queries like these?
Cost Based Optimizer (CBO) • Collect, infer and propagate table/column statistics on source/intermediate data • Calculate the cost for each operator in terms of number of output rows, size of output, etc. • Based on the cost calculation, pick the most optimal query execution plan 12
Rest of the Talk • Statistics Collection Framework – Table/Column Level Statistics Collected – Cardinality Estimation (Filters, Joins, Aggregates etc.) • Cost-based Optimizations – Build Side Selection – Multi-way Join Re-ordering • TPC-DS Benchmarks • Demo 13
Statistics Collection Framework and Cost Based Optimizations Ron Hu Huawei Technologies
Step 1: Collect, infer and propagate table and column statistics on source and intermediate data
Table Statistics Collected • Command to collect statistics of a table. – Ex: ANALYZE TABLE table-name COMPUTE STATISTICS • It collects table level statistics and saves into metastore. – Number of rows – Table size in bytes 16
Column Statistics Collected • Command to collect column level statistics of individual columns. – Ex: ANALYZE TABLE table-name COMPUTE STATISTICS FOR COLUMNS column-name1, column-name2, …. • It collects column level statistics and saves into meta-store. String/Binary type ✓ Distinct count ✓ Null count ✓ Average length ✓ Max length Numeric/Date/Timestamp type ✓ Distinct count ✓ Max ✓ Min ✓ Null count ✓ Average length (fixed length) ✓ Max length (fixed length) 17
Filter Cardinality Estimation • Between Logical expressions: AND, OR, NOT • In each logical expression: =, <, <=, >, >=, in, etc • Current support type in Expression – For <, <=, >, >=, <=>: Integer, Double, Date, Timestamp, etc – For = , <=>: String, Integer, Double, Date, Timestamp, etc. • Example: A <= B – Based on A, B’s min/max/distinct count/null count values, decide the relationships between A and B. After completing this expression, we set the new min/max/distinct count/null count – Assume all the data is evenly distributed if no histogram information. 18
Filter Operator Example • Column A (op) literal B – (op) can be “=“, “<”, “<=”, “>”, “>=”, “like” – Like the styles as “l_orderkey = 3”, “l_shipdate <= “1995-03-21” – Column’s max/min/distinct count/null countshould be updated – Example: Column A < value B Column AB B A.min A.max Filtering Factor = 0% need to changeA’s statistics Filtering Factor = 100% no need to changeA’s statistics Without histograms, supposedatais evenly distributed Filtering Factor = (B.value – A.min) / (A.max – A.min) A.min = no change A.max = B.value A.ndv = A.ndv * Filtering Factor 19
Filter Operator Example • Column A (op) Column B – (op) can be “<”, “<=”, “>”, “>=” – We cannot suppose the data is evenly distributed,so the empirical filtering factor is set to 1/3 – Example: Column A < Column B B A AA A B B B selectivity = 100% selectivity = 0% selectivity = 33.3% 20 selectivity = 33.3%
Join Cardinality Estimation • Inner-Join: The number of rows of “A join B on A.k1 = B.k1” is estimated as: num(A B) = num(A) * num(B) / max(distinct(A.k1), distinct(B.k1)), – where num(A) is the number of records in table A, distinct is the number of distinct values of that column. – The underlying assumption for this formula is that each value of the smaller domain is included in the larger domain. • We similarly estimate cardinalities for Left-Outer Join, Right-Outer Join and Full-Outer Join 21
Other Operator Estimation • Project: does not change row count • Aggregate: consider uniqueness of group-by columns • Limit, Sample, etc. 22
Step 2: Cost Estimation and Optimal Plan Selection
Build Side Selection • For two-way hash joins, we need to choose one operand as build side and the other as probe side. • Choose lower-cost child as build side of hash join. – Before: build side was selected based on original table sizes. ➔ BuildRight – Now with CBO: build side is selected based on estimated cost of various operators before join. ➔ BuildLeft Join Scan t2Filter Scan t15 billion records, 500 GB t1.value= 200 1 million records, 100 MB 100 million records, 20 GB 24
Hash Join Implementation: Broadcast vs. Shuffle Physical Plan ➢ SortMergeJoinExec/ BroadcastHashJoinExec/ ShuffledHashJoinExec ➢ CartesianProductExec/ BroadcastNestedLoopJoinExec Logical Plan ➢ Equi-join • Inner Join • LeftSemi/LeftAnti Join • LeftOuter/RightOuter Join ➢ Theta-join • Broadcast Criterion: whether the join side’s output size is small (default 10MB). Join Scan t2Filter Scan t15 billion records, 500 GB t1.value = 100 Only 1000 records, 100 KB 100 million records, 20 GB Join Scan t2Aggregate … Join Scan t2Join … … 25
Multi-way Join Reorder • Reorder the joins using a dynamic programming algorithm. 1. First we put all items (basic joined nodes) into level 0. 2. Build all two-way joins at level 1 from plans at level 0 (single items). 3. Build all 3-way joins from plans at previous levels (two-way joins and single items). 4. Build all 4-way joins etc, until we build all n-way joins and pick the best plan among them. • When building m-way joins, only keep the best plan (optimal sub-solution) for the same set of m items. – E.g., for 3-way joins of items {A, B, C}, we keep only the best plan among: (A J B) J C, (A J C) J B and (B J C) J A 26
Multi-way Join Reorder 27 Selinger et al. Access Path Selection in a Relational Database Management System. In SIGMOD 1979
Join Cost Formula • The cost of a plan is the sum of costs of all intermediate tables. • Cost = weight * Costcpu + CostIO * (1 - weight) – In Spark, we use weight * cardinality + size * (1 – weight) – weight is a tuning parameter configured via spark.sql.cbo.joinReorder.card.weight (0.7 as default) 28
TPC-DS Benchmarks and Query Analysis Zhenhua Wang Huawei Technologies
Session 2 Topics • Motivation • Statistics Collection Framework • Cost Based Optimizations • TPC-DS Benchmark and Query Analysis • Demo 30
Preliminary Performance Test • Setup: − TPC-DS size at 1 TB (scale factor 1000) − 4 node cluster (Huawei FusionServer RH2288: 40 cores, 384GB mem) − Apache Spark 2.2 RC (dated 5/12/2017) • Statistics collection – A total of 24 tables and 425 columns ➢ Take 14 minutes to collect statistics for all tables and all columns. – Fast because all statistics are computed by integrating with Spark’s built-in aggregate functions. – Should take much less time if we collect statistics for columns used in predicate, join, and group-by only. 31
TPC-DS Query Q11 32 WITH year_total AS ( SELECT c_customer_id customer_id, c_first_name customer_first_name, c_last_name customer_last_name, c_preferred_cust_flag customer_preferred_cust_flag, c_birth_country customer_birth_country, c_login customer_login, c_email_address customer_email_address, d_year dyear, sum(ss_ext_list_price - ss_ext_discount_amt) year_total, 's' sale_type FROM customer, store_sales, date_dim WHERE c_customer_sk = ss_customer_sk AND ss_sold_date_sk = d_date_sk GROUP BY c_customer_id, c_first_name, c_last_name, d_year , c_preferred_cust_flag, c_birth_country, c_login, c_email_address, d_year UNION ALL SELECT c_customer_id customer_id, c_first_name customer_first_name, c_last_name customer_last_name, c_preferred_cust_flag customer_preferred_cust_flag, c_birth_country customer_birth_country, c_login customer_login, c_email_address customer_email_address, d_year dyear, sum(ws_ext_list_price - ws_ext_discount_amt) year_total, 'w' sale_type FROM customer, web_sales, date_dim WHERE c_customer_sk = ws_bill_customer_sk AND ws_sold_date_sk = d_date_sk GROUP BY c_customer_id, c_first_name, c_last_name, c_preferred_cust_flag, c_birth_country, c_login, c_email_address, d_year) SELECT t_s_secyear.customer_preferred_cust_flag FROM year_total t_s_firstyear , year_total t_s_secyear , year_total t_w_firstyear , year_total t_w_secyear WHERE t_s_secyear.customer_id = t_s_firstyear.customer_id AND t_s_firstyear.customer_id = t_w_secyear.customer_id AND t_s_firstyear.customer_id = t_w_firstyear.customer_id AND t_s_firstyear.sale_type = 's' AND t_w_firstyear.sale_type = 'w' AND t_s_secyear.sale_type = 's' AND t_w_secyear.sale_type = 'w' AND t_s_firstyear.dyear = 2001 AND t_s_secyear.dyear = 2001 + 1 AND t_w_firstyear.dyear = 2001 AND t_w_secyear.dyear = 2001 + 1 AND t_s_firstyear.year_total > 0 AND t_w_firstyear.year_total > 0 AND CASE WHEN t_w_firstyear.year_total > 0 THEN t_w_secyear.year_total / t_w_firstyear.year_total ELSE NULL END > CASE WHEN t_s_firstyear.year_total > 0 THEN t_s_secyear.year_total / t_s_firstyear.year_total ELSE NULL END ORDER BY t_s_secyear.customer_preferred_cust_flag LIMIT 100
Query Analysis – Q11 CBO OFF Large join result Join #1 store_sales customer date_dim 2.9 billion … … Join #2 web_sales customer date_dim Join #4 … Join #3 12 million 2.7 billion 73,049 73,049 12 million720 million 534 million 719 million 144 million 33
Query Analysis – Q11 CBO ON Small join result Join #1 store_sales date_dim customer 2.9 billion … … Join #2 web_sales date_dim customer Join #4 … Join #3 73,049 534 million 12 million 12 million 73,049720 million 534 million 144 million 144 million 1.4x Speedup 34 80% less
TPC-DS Query Q72 35 SELECT i_item_desc, w_warehouse_name, d1.d_week_seq, count(CASE WHEN p_promo_sk IS NULL THEN 1 ELSE 0 END) no_promo, count(CASE WHEN p_promo_sk IS NOT NULL THEN 1 ELSE 0 END) promo, count(*) total_cnt FROM catalog_sales JOIN inventory ON (cs_item_sk = inv_item_sk) JOIN warehouse ON (w_warehouse_sk = inv_warehouse_sk) JOIN item ON (i_item_sk = cs_item_sk) JOIN customer_demographics ON (cs_bill_cdemo_sk = cd_demo_sk) JOIN household_demographics ON (cs_bill_hdemo_sk = hd_demo_sk) JOIN date_dim d1 ON (cs_sold_date_sk = d1.d_date_sk) JOIN date_dim d2 ON (inv_date_sk = d2.d_date_sk) JOIN date_dim d3 ON (cs_ship_date_sk = d3.d_date_sk) LEFT OUTER JOIN promotion ON (cs_promo_sk = p_promo_sk) LEFT OUTER JOIN catalog_returns ON (cr_item_sk = cs_item_sk AND cr_order_number = cs_order_number) WHERE d1.d_week_seq = d2.d_week_seq AND inv_quantity_on_hand < cs_quantity AND d3.d_date > (cast(d1.d_date AS DATE) + interval 5 days) AND hd_buy_potential = '>10000' AND d1.d_year = 1999 AND hd_buy_potential = '>10000' AND cd_marital_status = 'D' AND d1.d_year = 1999 GROUP BY i_item_desc, w_warehouse_name, d1.d_week_seq ORDER BY total_cnt DESC, i_item_desc, w_warehouse_name, d_week_seq LIMIT 100
Query Analysis – Q72 CBO OFF Join #1 Join #2 Join #3 Join #4 Join #5 Join #6 Join #7 Join #8 catalog_sales inventory warehouse item customer_demographics date_dim d1 date_dim d2 date_dim d3 household_demographics 1.4 billion 783 million 223 billion 20 300,000223 billion 223 billion 44.7 million 7.5 million 1.6 million 9 million 1.9 million 7,200 73,049 73,049 73,049 8.7 million Really large intermediate results 36
Query Analysis – Q72 CBO ON Join #1 Join #2 Join #3 Join #4 Join #8 Join #5 Join #6 Join #7 catalog_sales inventory warehouseitem customer_demographics date_dim d1 date_dim d2date_dim d3 household_demographics 1.4 billion 783 million 20300,000 1.9 million 7,200 73,049 73,049 73,049 238 million 238 million 47.6 million 47.6 million 2,555 1 billion 1 billion 8.7 million Much smaller intermediate results ! 37 2-3 orders of magnitude less! 8.1x Speedup
TPC-DS Query Performance 38 0 350 700 1050 1400 1750 q1 q5 q9 q13 q16 q20 q23b q26 q30 q34 q38 q41 q45 q49 q53 q57 q61 q65 q70 q74 q78 q82 q86 q90 q94 q98 Runtime(seconds) without CBO with CBO
TPC-DS Query Speedup • TPC-DS query speedup ratio with CBO versus without CBO • 16 queries show speedup > 30% • The max speedup is 8X. • The geo-mean of speedup is 2.2X. 39
TPC-DS Query 64 40 WITH cs_ui AS (SELECT cs_item_sk, sum(cs_ext_list_price) AS sale, sum(cr_refunded_cash + cr_reversed_charge + cr_store_credit) AS refund FROM catalog_sales, catalog_returns WHERE cs_item_sk = cr_item_sk AND cs_order_number = cr_order_number GROUP BY cs_item_sk HAVING sum(cs_ext_list_price) > 2 * sum(cr_refunded_cash + cr_reversed_charge + cr_store_credit)), cross_sales AS (SELECT i_product_name product_name, i_item_sk item_sk, s_store_name store_name, s_zip store_zip, ad1.ca_street_number b_street_number, ad1.ca_street_name b_streen_name, ad1.ca_city b_city, ad1.ca_zip b_zip, ad2.ca_street_number c_street_number, ad2.ca_street_name c_street_name, ad2.ca_city c_city, ad2.ca_zip c_zip, d1.d_year AS syear, d2.d_year AS fsyear, d3.d_year s2year, count(*) cnt, sum(ss_wholesale_cost) s1, sum(ss_list_price) s2, sum(ss_coupon_amt) s3 FROM store_sales, store_returns, cs_ui, date_dim d1, date_dim d2, date_dim d3, store, customer, customer_demographics cd1, customer_demographics cd2, promotion, household_demographics hd1, household_demographics hd2, customer_address ad1, customer_address ad2, income_band ib1, income_band ib2, item WHERE ss_store_sk = s_store_sk AND ss_sold_date_sk = d1.d_date_sk AND ss_customer_sk = c_customer_sk AND ss_cdemo_sk = cd1.cd_demo_sk AND ss_hdemo_sk = hd1.hd_demo_sk AND ss_addr_sk = ad1.ca_address_sk AND ss_item_sk = i_item_sk AND ss_item_sk = sr_item_sk AND ss_ticket_number = sr_ticket_number AND ss_item_sk = cs_ui.cs_item_sk AND c_current_cdemo_sk = cd2.cd_demo_sk AND c_current_hdemo_sk = hd2.hd_demo_sk AND c_current_addr_sk = ad2.ca_address_sk AND c_first_sales_date_sk = d2.d_date_sk AND c_first_shipto_date_sk = d3.d_date_sk AND ss_promo_sk = p_promo_sk AND hd1.hd_income_band_sk = ib1.ib_income_band_sk AND hd2.hd_income_band_sk = ib2.ib_income_band_sk AND cd1.cd_marital_status <> cd2.cd_marital_status AND i_color IN ('purple', 'burlywood', 'indian', 'spring', 'floral', 'medium') AND i_current_price BETWEEN 64 AND 64 + 10 AND i_current_price BETWEEN 64 + 1 AND 64 + 15 GROUP BY i_product_name, i_item_sk, s_store_name, s_zip, ad1.ca_street_number, ad1.ca_street_name, ad1.ca_city, ad1.ca_zip, ad2.ca_street_number, ad2.ca_street_name, ad2.ca_city, ad2.ca_zip, d1.d_year, d2.d_year, d3.d_year) SELECT cs1.product_name, cs1.store_name, cs1.store_zip, cs1.b_street_number, cs1.b_streen_name, cs1.b_city, cs1.b_zip, cs1.c_street_number, cs1.c_street_name, cs1.c_city, cs1.c_zip, cs1.syear, cs1.cnt, cs1.s1, cs1.s2, cs1.s3, cs2.s1, cs2.s2, cs2.s3, cs2.syear, cs2.cnt FROM cross_sales cs1, cross_sales cs2 WHERE cs1.item_sk = cs2.item_sk AND cs1.syear = 1999 AND cs2.syear = 1999 + 1 AND cs2.cnt <= cs1.cnt AND cs1.store_name = cs2.store_name AND cs1.store_zip = cs2.store_zip ORDER BY cs1.product_name, cs1.store_name, cs2.cnt
Query Analysis – Q64 CBO ON 41 10% slower FileScan (store_sales) Exchange ExchangeSort Aggregate SortMergeJoin BroadcastHashJoin FileScan (store_returns) Exchange Sort BroadcastExchange (cs_ui) ReusedExchange Sort Sort BroadcastHashJoin ReusedExchange ReusedExchange Sort SortMergeJoin Sort Fragment 1 Fragment 2
Query Analysis – Q64 CBO OFF 42 FileScan (store_sales) Exchange Exchange Sort Aggregate SortMergeJoin SortMergeJoin FileScan (store_returns) Exchange Sort Sort (cs_ui) SortMergeJoin Sort ReusedExchange Aggregate Sort (cs_ui) ReusedExchange Sort Exchange Fragment 1 Fragment 2
CBO Demo Wenchen Fan Databricks
Current Status, Credits and Future Work Ron Hu Huawei Technologies
Available in Apache Spark 2.2 • Configured via spark.sql.cbo.enabled • ‘Off By Default’. Why? – Spark is used in production – Many Spark users may already rely on “human intelligence” to write queries in best order – Plan on enabling this by default in Spark 2.3 • We encourage you test CBO with Spark 2.2! 45
Current Status • SPARK-16026 is the umbrella jira. – 32 sub-tasks have been resolved – A big project spanning 8 months – 10+ Spark contributors involved – 7000+ lines of Scala code have been contributed • Good framework to allow integrations – Use statistics to derive if a join attribute is unique – Benefit star schema detection and its integration into join reorder 46
Birth of Spark SQL CBO • Prototype – In 2015, Ron Hu, Fang Cao, etc. of Huawei’s research department prototyped the CBO concept on Spark 1.2. – After a successful prototype, we shared technology with Zhenhua Wang, Fei Wang, etc of Huawei’s product development team. • We delivered a talk at Spark Summit 2016: – “Enhancing Spark SQL Optimizer with Reliable Statistics”. • The talk was well received by the community. – https://issues.apache.org/jira/browse/SPARK-16026 47
Collaboration • Good community support – Developers: Zhenhua Wang, Ron Hu, Reynold Xin, Wenchen Fan, Xiao Li – Reviewers: Wenchen, Herman, Reynold, Xiao, Liang-chi, Ioana, Nattavut, Hyukjin, Shuai, ….. – Extensive discussion in JIRAs and PRs (tens to hundreds conversations). – All the comments made the development time longer, but improved code quality. • It was a pleasure working with community. 48
Future Work: Cost Based Optimizer • Current cost formula is coarse. Cost = cardinality * weight + size * (1 - weight) • Cannot tell the cost difference between sort- merge join and hash join – spark.sql.join.preferSortMergeJoin defaults to true. • Underestimates (or ignores) shuffle cost. • Will improve cost formula in next release. 49
Future Work: Statistics Collection Framework • Advanced statistics: e.g. histograms, sketches. • Hint mechanism. • Partition level statistics. • Speed up statistics collection by sampling data for large tables. 50
Conclusion • Motivation • Statistics Collection Framework – Table/Column Level Statistics Collected – Cardinality Estimation (Filters, Joins, Aggregates etc.) • Cost-based Optimizations – Build Side Selection – Multi-way Join Re-ordering • TPC-DS Benchmarks • Demo 51
Thank You. ron.hu@huawei.com wangzhenhua@huawei.com sameer@databricks.com wenchen@databricks.com Sameer’s Office Hours @ 4:30pm Today Wenchen’s Office Hours @ 3pm Tomorrow
Multi-way Join Reorder – Example • Given A J B J C J D with join conditions A.k1 = B.k1 and B.k2 = C.k2 and C.k3 = D.k3 level 0: p({A}), p({B}), p({C}), p({D}) level 1: p({A, B}), p({A, C}), p({A, D}), p({B, C}), p({B, D}), p({C, D}) level 2: p({A, B, C}), p({A, B, D}), p({A, C, D}), p({B, C, D}) level 3: p({A, B, C, D}) -- final output plan 53
Multi-way Join Reorder – Example • Pruning strategy: exclude cartesian product candidates. This significantly reduces the search space. level 0: p({A}), p({B}), p({C}), p({D}) level 1: p({A, B}), p({A, C}), p({A, D}), p({B, C}), p({B, D}), p({C, D}) level 2: p({A, B, C}), p({A, B, D}), p({A, C, D}), p({B, C, D}) level 3: p({A, B, C, D}) -- final output plan 54
New Commands in Apache Spark 2.2 • CBO commands – Collect table-level statistics • ANALYZE TABLE table_name COMPUTE STATISTICS – Collect column-level statistics • ANALYZE TABLE table-name COMPUTE STATISTICS FOR COLUMNS column_name1, column_name2, … – Display statistics in the optimized logical plan > EXPLAIN COST > SELECT cc_call_center_sk, cc_call_center_id FROM call_center; … == Optimized Logical Plan == Project [cc_call_center_sk#75, cc_call_center_id#76], Statistics(sizeInBytes=1680.0 B, rowCount=42, hints=none) +- Relation[…31 fields] parquet, Statistics(sizeInBytes=22.5 KB, rowCount=42, hints=none) … 55

More Related Content

PDF
Common Strategies for Improving Performance on Your Delta Lakehouse
Databricks
 
PDF
Understanding Query Plans and Spark UIs
Databricks
 
PDF
Dynamic Partition Pruning in Apache Spark
Databricks
 
PPTX
Tuning and Debugging in Apache Spark
Patrick Wendell
 
PDF
Improving SparkSQL Performance by 30%: How We Optimize Parquet Pushdown and P...
Databricks
 
PDF
Don’t optimize my queries, optimize my data!
Julian Hyde
 
PDF
Spark shuffle introduction
colorant
 
PDF
Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...
Databricks
 
Common Strategies for Improving Performance on Your Delta Lakehouse
Databricks
 
Understanding Query Plans and Spark UIs
Databricks
 
Dynamic Partition Pruning in Apache Spark
Databricks
 
Tuning and Debugging in Apache Spark
Patrick Wendell
 
Improving SparkSQL Performance by 30%: How We Optimize Parquet Pushdown and P...
Databricks
 
Don’t optimize my queries, optimize my data!
Julian Hyde
 
Spark shuffle introduction
colorant
 
Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...
Databricks
 

What's hot (20)

PDF
Apache Spark Core—Deep Dive—Proper Optimization
Databricks
 
PDF
Designing Structured Streaming Pipelines—How to Architect Things Right
Databricks
 
PPTX
Optimizing Apache Spark SQL Joins
Databricks
 
PDF
Parquet performance tuning: the missing guide
Ryan Blue
 
PDF
Apache Calcite Tutorial - BOSS 21
Stamatis Zampetakis
 
PDF
Cost-Based Optimizer Framework for Spark SQL: Spark Summit East talk by Ron H...
Spark Summit
 
PDF
SparkSQL: A Compiler from Queries to RDDs
Databricks
 
PDF
Spark tuning
GMO-Z.com Vietnam Lab Center
 
PPTX
Delta lake and the delta architecture
Adam Doyle
 
PDF
Deep Dive into the New Features of Apache Spark 3.0
Databricks
 
PDF
Making Apache Spark Better with Delta Lake
Databricks
 
PDF
Apache Calcite: A Foundational Framework for Optimized Query Processing Over ...
Julian Hyde
 
PDF
A Deep Dive into Query Execution Engine of Spark SQL
Databricks
 
PPTX
A Deep Dive into Spark SQL's Catalyst Optimizer with Yin Huai
Databricks
 
PDF
Data Storage Tips for Optimal Spark Performance-(Vida Ha, Databricks)
Spark Summit
 
PDF
Spark SQL Deep Dive @ Melbourne Spark Meetup
Databricks
 
PPTX
How to Actually Tune Your Spark Jobs So They Work
Ilya Ganelin
 
PDF
The Parquet Format and Performance Optimization Opportunities
Databricks
 
PDF
Spark SQL Catalyst Code Optimization using Function Outlining with Kavana Bha...
Databricks
 
PDF
Getting Started with Databricks SQL Analytics
Databricks
 
Apache Spark Core—Deep Dive—Proper Optimization
Databricks
 
Designing Structured Streaming Pipelines—How to Architect Things Right
Databricks
 
Optimizing Apache Spark SQL Joins
Databricks
 
Parquet performance tuning: the missing guide
Ryan Blue
 
Apache Calcite Tutorial - BOSS 21
Stamatis Zampetakis
 
Cost-Based Optimizer Framework for Spark SQL: Spark Summit East talk by Ron H...
Spark Summit
 
SparkSQL: A Compiler from Queries to RDDs
Databricks
 
Spark tuning
GMO-Z.com Vietnam Lab Center
 
Delta lake and the delta architecture
Adam Doyle
 
Deep Dive into the New Features of Apache Spark 3.0
Databricks
 
Making Apache Spark Better with Delta Lake
Databricks
 
Apache Calcite: A Foundational Framework for Optimized Query Processing Over ...
Julian Hyde
 
A Deep Dive into Query Execution Engine of Spark SQL
Databricks
 
A Deep Dive into Spark SQL's Catalyst Optimizer with Yin Huai
Databricks
 
Data Storage Tips for Optimal Spark Performance-(Vida Ha, Databricks)
Spark Summit
 
Spark SQL Deep Dive @ Melbourne Spark Meetup
Databricks
 
How to Actually Tune Your Spark Jobs So They Work
Ilya Ganelin
 
The Parquet Format and Performance Optimization Opportunities
Databricks
 
Spark SQL Catalyst Code Optimization using Function Outlining with Kavana Bha...
Databricks
 
Getting Started with Databricks SQL Analytics
Databricks
 
Ad

Similar to Cost-Based Optimizer in Apache Spark 2.2 (20)

PDF
Enhancing Spark SQL Optimizer with Reliable Statistics
Jen Aman
 
PDF
unit 3 DBMS.docx.pdf geometric transformer in query processing
FallenAngel35
 
PDF
unit 3 DBMS.docx.pdf geometry in query p
FallenAngel35
 
DOCX
DBMS ArchitectureQuery ExecutorBuffer ManagerStora
LinaCovington707
 
PPTX
Query optimization
Pooja Dixit
 
PPTX
Cost-Based-Query-Optimization-in-DBMS.pptx
maddishiva1989
 
PPTX
MySQL Optimizer Overview
Olav Sandstå
 
PDF
Issues in Query Processing and Optimization
Editor IJMTER
 
PDF
MySQL Optimizer Cost Model
Olav Sandstå
 
PPTX
Processes in Query Optimization in (ABMS) Advanced Database Management Systems
gamemaker762
 
PDF
Beyond EXPLAIN: Query Optimization From Theory To Code
Yuto Hayamizu
 
PDF
01Query Processing and Optimization-SUM25.pdf
sfsmj710f
 
PDF
What’s New in the Upcoming Apache Spark 3.0
Databricks
 
PPTX
Cost-based Query Optimization in Hive
DataWorks Summit
 
PPTX
Cost-based query optimization in Apache Hive
Julian Hyde
 
PDF
Implementation of query optimization for reducing run time
Alexander Decker
 
PPTX
Lazy beats Smart and Fast
Julian Hyde
 
PDF
dd presentation.pdf
AnSHiKa187943
 
PPT
probabilistic ranking
FELIX75
 
PPTX
DB
Samchu Li
 
Enhancing Spark SQL Optimizer with Reliable Statistics
Jen Aman
 
unit 3 DBMS.docx.pdf geometric transformer in query processing
FallenAngel35
 
unit 3 DBMS.docx.pdf geometry in query p
FallenAngel35
 
DBMS ArchitectureQuery ExecutorBuffer ManagerStora
LinaCovington707
 
Query optimization
Pooja Dixit
 
Cost-Based-Query-Optimization-in-DBMS.pptx
maddishiva1989
 
MySQL Optimizer Overview
Olav Sandstå
 
Issues in Query Processing and Optimization
Editor IJMTER
 
MySQL Optimizer Cost Model
Olav Sandstå
 
Processes in Query Optimization in (ABMS) Advanced Database Management Systems
gamemaker762
 
Beyond EXPLAIN: Query Optimization From Theory To Code
Yuto Hayamizu
 
01Query Processing and Optimization-SUM25.pdf
sfsmj710f
 
What’s New in the Upcoming Apache Spark 3.0
Databricks
 
Cost-based Query Optimization in Hive
DataWorks Summit
 
Cost-based query optimization in Apache Hive
Julian Hyde
 
Implementation of query optimization for reducing run time
Alexander Decker
 
Lazy beats Smart and Fast
Julian Hyde
 
dd presentation.pdf
AnSHiKa187943
 
probabilistic ranking
FELIX75
 
DB
Samchu Li
 
Ad

More from Databricks (20)

PPTX
DW Migration Webinar-March 2022.pptx
Databricks
 
PPTX
Data Lakehouse Symposium | Day 1 | Part 1
Databricks
 
PPT
Data Lakehouse Symposium | Day 1 | Part 2
Databricks
 
PPTX
Data Lakehouse Symposium | Day 2
Databricks
 
PPTX
Data Lakehouse Symposium | Day 4
Databricks
 
PDF
5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop
Databricks
 
PDF
Democratizing Data Quality Through a Centralized Platform
Databricks
 
PDF
Learn to Use Databricks for Data Science
Databricks
 
PDF
Why APM Is Not the Same As ML Monitoring
Databricks
 
PDF
The Function, the Context, and the Data—Enabling ML Ops at Stitch Fix
Databricks
 
PDF
Stage Level Scheduling Improving Big Data and AI Integration
Databricks
 
PDF
Simplify Data Conversion from Spark to TensorFlow and PyTorch
Databricks
 
PDF
Scaling your Data Pipelines with Apache Spark on Kubernetes
Databricks
 
PDF
Scaling and Unifying SciKit Learn and Apache Spark Pipelines
Databricks
 
PDF
Sawtooth Windows for Feature Aggregations
Databricks
 
PDF
Redis + Apache Spark = Swiss Army Knife Meets Kitchen Sink
Databricks
 
PDF
Re-imagine Data Monitoring with whylogs and Spark
Databricks
 
PDF
Raven: End-to-end Optimization of ML Prediction Queries
Databricks
 
PDF
Processing Large Datasets for ADAS Applications using Apache Spark
Databricks
 
PDF
Massive Data Processing in Adobe Using Delta Lake
Databricks
 
DW Migration Webinar-March 2022.pptx
Databricks
 
Data Lakehouse Symposium | Day 1 | Part 1
Databricks
 
Data Lakehouse Symposium | Day 1 | Part 2
Databricks
 
Data Lakehouse Symposium | Day 2
Databricks
 
Data Lakehouse Symposium | Day 4
Databricks
 
5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop
Databricks
 
Democratizing Data Quality Through a Centralized Platform
Databricks
 
Learn to Use Databricks for Data Science
Databricks
 
Why APM Is Not the Same As ML Monitoring
Databricks
 
The Function, the Context, and the Data—Enabling ML Ops at Stitch Fix
Databricks
 
Stage Level Scheduling Improving Big Data and AI Integration
Databricks
 
Simplify Data Conversion from Spark to TensorFlow and PyTorch
Databricks
 
Scaling your Data Pipelines with Apache Spark on Kubernetes
Databricks
 
Scaling and Unifying SciKit Learn and Apache Spark Pipelines
Databricks
 
Sawtooth Windows for Feature Aggregations
Databricks
 
Redis + Apache Spark = Swiss Army Knife Meets Kitchen Sink
Databricks
 
Re-imagine Data Monitoring with whylogs and Spark
Databricks
 
Raven: End-to-end Optimization of ML Prediction Queries
Databricks
 
Processing Large Datasets for ADAS Applications using Apache Spark
Databricks
 
Massive Data Processing in Adobe Using Delta Lake
Databricks
 

Recently uploaded (20)

PPTX
UNIT II: Software design, software .pptx
SriSamyukthaSethuram
 
PPTX
Chapter_05_System Modeling for software engineering
anant5150ca6
 
PPTX
Human-Computer Interaction for Lecture 1
ssuserca7fe9
 
PDF
Multiverse AI Review 2025_ The Ultimate All-in-One AI Platform.pdf
THEMESMO
 
PDF
infoteam HELLAS company profile 2025 presentation
alexispapadimitriou
 
PPTX
Presentation - Summer Internship at Samatrix.io_template_2.pptx
omchoksi99
 
PDF
SOFTWARE ENGINEERING Software Engineering (3rd Edition) by K.K. Aggarwal & Yo...
yaderyeka
 
PPTX
WJQSJXNAZJVCVSAXJHBZKSJXKJKXJSBHJBJEHHJB
soniatharv2010
 
PDF
Module 1 - Introduction to Generative AI.pdf
ElmerP1
 
PPTX
Streamlining Project Management in the AV Industry with D-Tools for Zoho CRM ...
thomaslopez7677
 
PPTX
HackYourBrain__UtrechtJUG__11092025.pptx
SimonedeGijt
 
PPTX
Human Computer Interaction lecture Chapter 2.pptx
ubaidzafar2
 
PPTX
MCP empowers AI Agents from Zero to Production
innovaterz
 
PPTX
Human-Computer Interaction for Lecture 2
ssuserca7fe9
 
PPTX
Why 2025 Is the Best Year to Hire Software Developers in India
Milan H
 
PDF
What Makes a Great Data Visualization Consulting Service.pdf
Dreamer Technoland
 
PPTX
AI Tools Revolutionizing Software Development Workflows
amngrg0511
 
PPTX
Chapter 1 - Transaction Processing and Mgt.pptx
LeulPrince
 
PDF
Odoo Construction Management System by CandidRoot
CandidRoot Solutions Private Limited
 
PDF
IT Consulting Services to Secure Future Growth
Shiv Technolabs Pvt. Ltd.
 
UNIT II: Software design, software .pptx
SriSamyukthaSethuram
 
Chapter_05_System Modeling for software engineering
anant5150ca6
 
Human-Computer Interaction for Lecture 1
ssuserca7fe9
 
Multiverse AI Review 2025_ The Ultimate All-in-One AI Platform.pdf
THEMESMO
 
infoteam HELLAS company profile 2025 presentation
alexispapadimitriou
 
Presentation - Summer Internship at Samatrix.io_template_2.pptx
omchoksi99
 
SOFTWARE ENGINEERING Software Engineering (3rd Edition) by K.K. Aggarwal & Yo...
yaderyeka
 
WJQSJXNAZJVCVSAXJHBZKSJXKJKXJSBHJBJEHHJB
soniatharv2010
 
Module 1 - Introduction to Generative AI.pdf
ElmerP1
 
Streamlining Project Management in the AV Industry with D-Tools for Zoho CRM ...
thomaslopez7677
 
HackYourBrain__UtrechtJUG__11092025.pptx
SimonedeGijt
 
Human Computer Interaction lecture Chapter 2.pptx
ubaidzafar2
 
MCP empowers AI Agents from Zero to Production
innovaterz
 
Human-Computer Interaction for Lecture 2
ssuserca7fe9
 
Why 2025 Is the Best Year to Hire Software Developers in India
Milan H
 
What Makes a Great Data Visualization Consulting Service.pdf
Dreamer Technoland
 
AI Tools Revolutionizing Software Development Workflows
amngrg0511
 
Chapter 1 - Transaction Processing and Mgt.pptx
LeulPrince
 
Odoo Construction Management System by CandidRoot
CandidRoot Solutions Private Limited
 
IT Consulting Services to Secure Future Growth
Shiv Technolabs Pvt. Ltd.
 

Cost-Based Optimizer in Apache Spark 2.2

  • 1. Ron Hu, Zhenhua Wang Huawei Technologies, Inc. Sameer Agarwal, Wenchen Fan Databricks Inc. Cost-Based Optimizer in Apache Spark 2.2
  • 2. Session 1 Topics • Motivation • Statistics Collection Framework • Cost Based Optimizations • TPC-DS Benchmark and Query Analysis • Demo 2
  • 3. How Spark Executes a Query? Logical Plan Physical Plan Catalog Optimizer RDDs … SQL Code Generator Data Frames
  • 4. How Spark Executes a Query? Logical Plan Physical Plan Catalog Optimizer RDDs … SQL Code Generator Data Frames Focus of Today’s Talk
  • 5. Catalyst Optimizer: An Overview 5 events = sc.read.json(“/logs”) stats = events.join(users) .groupBy(“loc”,“status”) .avg(“duration”) errors = stats.where( stats.status == “ERR”) Query Plan is an internal representation of a user’s program Series of Transformations that convert the initial query plan into an optimized plan SCAN logs JOIN FILTER AGG SCAN users SCAN logsSCAN users JOIN FILTER AGG SCAN users
  • 6. Catalyst Optimizer: An Overview 6 In Spark, the optimizer’s goal is to minimize end-to-end query response time. Two key ideas: - Prune unnecessary data as early as possible - e.g., filter pushdown, column pruning - Minimize per-operator cost - e.g., broadcast vs shuffle, optimal join order SCAN logsSCAN users JOIN FILTER AGG SCAN users
  • 7. Rule-based Optimizer in Spark 2.1 • Most of Spark SQL optimizer’s rules are heuristics rules. – PushDownPredicate, ColumnPruning, ConstantFolding,… • Does NOT consider the cost of each operator • Does NOT consider selectivity when estimating join relation size • Therefore: – Join order is mostly decided by its position in the SQL queries – Physical Join implementation is decided based on heuristics 7
  • 8. An Example (TPC-DS q11 variant) 8 SCAN: store_sales SCAN: customer SCAN: date_dim FILTER JOIN JOIN SELECT customer_id FROM customer, store_sales, date_dim WHERE c_customer_sk = ss_customer_sk AND ss_sold_date_sk = d_date_sk AND c_customer_sk > 1000
  • 9. An Example (TPC-DS q11 variant) 9 SCAN: store_sales SCAN: customer SCAN: date_dim FILTER JOIN JOIN 3 billion 12 million 2.5 billion 10 million 500 million 0.1 million
  • 10. An Example (TPC-DS q11 variant) 10 SCAN: store_sales SCAN: customer SCAN: date_dim FILTERJOIN JOIN 3 billion 12 million 2.5 billion 500 million 10 million 500 million 0.1 million 40% faster 80% less data
  • 11. An Example (TPC-DS q11 variant) 11 SCAN: store_sales SCAN: customer SCAN: date_dim FILTERJOIN JOIN 3 billion 12 million 2.5 billion 500 million 10 million 500 million 0.1 million How do we automatically optimize queries like these?
  • 12. Cost Based Optimizer (CBO) • Collect, infer and propagate table/column statistics on source/intermediate data • Calculate the cost for each operator in terms of number of output rows, size of output, etc. • Based on the cost calculation, pick the most optimal query execution plan 12
  • 13. Rest of the Talk • Statistics Collection Framework – Table/Column Level Statistics Collected – Cardinality Estimation (Filters, Joins, Aggregates etc.) • Cost-based Optimizations – Build Side Selection – Multi-way Join Re-ordering • TPC-DS Benchmarks • Demo 13
  • 14. Statistics Collection Framework and Cost Based Optimizations Ron Hu Huawei Technologies
  • 15. Step 1: Collect, infer and propagate table and column statistics on source and intermediate data
  • 16. Table Statistics Collected • Command to collect statistics of a table. – Ex: ANALYZE TABLE table-name COMPUTE STATISTICS • It collects table level statistics and saves into metastore. – Number of rows – Table size in bytes 16
  • 17. Column Statistics Collected • Command to collect column level statistics of individual columns. – Ex: ANALYZE TABLE table-name COMPUTE STATISTICS FOR COLUMNS column-name1, column-name2, …. • It collects column level statistics and saves into meta-store. String/Binary type ✓ Distinct count ✓ Null count ✓ Average length ✓ Max length Numeric/Date/Timestamp type ✓ Distinct count ✓ Max ✓ Min ✓ Null count ✓ Average length (fixed length) ✓ Max length (fixed length) 17
  • 18. Filter Cardinality Estimation • Between Logical expressions: AND, OR, NOT • In each logical expression: =, <, <=, >, >=, in, etc • Current support type in Expression – For <, <=, >, >=, <=>: Integer, Double, Date, Timestamp, etc – For = , <=>: String, Integer, Double, Date, Timestamp, etc. • Example: A <= B – Based on A, B’s min/max/distinct count/null count values, decide the relationships between A and B. After completing this expression, we set the new min/max/distinct count/null count – Assume all the data is evenly distributed if no histogram information. 18
  • 19. Filter Operator Example • Column A (op) literal B – (op) can be “=“, “<”, “<=”, “>”, “>=”, “like” – Like the styles as “l_orderkey = 3”, “l_shipdate <= “1995-03-21” – Column’s max/min/distinct count/null countshould be updated – Example: Column A < value B Column AB B A.min A.max Filtering Factor = 0% need to changeA’s statistics Filtering Factor = 100% no need to changeA’s statistics Without histograms, supposedatais evenly distributed Filtering Factor = (B.value – A.min) / (A.max – A.min) A.min = no change A.max = B.value A.ndv = A.ndv * Filtering Factor 19
  • 20. Filter Operator Example • Column A (op) Column B – (op) can be “<”, “<=”, “>”, “>=” – We cannot suppose the data is evenly distributed,so the empirical filtering factor is set to 1/3 – Example: Column A < Column B B A AA A B B B selectivity = 100% selectivity = 0% selectivity = 33.3% 20 selectivity = 33.3%
  • 21. Join Cardinality Estimation • Inner-Join: The number of rows of “A join B on A.k1 = B.k1” is estimated as: num(A B) = num(A) * num(B) / max(distinct(A.k1), distinct(B.k1)), – where num(A) is the number of records in table A, distinct is the number of distinct values of that column. – The underlying assumption for this formula is that each value of the smaller domain is included in the larger domain. • We similarly estimate cardinalities for Left-Outer Join, Right-Outer Join and Full-Outer Join 21
  • 22. Other Operator Estimation • Project: does not change row count • Aggregate: consider uniqueness of group-by columns • Limit, Sample, etc. 22
  • 23. Step 2: Cost Estimation and Optimal Plan Selection
  • 24. Build Side Selection • For two-way hash joins, we need to choose one operand as build side and the other as probe side. • Choose lower-cost child as build side of hash join. – Before: build side was selected based on original table sizes. ➔ BuildRight – Now with CBO: build side is selected based on estimated cost of various operators before join. ➔ BuildLeft Join Scan t2Filter Scan t15 billion records, 500 GB t1.value= 200 1 million records, 100 MB 100 million records, 20 GB 24
  • 25. Hash Join Implementation: Broadcast vs. Shuffle Physical Plan ➢ SortMergeJoinExec/ BroadcastHashJoinExec/ ShuffledHashJoinExec ➢ CartesianProductExec/ BroadcastNestedLoopJoinExec Logical Plan ➢ Equi-join • Inner Join • LeftSemi/LeftAnti Join • LeftOuter/RightOuter Join ➢ Theta-join • Broadcast Criterion: whether the join side’s output size is small (default 10MB). Join Scan t2Filter Scan t15 billion records, 500 GB t1.value = 100 Only 1000 records, 100 KB 100 million records, 20 GB Join Scan t2Aggregate … Join Scan t2Join … … 25
  • 26. Multi-way Join Reorder • Reorder the joins using a dynamic programming algorithm. 1. First we put all items (basic joined nodes) into level 0. 2. Build all two-way joins at level 1 from plans at level 0 (single items). 3. Build all 3-way joins from plans at previous levels (two-way joins and single items). 4. Build all 4-way joins etc, until we build all n-way joins and pick the best plan among them. • When building m-way joins, only keep the best plan (optimal sub-solution) for the same set of m items. – E.g., for 3-way joins of items {A, B, C}, we keep only the best plan among: (A J B) J C, (A J C) J B and (B J C) J A 26
  • 27. Multi-way Join Reorder 27 Selinger et al. Access Path Selection in a Relational Database Management System. In SIGMOD 1979
  • 28. Join Cost Formula • The cost of a plan is the sum of costs of all intermediate tables. • Cost = weight * Costcpu + CostIO * (1 - weight) – In Spark, we use weight * cardinality + size * (1 – weight) – weight is a tuning parameter configured via spark.sql.cbo.joinReorder.card.weight (0.7 as default) 28
  • 29. TPC-DS Benchmarks and Query Analysis Zhenhua Wang Huawei Technologies
  • 30. Session 2 Topics • Motivation • Statistics Collection Framework • Cost Based Optimizations • TPC-DS Benchmark and Query Analysis • Demo 30
  • 31. Preliminary Performance Test • Setup: − TPC-DS size at 1 TB (scale factor 1000) − 4 node cluster (Huawei FusionServer RH2288: 40 cores, 384GB mem) − Apache Spark 2.2 RC (dated 5/12/2017) • Statistics collection – A total of 24 tables and 425 columns ➢ Take 14 minutes to collect statistics for all tables and all columns. – Fast because all statistics are computed by integrating with Spark’s built-in aggregate functions. – Should take much less time if we collect statistics for columns used in predicate, join, and group-by only. 31
  • 32. TPC-DS Query Q11 32 WITH year_total AS ( SELECT c_customer_id customer_id, c_first_name customer_first_name, c_last_name customer_last_name, c_preferred_cust_flag customer_preferred_cust_flag, c_birth_country customer_birth_country, c_login customer_login, c_email_address customer_email_address, d_year dyear, sum(ss_ext_list_price - ss_ext_discount_amt) year_total, 's' sale_type FROM customer, store_sales, date_dim WHERE c_customer_sk = ss_customer_sk AND ss_sold_date_sk = d_date_sk GROUP BY c_customer_id, c_first_name, c_last_name, d_year , c_preferred_cust_flag, c_birth_country, c_login, c_email_address, d_year UNION ALL SELECT c_customer_id customer_id, c_first_name customer_first_name, c_last_name customer_last_name, c_preferred_cust_flag customer_preferred_cust_flag, c_birth_country customer_birth_country, c_login customer_login, c_email_address customer_email_address, d_year dyear, sum(ws_ext_list_price - ws_ext_discount_amt) year_total, 'w' sale_type FROM customer, web_sales, date_dim WHERE c_customer_sk = ws_bill_customer_sk AND ws_sold_date_sk = d_date_sk GROUP BY c_customer_id, c_first_name, c_last_name, c_preferred_cust_flag, c_birth_country, c_login, c_email_address, d_year) SELECT t_s_secyear.customer_preferred_cust_flag FROM year_total t_s_firstyear , year_total t_s_secyear , year_total t_w_firstyear , year_total t_w_secyear WHERE t_s_secyear.customer_id = t_s_firstyear.customer_id AND t_s_firstyear.customer_id = t_w_secyear.customer_id AND t_s_firstyear.customer_id = t_w_firstyear.customer_id AND t_s_firstyear.sale_type = 's' AND t_w_firstyear.sale_type = 'w' AND t_s_secyear.sale_type = 's' AND t_w_secyear.sale_type = 'w' AND t_s_firstyear.dyear = 2001 AND t_s_secyear.dyear = 2001 + 1 AND t_w_firstyear.dyear = 2001 AND t_w_secyear.dyear = 2001 + 1 AND t_s_firstyear.year_total > 0 AND t_w_firstyear.year_total > 0 AND CASE WHEN t_w_firstyear.year_total > 0 THEN t_w_secyear.year_total / t_w_firstyear.year_total ELSE NULL END > CASE WHEN t_s_firstyear.year_total > 0 THEN t_s_secyear.year_total / t_s_firstyear.year_total ELSE NULL END ORDER BY t_s_secyear.customer_preferred_cust_flag LIMIT 100
  • 33. Query Analysis – Q11 CBO OFF Large join result Join #1 store_sales customer date_dim 2.9 billion … … Join #2 web_sales customer date_dim Join #4 … Join #3 12 million 2.7 billion 73,049 73,049 12 million720 million 534 million 719 million 144 million 33
  • 34. Query Analysis – Q11 CBO ON Small join result Join #1 store_sales date_dim customer 2.9 billion … … Join #2 web_sales date_dim customer Join #4 … Join #3 73,049 534 million 12 million 12 million 73,049720 million 534 million 144 million 144 million 1.4x Speedup 34 80% less
  • 35. TPC-DS Query Q72 35 SELECT i_item_desc, w_warehouse_name, d1.d_week_seq, count(CASE WHEN p_promo_sk IS NULL THEN 1 ELSE 0 END) no_promo, count(CASE WHEN p_promo_sk IS NOT NULL THEN 1 ELSE 0 END) promo, count(*) total_cnt FROM catalog_sales JOIN inventory ON (cs_item_sk = inv_item_sk) JOIN warehouse ON (w_warehouse_sk = inv_warehouse_sk) JOIN item ON (i_item_sk = cs_item_sk) JOIN customer_demographics ON (cs_bill_cdemo_sk = cd_demo_sk) JOIN household_demographics ON (cs_bill_hdemo_sk = hd_demo_sk) JOIN date_dim d1 ON (cs_sold_date_sk = d1.d_date_sk) JOIN date_dim d2 ON (inv_date_sk = d2.d_date_sk) JOIN date_dim d3 ON (cs_ship_date_sk = d3.d_date_sk) LEFT OUTER JOIN promotion ON (cs_promo_sk = p_promo_sk) LEFT OUTER JOIN catalog_returns ON (cr_item_sk = cs_item_sk AND cr_order_number = cs_order_number) WHERE d1.d_week_seq = d2.d_week_seq AND inv_quantity_on_hand < cs_quantity AND d3.d_date > (cast(d1.d_date AS DATE) + interval 5 days) AND hd_buy_potential = '>10000' AND d1.d_year = 1999 AND hd_buy_potential = '>10000' AND cd_marital_status = 'D' AND d1.d_year = 1999 GROUP BY i_item_desc, w_warehouse_name, d1.d_week_seq ORDER BY total_cnt DESC, i_item_desc, w_warehouse_name, d_week_seq LIMIT 100
  • 36. Query Analysis – Q72 CBO OFF Join #1 Join #2 Join #3 Join #4 Join #5 Join #6 Join #7 Join #8 catalog_sales inventory warehouse item customer_demographics date_dim d1 date_dim d2 date_dim d3 household_demographics 1.4 billion 783 million 223 billion 20 300,000223 billion 223 billion 44.7 million 7.5 million 1.6 million 9 million 1.9 million 7,200 73,049 73,049 73,049 8.7 million Really large intermediate results 36
  • 37. Query Analysis – Q72 CBO ON Join #1 Join #2 Join #3 Join #4 Join #8 Join #5 Join #6 Join #7 catalog_sales inventory warehouseitem customer_demographics date_dim d1 date_dim d2date_dim d3 household_demographics 1.4 billion 783 million 20300,000 1.9 million 7,200 73,049 73,049 73,049 238 million 238 million 47.6 million 47.6 million 2,555 1 billion 1 billion 8.7 million Much smaller intermediate results ! 37 2-3 orders of magnitude less! 8.1x Speedup
  • 38. TPC-DS Query Performance 38 0 350 700 1050 1400 1750 q1 q5 q9 q13 q16 q20 q23b q26 q30 q34 q38 q41 q45 q49 q53 q57 q61 q65 q70 q74 q78 q82 q86 q90 q94 q98 Runtime(seconds) without CBO with CBO
  • 39. TPC-DS Query Speedup • TPC-DS query speedup ratio with CBO versus without CBO • 16 queries show speedup > 30% • The max speedup is 8X. • The geo-mean of speedup is 2.2X. 39
  • 40. TPC-DS Query 64 40 WITH cs_ui AS (SELECT cs_item_sk, sum(cs_ext_list_price) AS sale, sum(cr_refunded_cash + cr_reversed_charge + cr_store_credit) AS refund FROM catalog_sales, catalog_returns WHERE cs_item_sk = cr_item_sk AND cs_order_number = cr_order_number GROUP BY cs_item_sk HAVING sum(cs_ext_list_price) > 2 * sum(cr_refunded_cash + cr_reversed_charge + cr_store_credit)), cross_sales AS (SELECT i_product_name product_name, i_item_sk item_sk, s_store_name store_name, s_zip store_zip, ad1.ca_street_number b_street_number, ad1.ca_street_name b_streen_name, ad1.ca_city b_city, ad1.ca_zip b_zip, ad2.ca_street_number c_street_number, ad2.ca_street_name c_street_name, ad2.ca_city c_city, ad2.ca_zip c_zip, d1.d_year AS syear, d2.d_year AS fsyear, d3.d_year s2year, count(*) cnt, sum(ss_wholesale_cost) s1, sum(ss_list_price) s2, sum(ss_coupon_amt) s3 FROM store_sales, store_returns, cs_ui, date_dim d1, date_dim d2, date_dim d3, store, customer, customer_demographics cd1, customer_demographics cd2, promotion, household_demographics hd1, household_demographics hd2, customer_address ad1, customer_address ad2, income_band ib1, income_band ib2, item WHERE ss_store_sk = s_store_sk AND ss_sold_date_sk = d1.d_date_sk AND ss_customer_sk = c_customer_sk AND ss_cdemo_sk = cd1.cd_demo_sk AND ss_hdemo_sk = hd1.hd_demo_sk AND ss_addr_sk = ad1.ca_address_sk AND ss_item_sk = i_item_sk AND ss_item_sk = sr_item_sk AND ss_ticket_number = sr_ticket_number AND ss_item_sk = cs_ui.cs_item_sk AND c_current_cdemo_sk = cd2.cd_demo_sk AND c_current_hdemo_sk = hd2.hd_demo_sk AND c_current_addr_sk = ad2.ca_address_sk AND c_first_sales_date_sk = d2.d_date_sk AND c_first_shipto_date_sk = d3.d_date_sk AND ss_promo_sk = p_promo_sk AND hd1.hd_income_band_sk = ib1.ib_income_band_sk AND hd2.hd_income_band_sk = ib2.ib_income_band_sk AND cd1.cd_marital_status <> cd2.cd_marital_status AND i_color IN ('purple', 'burlywood', 'indian', 'spring', 'floral', 'medium') AND i_current_price BETWEEN 64 AND 64 + 10 AND i_current_price BETWEEN 64 + 1 AND 64 + 15 GROUP BY i_product_name, i_item_sk, s_store_name, s_zip, ad1.ca_street_number, ad1.ca_street_name, ad1.ca_city, ad1.ca_zip, ad2.ca_street_number, ad2.ca_street_name, ad2.ca_city, ad2.ca_zip, d1.d_year, d2.d_year, d3.d_year) SELECT cs1.product_name, cs1.store_name, cs1.store_zip, cs1.b_street_number, cs1.b_streen_name, cs1.b_city, cs1.b_zip, cs1.c_street_number, cs1.c_street_name, cs1.c_city, cs1.c_zip, cs1.syear, cs1.cnt, cs1.s1, cs1.s2, cs1.s3, cs2.s1, cs2.s2, cs2.s3, cs2.syear, cs2.cnt FROM cross_sales cs1, cross_sales cs2 WHERE cs1.item_sk = cs2.item_sk AND cs1.syear = 1999 AND cs2.syear = 1999 + 1 AND cs2.cnt <= cs1.cnt AND cs1.store_name = cs2.store_name AND cs1.store_zip = cs2.store_zip ORDER BY cs1.product_name, cs1.store_name, cs2.cnt
  • 41. Query Analysis – Q64 CBO ON 41 10% slower FileScan (store_sales) Exchange ExchangeSort Aggregate SortMergeJoin BroadcastHashJoin FileScan (store_returns) Exchange Sort BroadcastExchange (cs_ui) ReusedExchange Sort Sort BroadcastHashJoin ReusedExchange ReusedExchange Sort SortMergeJoin Sort Fragment 1 Fragment 2
  • 42. Query Analysis – Q64 CBO OFF 42 FileScan (store_sales) Exchange Exchange Sort Aggregate SortMergeJoin SortMergeJoin FileScan (store_returns) Exchange Sort Sort (cs_ui) SortMergeJoin Sort ReusedExchange Aggregate Sort (cs_ui) ReusedExchange Sort Exchange Fragment 1 Fragment 2
  • 44. Current Status, Credits and Future Work Ron Hu Huawei Technologies
  • 45. Available in Apache Spark 2.2 • Configured via spark.sql.cbo.enabled • ‘Off By Default’. Why? – Spark is used in production – Many Spark users may already rely on “human intelligence” to write queries in best order – Plan on enabling this by default in Spark 2.3 • We encourage you test CBO with Spark 2.2! 45
  • 46. Current Status • SPARK-16026 is the umbrella jira. – 32 sub-tasks have been resolved – A big project spanning 8 months – 10+ Spark contributors involved – 7000+ lines of Scala code have been contributed • Good framework to allow integrations – Use statistics to derive if a join attribute is unique – Benefit star schema detection and its integration into join reorder 46
  • 47. Birth of Spark SQL CBO • Prototype – In 2015, Ron Hu, Fang Cao, etc. of Huawei’s research department prototyped the CBO concept on Spark 1.2. – After a successful prototype, we shared technology with Zhenhua Wang, Fei Wang, etc of Huawei’s product development team. • We delivered a talk at Spark Summit 2016: – “Enhancing Spark SQL Optimizer with Reliable Statistics”. • The talk was well received by the community. – https://issues.apache.org/jira/browse/SPARK-16026 47
  • 48. Collaboration • Good community support – Developers: Zhenhua Wang, Ron Hu, Reynold Xin, Wenchen Fan, Xiao Li – Reviewers: Wenchen, Herman, Reynold, Xiao, Liang-chi, Ioana, Nattavut, Hyukjin, Shuai, ….. – Extensive discussion in JIRAs and PRs (tens to hundreds conversations). – All the comments made the development time longer, but improved code quality. • It was a pleasure working with community. 48
  • 49. Future Work: Cost Based Optimizer • Current cost formula is coarse. Cost = cardinality * weight + size * (1 - weight) • Cannot tell the cost difference between sort- merge join and hash join – spark.sql.join.preferSortMergeJoin defaults to true. • Underestimates (or ignores) shuffle cost. • Will improve cost formula in next release. 49
  • 50. Future Work: Statistics Collection Framework • Advanced statistics: e.g. histograms, sketches. • Hint mechanism. • Partition level statistics. • Speed up statistics collection by sampling data for large tables. 50
  • 51. Conclusion • Motivation • Statistics Collection Framework – Table/Column Level Statistics Collected – Cardinality Estimation (Filters, Joins, Aggregates etc.) • Cost-based Optimizations – Build Side Selection – Multi-way Join Re-ordering • TPC-DS Benchmarks • Demo 51
  • 53. Multi-way Join Reorder – Example • Given A J B J C J D with join conditions A.k1 = B.k1 and B.k2 = C.k2 and C.k3 = D.k3 level 0: p({A}), p({B}), p({C}), p({D}) level 1: p({A, B}), p({A, C}), p({A, D}), p({B, C}), p({B, D}), p({C, D}) level 2: p({A, B, C}), p({A, B, D}), p({A, C, D}), p({B, C, D}) level 3: p({A, B, C, D}) -- final output plan 53
  • 54. Multi-way Join Reorder – Example • Pruning strategy: exclude cartesian product candidates. This significantly reduces the search space. level 0: p({A}), p({B}), p({C}), p({D}) level 1: p({A, B}), p({A, C}), p({A, D}), p({B, C}), p({B, D}), p({C, D}) level 2: p({A, B, C}), p({A, B, D}), p({A, C, D}), p({B, C, D}) level 3: p({A, B, C, D}) -- final output plan 54
  • 55. New Commands in Apache Spark 2.2 • CBO commands – Collect table-level statistics • ANALYZE TABLE table_name COMPUTE STATISTICS – Collect column-level statistics • ANALYZE TABLE table-name COMPUTE STATISTICS FOR COLUMNS column_name1, column_name2, … – Display statistics in the optimized logical plan > EXPLAIN COST > SELECT cc_call_center_sk, cc_call_center_id FROM call_center; … == Optimized Logical Plan == Project [cc_call_center_sk#75, cc_call_center_id#76], Statistics(sizeInBytes=1680.0 B, rowCount=42, hints=none) +- Relation[…31 fields] parquet, Statistics(sizeInBytes=22.5 KB, rowCount=42, hints=none) … 55