SlideShare a Scribd company logo

Spark SQL Bucketing at Facebook

•
Databricks, profile picture
Databricks

The document provides an overview of bucketing in Spark, as presented by Cheng Su from Facebook at the AI Summit. It covers the concept of bucketing, optimization strategies for Spark bucketing when performing joins and aggregations, and future directions for bucketing compatibility across SQL engines. The agenda includes practical examples and recommendations for effective bucketing implementation.

Data & Analytics
1 of 38
Downloaded 59 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
WIFI SSID:Spark+AISummit | Password: UnifiedDataAnalytics
Cheng Su, Facebook Spark SQL Bucketing at Facebook #UnifiedDataAnalytics #SparkAISummit
About me Cheng Su • Software Engineer at Facebook (Data Infrastructure Organization) • Working in Spark team • Previously worked in Hive/Corona team 3#UnifiedDataAnalytics #SparkAISummit
Agenda • Spark at Facebook • What is Bucketing • Spark Bucketing Optimizations (JIRA: SPARK-19256) • Bucketing Compatability across SQL Engines • The Road Ahead 4#UnifiedDataAnalytics #SparkAISummit
Spark at Facebook 5#UnifiedDataAnalytics #SparkAISummit
What is Bucketing 6#UnifiedDataAnalytics #SparkAISummit Pre-shuffle and (optionally) pre-sort when writing table. Avoid shuffle and (optionally) sort when reading table. table user(id, info) write normal table . . . . . . (2, ) (1, ) (5, ) (1, ) (2, ) (4, ) (3, ) (0, ) write bucketed sorted table . . . . . . (2, ) (1, ) (5, ) (1, ) (2, ) (4, ) (3, ) (0, ) file0 file1 file9 (0, ) (0, ) (4, ) (1, ) (1, ) (5, ) (3, ) (3, ) (2, ) (2, ) shuffle(id) sort(id)
What is Bucketing (query plan) CREATE TABLE user (id INT, info STRING) CLUSTERED BY (id) SORTED BY (id) INTO 8 BUCKETS 7#UnifiedDataAnalytics #SparkAISummit SQL query to create bucketed table InsertIntoTable Sort(id) ShuffleExechange (id, 8, HashFunc) . . . Query plan to write bucketed table INSERT OVERWRITE TABLE user SELECT id, info FROM . . . WHERE . . . SQL query to write bucketed table
What is Bucketing (write path) 8#UnifiedDataAnalytics #SparkAISummit
Spark Bucketing Optimizations (join) 9#UnifiedDataAnalytics #SparkAISummit Avoid shuffle and sort when sort-merge-join bucketed tables SELECT . . . FROM left L JOIN right R ON L.id = R.id SQL query to join tables SortMergeJoin Sort(id) Shuffle(id) Sort(id) Shuffle(id) TableScan(L) TableScan(R) SortMergeJoin TableScan(L) TableScan(R) Query plan to sort-merge- join two bucketed tables with same buckets
Table Scan L Table Scan R ShuffleShuffleShuffle Join Sort Join Sort . . . . . . (2, ) (1, ) (5, ) (0, ) (2, ) (4, ) (3, ) (0, ) . . . . . .(3, ) (9, ) (5, ) (4, ) (2, ) (8, ) (2, ) (1, ) Sort merge join - Shuffle both tables - Sort both tables - Join by buffer one, stream the bigger one Join Sort
Table Scan L Table Scan R Join Join Join Sort merge join of bucketed sorted table - Join by buffer one, stream the bigger one . . . . . . (1, ) (1, ) (5, ) (0, ) (0, ) (4, ) (3, ) (3, ) . . . . . .(1, ) (9, ) (0, ) (4, ) (4, ) (3, ) (7, ) (7, )
12#UnifiedDataAnalytics #SparkAISummit Avoid shuffle when shuffled-hash-join bucketed tables SELECT . . . FROM left L JOIN right R ON L.id = R.id SQL query to join tables ShuffledHashJoin Shuffle(id) Shuffle(id) TableScan(L) TableScan(R) ShuffledHashJoin TableScan(L) TableScan(R) Query plan to shuffled- hash-join two bucketed tables with same buckets Spark Bucketing Optimizations (join)
Table Scan L Table Scan R ShuffleShuffleShuffle Join Build hash table Join . . . . . . (2, ) (1, ) (5, ) (0, ) (2, ) (4, ) (3, ) (0, ) . . . . . .(3, ) (9, ) (5, ) (4, ) (2, ) (8, ) (2, ) (1, ) Shuffled hash join - Shuffle both tables - Join by hash one, stream the bigger one Join Build hash table Build hash table
Table Scan L Table Scan R Join Build hash table Join Shuffled hash join of bucketed table - Join by hash one, stream the bigger one Join Build hash table Build hash table . . . . . . (5 ) (1, ) (5, ) (0, ) (4, ) (8, ) (3, ) (3, ) . . . . . .(9, ) (1, ) (0, ) (4, ) (4, ) (7, ) (3, ) (7, )
15#UnifiedDataAnalytics #SparkAISummit Avoid shuffle and sort when joining non-bucketed, and bucketed table SELECT . . . FROM left L JOIN right R ON L.id = R.id SQL query to join tables SortMergeJoin Sort(id) Shuffle(id) TableScan(L) TableScan(R) Query plan to sort-merge-join non-bucketed table (L) with bucketed table (R) Spark Bucketing Optimizations (join)
Table Scan L (non-bucketed) Table Scan R (bucketed) ShuffleShuffleShuffle Join Sort Join Sort . . . . . . (2, ) (1, ) (5, ) (0, ) (2, ) (4, ) (3, ) (0, ) Sort merge join of non-bucketed and bucketed table - Shuffle non-bucketed table - Sort non-bucketed table - Join by buffer one, stream the bigger one Join Sort . . . . . .(1, ) (9, ) (0, ) (4, ) (4, ) (3, ) (7, ) (7, )
17#UnifiedDataAnalytics #SparkAISummit Avoid shuffle and sort when joining bucketed tables with different buckets SELECT . . . FROM left L JOIN right R ON L.id = R.id SQL query to join tables SortMergeJoin TableScan(L) TableScan(R) Query plan to join 4-buckets-table (L) with 16-buckets-table (R) Spark Bucketing Optimizations (join) SortedCoalesce(4) SortedCoalesceExec (physical plan operator inherits child ordering ) SortedCoalescedRDD (extends CoalescedRDD to read children RDDs in sort-merge-way) (priority-queue)
Table Scan L Table Scan R Join Sort merge join of bucketed sorted table with different buckets - Coalesce the bigger one in sort-merge way - Join by buffer one, stream the bigger one (1, ) (1, ) (3, ) (0, ) (0, ) (2, ) (1, ) (9, ) (0, ) (4, ) (3, ) (7, ) (7, ) (2, ) (2, ) (6, ) (0, ) (0, ) (2, ) (0, ) (2, ) (2, ) (4, ) (6, ) Sorted-Coalesce Join (1, ) (1, ) (3, ) (1, ) (3, ) (7, ) (7, ) (9, ) Sorted-Coalesce
19#UnifiedDataAnalytics #SparkAISummit Avoid shuffle and sort when joining bucketed tables with different buckets SELECT . . . FROM left L JOIN right R ON L.id = R.id SQL query to join tables SortMergeJoin TableScan(L) TableScan(R) Query plan to join 4-buckets-table (L) with 16-buckets-table (R) Spark Bucketing Optimizations (join) Repartition(16) RepartitionWithoutShuffleExe c (physical plan operator inherits child ordering) RepartitionWithoutShuffleRD D (divide-read-filter children RDD partitions)
Table Scan L Table Scan R Join Sort merge join of bucketed sorted table with different buckets - Divide (repartition-w/o- shuffle) the smaller one - Join by buffer one, stream the bigger one (1, ) (1, ) (3, ) (0, ) (0, ) (2, ) (1, ) (9, ) (0, ) (4, ) (3, ) (7, ) (7, ) (2, ) (2, ) (6, ) (0, ) (0, ) Divide (0, ) (4, ) Join (1, ) (1, ) Divide (1, ) (9, ) Join (2, ) Divide (2, ) (2, ) (6, ) Join (3, ) Divide (3, ) (7, ) (7, )
Spark Bucketing Optimizations (group-by) 21#UnifiedDataAnalytics #SparkAISummit Avoid shuffle and sort when sort-aggregate bucketed tables SELECT . . . FROM t GROUP BY id SQL query to group- by table SortAggregate Sort(id) Shuffle(id) TableScan(t) Query plan to sort- aggregate bucketed table SortAggregate TableScan(t)
Table Scan t ShuffleShuffleShuffle Sort . . . . . .(3, ) (9, ) (5, ) (4, ) (2, ) (8, ) (2, ) (1, ) Sort aggregation - Shuffle table - Sort table - Aggregate Aggregate Sort Aggregate Sort Aggregate
Table Scan t Sort aggregation of bucketed table - Aggregate Aggregate Aggregate Aggregate . . . . . .(1, ) (9, ) (0, ) (4, ) (4, ) (3, ) (7, ) (7, )
Spark Bucketing Optimizations (group-by) 24#UnifiedDataAnalytics #SparkAISummit Avoid shuffle when hash-aggregate bucketed tables SELECT . . . FROM t GROUP BY id SQL query to group- by table HashAggregate Shuffle(id) TableScan(t) Query plan to hash- aggregate bucketed table HashAggregate TableScan(t)
Table Scan t ShuffleShuffleShuffle . . . . . .(3, ) (9, ) (5, ) (4, ) (2, ) (8, ) (2, ) (1, ) Hash aggregation - Shuffle table - Aggregate Aggregate Aggregate Aggregate
Table Scan t Hash aggregation of bucketed table - Aggregate Aggregate Aggregate Aggregate . . . . . .(9, ) (1, ) (4 ) (0, ) (4, ) (7, ) (3, ) (7, )
Spark Bucketing Optimizations (union all) 27#UnifiedDataAnalytics #SparkAISummit Avoid shuffle and sort when join/group-by on union-all of bucketed tables SELECT . . . FROM ( SELECT … FROM L UNION ALL SELECT … FROM R ) GROUP BY id SQL query to group-by on union-all of tables SortAggregate Union TableScan(L) Query plan to hash- aggregate union-all of bucketed tables TableScan(R) Change UnionExec to produce SortedCoalescedRDD instead of CoalescedRDD
Table Scan L Table Scan R Union-all Aggregate . . . . . . (2, ) (1, ) (5, ) (0, ) (2, ) (4, ) (3, ) (0, ) . . . . . .(3, ) (9, ) (5, ) (4, ) (2, ) (8, ) (2, ) (1, ) Aggregate after union-all - Union-all of both tables - Shuffle both tables - Sort both tables - Aggregate Union-all Aggregate Union-all Aggregate Shuffle & Sort Shuffle & Sort Shuffle & Sort
Table Scan L Table Scan R Union-all Aggregate Aggregate after union-all of bucketed sorted table - Union-all of both tables in sort-merge way - Aggregate Union-all Union-all . . . . . . (1, ) (1, ) (5, ) (0, ) (0, ) (4, ) (3, ) (3, ) . . . . . .(1, ) (9, ) (0, ) (4, ) (4, ) (3, ) (7, ) (7, ) (0, ) (0, ) (0, ) (4, ) (4, ) (4, ) Aggregate (1, ) (1, ) (1, ) (5, ) (9, ) Aggregate (3, ) (3, ) (3, ) (7, ) (7, )
Spark Bucketing Optimizations (filter) 30#UnifiedDataAnalytics #SparkAISummit Filter pushdown for bucketed table SELECT … FROM t WHERE id = 1 SQL query to read bucketed table with filter on bucketed column (id) Filter Query plan to read bucketed table with filter pushdown PushDownBucketFilter physical plan rule to extract bucketed column filter from FilterExec, then filtering out unnecessary buckets from e.g. HiveTableScanExec (i.e. not read unrelated buckets at all) TableScan(t)SELECT … FROM t WHERE id IN (1, 2, 3)
Bucket Filter Push Down SELECT … FROM t WHERE id = 1 Normal Filter Bucket Filter Push Down . . . . . .(9, ) (1, ) (4 ) (0, ) (4, ) (7, ) (3, ) (7, ) (1, ) - Only read required bucket files (9, ) (1, ) (1, )
Spark Bucketing Optimizations (validation) 32#UnifiedDataAnalytics #SparkAISummit Validate bucketing and sorting before writing bucketed tables INSERT OVERWRITE TABLE t SELECT … FROM … SQL query to write bucketed table InsertIntoTable(t) SortVerifie r Query plan to validate bucketing and sorting before writing table ShuffleVerifierExec compute bucket-id for each row on-the-fly, compare bucket-id with RDD-partition-id ShuffleVerifie r SortVerifierExec compare ordering between current and previous rows
Shuffle Verifier Shuffle Verifier Sort Verifier - Validate bucket id - Validate sort order - Write to table . . . . . .(9, ) (1, ) (0, ) (4, ) (4, ) (3, ) (6, ) (7, ) . . . . . .(9, ) (1, ) (0, ) (4, ) (4, ) (3, ) (6, ) (7, ) Sort Verifier . . . . . .(9, ) (1, ) (0, ) (4, ) (4, )
Spark Bucketing Optimizations (others) 34#UnifiedDataAnalytics #SparkAISummit • Sorted-coalesced-read multiple partitions of bucketed table • Prefer sort-merge-join for bucketed sorted tables • Prefer sort-aggregate for bucketed sorted tables • Avoid shuffle for NULL-safe-equal join (<=>) on bucketed tables • Allow to skip shuffle and sort before writing bucketed table • Automatically align dynamic allocation maximal executors, with buckets • Efficiently hive table sampling support
• Hive hash is different from murmur3 hash! (bitwise-and with 2^31-1 in org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getBucketNumber) • Should use same bucketing hash function (e.g. hive hash) across SQL engines (Spark/Presto/Hive) • Number of buckets of all tables should be divisible by each other (e.g. power-of-two) 35#UnifiedDataAnalytics #SparkAISummit Bucketing Compatability across SQL Engines
• Change number of buckets should be easy and pain-less across compute engines for SQL users • When and What to bucket? • Have more than one query to do join or group-by on some columns 36#UnifiedDataAnalytics #SparkAISummit Bucketing Compatability across SQL Engines
The Road Ahead • Bucketing should be user-transparent • Auto-bucketing project • Audit join/group-by columns information for all warehouse queries • Recommend bucketed columns and number of buckets based on computational cost models • What is problem of bucketing? Can we have better data placement, besides bucketing and partitioning? 37#UnifiedDataAnalytics #SparkAISummit
DON’T FORGET TO RATE AND REVIEW THE SESSIONS SEARCH SPARK + AI SUMMIT

More Related Content

PDF
Deep Dive into the New Features of Apache Spark 3.0
Databricks
 
PDF
Understanding Query Plans and Spark UIs
Databricks
 
PDF
Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...
Databricks
 
PDF
Improving SparkSQL Performance by 30%: How We Optimize Parquet Pushdown and P...
Databricks
 
PDF
Physical Plans in Spark SQL
Databricks
 
PDF
How to Extend Apache Spark with Customized Optimizations
Databricks
 
PDF
Spark + Parquet In Depth: Spark Summit East Talk by Emily Curtin and Robbie S...
Spark Summit
 
PDF
Spark shuffle introduction
colorant
 
Deep Dive into the New Features of Apache Spark 3.0
Databricks
 
Understanding Query Plans and Spark UIs
Databricks
 
Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...
Databricks
 
Improving SparkSQL Performance by 30%: How We Optimize Parquet Pushdown and P...
Databricks
 
Physical Plans in Spark SQL
Databricks
 
How to Extend Apache Spark with Customized Optimizations
Databricks
 
Spark + Parquet In Depth: Spark Summit East Talk by Emily Curtin and Robbie S...
Spark Summit
 
Spark shuffle introduction
colorant
 

What's hot (20)

PDF
Hive Bucketing in Apache Spark with Tejas Patil
Databricks
 
PDF
A Deep Dive into Query Execution Engine of Spark SQL
Databricks
 
PDF
Dynamic Partition Pruning in Apache Spark
Databricks
 
PPTX
Optimizing Apache Spark SQL Joins
Databricks
 
PDF
Understanding and Improving Code Generation
Databricks
 
PDF
The Parquet Format and Performance Optimization Opportunities
Databricks
 
PDF
Apache Spark Core—Deep Dive—Proper Optimization
Databricks
 
PPTX
Spark Shuffle Deep Dive (Explained In Depth) - How Shuffle Works in Spark
Bo Yang
 
PDF
Modularized ETL Writing with Apache Spark
Databricks
 
PDF
Building a SIMD Supported Vectorized Native Engine for Spark SQL
Databricks
 
PDF
Apache Spark Data Source V2 with Wenchen Fan and Gengliang Wang
Databricks
 
PDF
Spark SQL Join Improvement at Facebook
Databricks
 
PDF
Apache Spark Core – Practical Optimization
Databricks
 
PDF
The Apache Spark File Format Ecosystem
Databricks
 
PDF
Parquet performance tuning: the missing guide
Ryan Blue
 
PDF
Introduction to apache spark
Aakashdata
 
PDF
Dive into PySpark
Mateusz Buśkiewicz
 
PDF
Pyspark Tutorial | Introduction to Apache Spark with Python | PySpark Trainin...
Edureka!
 
PPTX
Using Apache Arrow, Calcite, and Parquet to Build a Relational Cache
Dremio Corporation
 
PDF
SQL Performance Improvements At a Glance in Apache Spark 3.0
Kazuaki Ishizaki
 
Hive Bucketing in Apache Spark with Tejas Patil
Databricks
 
A Deep Dive into Query Execution Engine of Spark SQL
Databricks
 
Dynamic Partition Pruning in Apache Spark
Databricks
 
Optimizing Apache Spark SQL Joins
Databricks
 
Understanding and Improving Code Generation
Databricks
 
The Parquet Format and Performance Optimization Opportunities
Databricks
 
Apache Spark Core—Deep Dive—Proper Optimization
Databricks
 
Spark Shuffle Deep Dive (Explained In Depth) - How Shuffle Works in Spark
Bo Yang
 
Modularized ETL Writing with Apache Spark
Databricks
 
Building a SIMD Supported Vectorized Native Engine for Spark SQL
Databricks
 
Apache Spark Data Source V2 with Wenchen Fan and Gengliang Wang
Databricks
 
Spark SQL Join Improvement at Facebook
Databricks
 
Apache Spark Core – Practical Optimization
Databricks
 
The Apache Spark File Format Ecosystem
Databricks
 
Parquet performance tuning: the missing guide
Ryan Blue
 
Introduction to apache spark
Aakashdata
 
Dive into PySpark
Mateusz Buśkiewicz
 
Pyspark Tutorial | Introduction to Apache Spark with Python | PySpark Trainin...
Edureka!
 
Using Apache Arrow, Calcite, and Parquet to Build a Relational Cache
Dremio Corporation
 
SQL Performance Improvements At a Glance in Apache Spark 3.0
Kazuaki Ishizaki
 
Ad

Similar to Spark SQL Bucketing at Facebook (20)

PPTX
Hive Bucketing in Apache Spark
Tejas Patil
 
ODP
The PostgreSQL Query Planner
Command Prompt., Inc
 
PPTX
Hashing Technique In Data Structures
SHAKOOR AB
 
PPT
Hands on data science with r.pptx
Nimrita Koul
 
PPTX
Data engineering and analytics using python
Purna Chander
 
PDF
Tree representation in map reduce world
Yu Liu
 
PPT
SPARK bigdata pyspark databricks learn spark.ppt
HarshjainSinghvi
 
PPT
apache spark presentation for distributed processing
iamdrnaeem
 
PPTX
vFabric SQLFire Introduction
Jags Ramnarayan
 
PDF
Don’t optimize my queries, optimize my data!
Julian Hyde
 
DOCX
Spark_Documentation_Template1
Nagavarunkumar Kolla
 
PDF
20140908 spark sql & catalyst
Takuya UESHIN
 
PDF
Scaling Machine Learning Feature Engineering in Apache Spark at Facebook
Databricks
 
PPTX
Postgresql Database Administration Basic - Day2
PoguttuezhiniVP
 
PPTX
Presentation
Sayed Hoque
 
PPTX
Lazy beats Smart and Fast
Julian Hyde
 
PDF
Rdf conjunctive query selectivity estimation
INRIA-OAK
 
PDF
JDD 2016 - Pawel Szulc - Writing Your Wwn RDD For Fun And Profit
PROIDEA
 
PPTX
SQL Plan Directives explained
Mauro Pagano
 
PDF
Ontop: Answering SPARQL Queries over Relational Databases
Guohui Xiao
 
Hive Bucketing in Apache Spark
Tejas Patil
 
The PostgreSQL Query Planner
Command Prompt., Inc
 
Hashing Technique In Data Structures
SHAKOOR AB
 
Hands on data science with r.pptx
Nimrita Koul
 
Data engineering and analytics using python
Purna Chander
 
Tree representation in map reduce world
Yu Liu
 
SPARK bigdata pyspark databricks learn spark.ppt
HarshjainSinghvi
 
apache spark presentation for distributed processing
iamdrnaeem
 
vFabric SQLFire Introduction
Jags Ramnarayan
 
Don’t optimize my queries, optimize my data!
Julian Hyde
 
Spark_Documentation_Template1
Nagavarunkumar Kolla
 
20140908 spark sql & catalyst
Takuya UESHIN
 
Scaling Machine Learning Feature Engineering in Apache Spark at Facebook
Databricks
 
Postgresql Database Administration Basic - Day2
PoguttuezhiniVP
 
Presentation
Sayed Hoque
 
Lazy beats Smart and Fast
Julian Hyde
 
Rdf conjunctive query selectivity estimation
INRIA-OAK
 
JDD 2016 - Pawel Szulc - Writing Your Wwn RDD For Fun And Profit
PROIDEA
 
SQL Plan Directives explained
Mauro Pagano
 
Ontop: Answering SPARQL Queries over Relational Databases
Guohui Xiao
 
Ad

More from Databricks (20)

PPTX
DW Migration Webinar-March 2022.pptx
Databricks
 
PPTX
Data Lakehouse Symposium | Day 1 | Part 1
Databricks
 
PPT
Data Lakehouse Symposium | Day 1 | Part 2
Databricks
 
PPTX
Data Lakehouse Symposium | Day 2
Databricks
 
PPTX
Data Lakehouse Symposium | Day 4
Databricks
 
PDF
5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop
Databricks
 
PDF
Democratizing Data Quality Through a Centralized Platform
Databricks
 
PDF
Learn to Use Databricks for Data Science
Databricks
 
PDF
Why APM Is Not the Same As ML Monitoring
Databricks
 
PDF
The Function, the Context, and the Data—Enabling ML Ops at Stitch Fix
Databricks
 
PDF
Stage Level Scheduling Improving Big Data and AI Integration
Databricks
 
PDF
Simplify Data Conversion from Spark to TensorFlow and PyTorch
Databricks
 
PDF
Scaling your Data Pipelines with Apache Spark on Kubernetes
Databricks
 
PDF
Scaling and Unifying SciKit Learn and Apache Spark Pipelines
Databricks
 
PDF
Sawtooth Windows for Feature Aggregations
Databricks
 
PDF
Redis + Apache Spark = Swiss Army Knife Meets Kitchen Sink
Databricks
 
PDF
Re-imagine Data Monitoring with whylogs and Spark
Databricks
 
PDF
Raven: End-to-end Optimization of ML Prediction Queries
Databricks
 
PDF
Processing Large Datasets for ADAS Applications using Apache Spark
Databricks
 
PDF
Massive Data Processing in Adobe Using Delta Lake
Databricks
 
DW Migration Webinar-March 2022.pptx
Databricks
 
Data Lakehouse Symposium | Day 1 | Part 1
Databricks
 
Data Lakehouse Symposium | Day 1 | Part 2
Databricks
 
Data Lakehouse Symposium | Day 2
Databricks
 
Data Lakehouse Symposium | Day 4
Databricks
 
5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop
Databricks
 
Democratizing Data Quality Through a Centralized Platform
Databricks
 
Learn to Use Databricks for Data Science
Databricks
 
Why APM Is Not the Same As ML Monitoring
Databricks
 
The Function, the Context, and the Data—Enabling ML Ops at Stitch Fix
Databricks
 
Stage Level Scheduling Improving Big Data and AI Integration
Databricks
 
Simplify Data Conversion from Spark to TensorFlow and PyTorch
Databricks
 
Scaling your Data Pipelines with Apache Spark on Kubernetes
Databricks
 
Scaling and Unifying SciKit Learn and Apache Spark Pipelines
Databricks
 
Sawtooth Windows for Feature Aggregations
Databricks
 
Redis + Apache Spark = Swiss Army Knife Meets Kitchen Sink
Databricks
 
Re-imagine Data Monitoring with whylogs and Spark
Databricks
 
Raven: End-to-end Optimization of ML Prediction Queries
Databricks
 
Processing Large Datasets for ADAS Applications using Apache Spark
Databricks
 
Massive Data Processing in Adobe Using Delta Lake
Databricks
 

Recently uploaded (20)

PPTX
Fluvial_Civilizations_Presentation (1).pptx
alisslovemendoza7
 
PDF
TIC ACTIVIDAD 1geeeeeeeeeeeeeeeeeeeeeeeeeeeeeer3.pdf
Thais Ruiz
 
PPTX
INFO8116 - Week 10 - Slides.pptx data analutics
guddipatel10
 
PDF
An Uncut Conversation With Grok | PDF Document
Mike Hydes
 
PPTX
The whitetiger novel review for collegeassignment.pptx
DhruvPatel754154
 
PDF
blockchain123456789012345678901234567890
tanvikhunt1003
 
PPTX
HSE WEEKLY REPORT for dummies and lazzzzy.pptx
ahmedibrahim691723
 
PPT
Real Life Application of Set theory, Relations and Functions
manavparmar205
 
PPTX
Data Security Breach: Immediate Action Plan
varmabhuvan266
 
PDF
WISE main accomplishments for ISQOLS award July 2025.pdf
StatsCommunications
 
PPTX
Data-Driven Machine Learning for Rail Infrastructure Health Monitoring
Sione Palu
 
PPTX
Introduction to Biostatistics Presentation.pptx
AtemJoshua
 
PPTX
Pipeline Automatic Leak Detection for Water Distribution Systems
Sione Palu
 
PPTX
INFO8116 -Big data architecture and analytics
guddipatel10
 
PPTX
short term project on AI Driven Data Analytics
JMJCollegeComputerde
 
PPT
From Vision to Reality: The Digital India Revolution
Harsh Bharvadiya
 
PPTX
Multiscale Segmentation of Survey Respondents: Seeing the Trees and the Fores...
Sione Palu
 
PPTX
IP_Journal_Articles_2025IP_Journal_Articles_2025
mishell212144
 
PPTX
World-population.pptx fire bunberbpeople
umutunsalnsl4402
 
PDF
202501214233242351219 QASS Session 2.pdf
lauramejiamillan
 
Fluvial_Civilizations_Presentation (1).pptx
alisslovemendoza7
 
TIC ACTIVIDAD 1geeeeeeeeeeeeeeeeeeeeeeeeeeeeeer3.pdf
Thais Ruiz
 
INFO8116 - Week 10 - Slides.pptx data analutics
guddipatel10
 
An Uncut Conversation With Grok | PDF Document
Mike Hydes
 
The whitetiger novel review for collegeassignment.pptx
DhruvPatel754154
 
blockchain123456789012345678901234567890
tanvikhunt1003
 
HSE WEEKLY REPORT for dummies and lazzzzy.pptx
ahmedibrahim691723
 
Real Life Application of Set theory, Relations and Functions
manavparmar205
 
Data Security Breach: Immediate Action Plan
varmabhuvan266
 
WISE main accomplishments for ISQOLS award July 2025.pdf
StatsCommunications
 
Data-Driven Machine Learning for Rail Infrastructure Health Monitoring
Sione Palu
 
Introduction to Biostatistics Presentation.pptx
AtemJoshua
 
Pipeline Automatic Leak Detection for Water Distribution Systems
Sione Palu
 
INFO8116 -Big data architecture and analytics
guddipatel10
 
short term project on AI Driven Data Analytics
JMJCollegeComputerde
 
From Vision to Reality: The Digital India Revolution
Harsh Bharvadiya
 
Multiscale Segmentation of Survey Respondents: Seeing the Trees and the Fores...
Sione Palu
 
IP_Journal_Articles_2025IP_Journal_Articles_2025
mishell212144
 
World-population.pptx fire bunberbpeople
umutunsalnsl4402
 
202501214233242351219 QASS Session 2.pdf
lauramejiamillan
 

Spark SQL Bucketing at Facebook

  • 1. WIFI SSID:Spark+AISummit | Password: UnifiedDataAnalytics
  • 2. Cheng Su, Facebook Spark SQL Bucketing at Facebook #UnifiedDataAnalytics #SparkAISummit
  • 3. About me Cheng Su • Software Engineer at Facebook (Data Infrastructure Organization) • Working in Spark team • Previously worked in Hive/Corona team 3#UnifiedDataAnalytics #SparkAISummit
  • 4. Agenda • Spark at Facebook • What is Bucketing • Spark Bucketing Optimizations (JIRA: SPARK-19256) • Bucketing Compatability across SQL Engines • The Road Ahead 4#UnifiedDataAnalytics #SparkAISummit
  • 6. What is Bucketing 6#UnifiedDataAnalytics #SparkAISummit Pre-shuffle and (optionally) pre-sort when writing table. Avoid shuffle and (optionally) sort when reading table. table user(id, info) write normal table . . . . . . (2, ) (1, ) (5, ) (1, ) (2, ) (4, ) (3, ) (0, ) write bucketed sorted table . . . . . . (2, ) (1, ) (5, ) (1, ) (2, ) (4, ) (3, ) (0, ) file0 file1 file9 (0, ) (0, ) (4, ) (1, ) (1, ) (5, ) (3, ) (3, ) (2, ) (2, ) shuffle(id) sort(id)
  • 7. What is Bucketing (query plan) CREATE TABLE user (id INT, info STRING) CLUSTERED BY (id) SORTED BY (id) INTO 8 BUCKETS 7#UnifiedDataAnalytics #SparkAISummit SQL query to create bucketed table InsertIntoTable Sort(id) ShuffleExechange (id, 8, HashFunc) . . . Query plan to write bucketed table INSERT OVERWRITE TABLE user SELECT id, info FROM . . . WHERE . . . SQL query to write bucketed table
  • 8. What is Bucketing (write path) 8#UnifiedDataAnalytics #SparkAISummit
  • 9. Spark Bucketing Optimizations (join) 9#UnifiedDataAnalytics #SparkAISummit Avoid shuffle and sort when sort-merge-join bucketed tables SELECT . . . FROM left L JOIN right R ON L.id = R.id SQL query to join tables SortMergeJoin Sort(id) Shuffle(id) Sort(id) Shuffle(id) TableScan(L) TableScan(R) SortMergeJoin TableScan(L) TableScan(R) Query plan to sort-merge- join two bucketed tables with same buckets
  • 10. Table Scan L Table Scan R ShuffleShuffleShuffle Join Sort Join Sort . . . . . . (2, ) (1, ) (5, ) (0, ) (2, ) (4, ) (3, ) (0, ) . . . . . .(3, ) (9, ) (5, ) (4, ) (2, ) (8, ) (2, ) (1, ) Sort merge join - Shuffle both tables - Sort both tables - Join by buffer one, stream the bigger one Join Sort
  • 11. Table Scan L Table Scan R Join Join Join Sort merge join of bucketed sorted table - Join by buffer one, stream the bigger one . . . . . . (1, ) (1, ) (5, ) (0, ) (0, ) (4, ) (3, ) (3, ) . . . . . .(1, ) (9, ) (0, ) (4, ) (4, ) (3, ) (7, ) (7, )
  • 12. 12#UnifiedDataAnalytics #SparkAISummit Avoid shuffle when shuffled-hash-join bucketed tables SELECT . . . FROM left L JOIN right R ON L.id = R.id SQL query to join tables ShuffledHashJoin Shuffle(id) Shuffle(id) TableScan(L) TableScan(R) ShuffledHashJoin TableScan(L) TableScan(R) Query plan to shuffled- hash-join two bucketed tables with same buckets Spark Bucketing Optimizations (join)
  • 13. Table Scan L Table Scan R ShuffleShuffleShuffle Join Build hash table Join . . . . . . (2, ) (1, ) (5, ) (0, ) (2, ) (4, ) (3, ) (0, ) . . . . . .(3, ) (9, ) (5, ) (4, ) (2, ) (8, ) (2, ) (1, ) Shuffled hash join - Shuffle both tables - Join by hash one, stream the bigger one Join Build hash table Build hash table
  • 14. Table Scan L Table Scan R Join Build hash table Join Shuffled hash join of bucketed table - Join by hash one, stream the bigger one Join Build hash table Build hash table . . . . . . (5 ) (1, ) (5, ) (0, ) (4, ) (8, ) (3, ) (3, ) . . . . . .(9, ) (1, ) (0, ) (4, ) (4, ) (7, ) (3, ) (7, )
  • 15. 15#UnifiedDataAnalytics #SparkAISummit Avoid shuffle and sort when joining non-bucketed, and bucketed table SELECT . . . FROM left L JOIN right R ON L.id = R.id SQL query to join tables SortMergeJoin Sort(id) Shuffle(id) TableScan(L) TableScan(R) Query plan to sort-merge-join non-bucketed table (L) with bucketed table (R) Spark Bucketing Optimizations (join)
  • 16. Table Scan L (non-bucketed) Table Scan R (bucketed) ShuffleShuffleShuffle Join Sort Join Sort . . . . . . (2, ) (1, ) (5, ) (0, ) (2, ) (4, ) (3, ) (0, ) Sort merge join of non-bucketed and bucketed table - Shuffle non-bucketed table - Sort non-bucketed table - Join by buffer one, stream the bigger one Join Sort . . . . . .(1, ) (9, ) (0, ) (4, ) (4, ) (3, ) (7, ) (7, )
  • 17. 17#UnifiedDataAnalytics #SparkAISummit Avoid shuffle and sort when joining bucketed tables with different buckets SELECT . . . FROM left L JOIN right R ON L.id = R.id SQL query to join tables SortMergeJoin TableScan(L) TableScan(R) Query plan to join 4-buckets-table (L) with 16-buckets-table (R) Spark Bucketing Optimizations (join) SortedCoalesce(4) SortedCoalesceExec (physical plan operator inherits child ordering ) SortedCoalescedRDD (extends CoalescedRDD to read children RDDs in sort-merge-way) (priority-queue)
  • 18. Table Scan L Table Scan R Join Sort merge join of bucketed sorted table with different buckets - Coalesce the bigger one in sort-merge way - Join by buffer one, stream the bigger one (1, ) (1, ) (3, ) (0, ) (0, ) (2, ) (1, ) (9, ) (0, ) (4, ) (3, ) (7, ) (7, ) (2, ) (2, ) (6, ) (0, ) (0, ) (2, ) (0, ) (2, ) (2, ) (4, ) (6, ) Sorted-Coalesce Join (1, ) (1, ) (3, ) (1, ) (3, ) (7, ) (7, ) (9, ) Sorted-Coalesce
  • 19. 19#UnifiedDataAnalytics #SparkAISummit Avoid shuffle and sort when joining bucketed tables with different buckets SELECT . . . FROM left L JOIN right R ON L.id = R.id SQL query to join tables SortMergeJoin TableScan(L) TableScan(R) Query plan to join 4-buckets-table (L) with 16-buckets-table (R) Spark Bucketing Optimizations (join) Repartition(16) RepartitionWithoutShuffleExe c (physical plan operator inherits child ordering) RepartitionWithoutShuffleRD D (divide-read-filter children RDD partitions)
  • 20. Table Scan L Table Scan R Join Sort merge join of bucketed sorted table with different buckets - Divide (repartition-w/o- shuffle) the smaller one - Join by buffer one, stream the bigger one (1, ) (1, ) (3, ) (0, ) (0, ) (2, ) (1, ) (9, ) (0, ) (4, ) (3, ) (7, ) (7, ) (2, ) (2, ) (6, ) (0, ) (0, ) Divide (0, ) (4, ) Join (1, ) (1, ) Divide (1, ) (9, ) Join (2, ) Divide (2, ) (2, ) (6, ) Join (3, ) Divide (3, ) (7, ) (7, )
  • 21. Spark Bucketing Optimizations (group-by) 21#UnifiedDataAnalytics #SparkAISummit Avoid shuffle and sort when sort-aggregate bucketed tables SELECT . . . FROM t GROUP BY id SQL query to group- by table SortAggregate Sort(id) Shuffle(id) TableScan(t) Query plan to sort- aggregate bucketed table SortAggregate TableScan(t)
  • 22. Table Scan t ShuffleShuffleShuffle Sort . . . . . .(3, ) (9, ) (5, ) (4, ) (2, ) (8, ) (2, ) (1, ) Sort aggregation - Shuffle table - Sort table - Aggregate Aggregate Sort Aggregate Sort Aggregate
  • 23. Table Scan t Sort aggregation of bucketed table - Aggregate Aggregate Aggregate Aggregate . . . . . .(1, ) (9, ) (0, ) (4, ) (4, ) (3, ) (7, ) (7, )
  • 24. Spark Bucketing Optimizations (group-by) 24#UnifiedDataAnalytics #SparkAISummit Avoid shuffle when hash-aggregate bucketed tables SELECT . . . FROM t GROUP BY id SQL query to group- by table HashAggregate Shuffle(id) TableScan(t) Query plan to hash- aggregate bucketed table HashAggregate TableScan(t)
  • 25. Table Scan t ShuffleShuffleShuffle . . . . . .(3, ) (9, ) (5, ) (4, ) (2, ) (8, ) (2, ) (1, ) Hash aggregation - Shuffle table - Aggregate Aggregate Aggregate Aggregate
  • 26. Table Scan t Hash aggregation of bucketed table - Aggregate Aggregate Aggregate Aggregate . . . . . .(9, ) (1, ) (4 ) (0, ) (4, ) (7, ) (3, ) (7, )
  • 27. Spark Bucketing Optimizations (union all) 27#UnifiedDataAnalytics #SparkAISummit Avoid shuffle and sort when join/group-by on union-all of bucketed tables SELECT . . . FROM ( SELECT … FROM L UNION ALL SELECT … FROM R ) GROUP BY id SQL query to group-by on union-all of tables SortAggregate Union TableScan(L) Query plan to hash- aggregate union-all of bucketed tables TableScan(R) Change UnionExec to produce SortedCoalescedRDD instead of CoalescedRDD
  • 28. Table Scan L Table Scan R Union-all Aggregate . . . . . . (2, ) (1, ) (5, ) (0, ) (2, ) (4, ) (3, ) (0, ) . . . . . .(3, ) (9, ) (5, ) (4, ) (2, ) (8, ) (2, ) (1, ) Aggregate after union-all - Union-all of both tables - Shuffle both tables - Sort both tables - Aggregate Union-all Aggregate Union-all Aggregate Shuffle & Sort Shuffle & Sort Shuffle & Sort
  • 29. Table Scan L Table Scan R Union-all Aggregate Aggregate after union-all of bucketed sorted table - Union-all of both tables in sort-merge way - Aggregate Union-all Union-all . . . . . . (1, ) (1, ) (5, ) (0, ) (0, ) (4, ) (3, ) (3, ) . . . . . .(1, ) (9, ) (0, ) (4, ) (4, ) (3, ) (7, ) (7, ) (0, ) (0, ) (0, ) (4, ) (4, ) (4, ) Aggregate (1, ) (1, ) (1, ) (5, ) (9, ) Aggregate (3, ) (3, ) (3, ) (7, ) (7, )
  • 30. Spark Bucketing Optimizations (filter) 30#UnifiedDataAnalytics #SparkAISummit Filter pushdown for bucketed table SELECT … FROM t WHERE id = 1 SQL query to read bucketed table with filter on bucketed column (id) Filter Query plan to read bucketed table with filter pushdown PushDownBucketFilter physical plan rule to extract bucketed column filter from FilterExec, then filtering out unnecessary buckets from e.g. HiveTableScanExec (i.e. not read unrelated buckets at all) TableScan(t)SELECT … FROM t WHERE id IN (1, 2, 3)
  • 31. Bucket Filter Push Down SELECT … FROM t WHERE id = 1 Normal Filter Bucket Filter Push Down . . . . . .(9, ) (1, ) (4 ) (0, ) (4, ) (7, ) (3, ) (7, ) (1, ) - Only read required bucket files (9, ) (1, ) (1, )
  • 32. Spark Bucketing Optimizations (validation) 32#UnifiedDataAnalytics #SparkAISummit Validate bucketing and sorting before writing bucketed tables INSERT OVERWRITE TABLE t SELECT … FROM … SQL query to write bucketed table InsertIntoTable(t) SortVerifie r Query plan to validate bucketing and sorting before writing table ShuffleVerifierExec compute bucket-id for each row on-the-fly, compare bucket-id with RDD-partition-id ShuffleVerifie r SortVerifierExec compare ordering between current and previous rows
  • 33. Shuffle Verifier Shuffle Verifier Sort Verifier - Validate bucket id - Validate sort order - Write to table . . . . . .(9, ) (1, ) (0, ) (4, ) (4, ) (3, ) (6, ) (7, ) . . . . . .(9, ) (1, ) (0, ) (4, ) (4, ) (3, ) (6, ) (7, ) Sort Verifier . . . . . .(9, ) (1, ) (0, ) (4, ) (4, )
  • 34. Spark Bucketing Optimizations (others) 34#UnifiedDataAnalytics #SparkAISummit • Sorted-coalesced-read multiple partitions of bucketed table • Prefer sort-merge-join for bucketed sorted tables • Prefer sort-aggregate for bucketed sorted tables • Avoid shuffle for NULL-safe-equal join (<=>) on bucketed tables • Allow to skip shuffle and sort before writing bucketed table • Automatically align dynamic allocation maximal executors, with buckets • Efficiently hive table sampling support
  • 35. • Hive hash is different from murmur3 hash! (bitwise-and with 2^31-1 in org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getBucketNumber) • Should use same bucketing hash function (e.g. hive hash) across SQL engines (Spark/Presto/Hive) • Number of buckets of all tables should be divisible by each other (e.g. power-of-two) 35#UnifiedDataAnalytics #SparkAISummit Bucketing Compatability across SQL Engines
  • 36. • Change number of buckets should be easy and pain-less across compute engines for SQL users • When and What to bucket? • Have more than one query to do join or group-by on some columns 36#UnifiedDataAnalytics #SparkAISummit Bucketing Compatability across SQL Engines
  • 37. The Road Ahead • Bucketing should be user-transparent • Auto-bucketing project • Audit join/group-by columns information for all warehouse queries • Recommend bucketed columns and number of buckets based on computational cost models • What is problem of bucketing? Can we have better data placement, besides bucketing and partitioning? 37#UnifiedDataAnalytics #SparkAISummit
  • 38. DON’T FORGET TO RATE AND REVIEW THE SESSIONS SEARCH SPARK + AI SUMMIT