Optimizing Delta/Parquet Data Lakes for Apache Spark
3 likes2,674 views
Matthew Powers gave a presentation on optimizing Delta and Parquet data lakes. He discussed the benefits of using Delta lakes such as built-in time travel, compacting, and vacuuming capabilities. Delta lakes provide these features for free on top of Parquet files and a transaction log. Powers demonstrated how to create, compact, vacuum, partition, filter, and update Delta lakes in Spark. He showed that partitioning data significantly improves query performance by enabling data skipping and filtering at the partition level.
![Filtering unpartitioned lake
31
== Physical Plan ==
Project [first_name#12, last_name#13, country#14]
+- Filter (((isnotnull(country#14) && isnotnull(first_name#12)) && (country#14 = Russia)) &&
StartsWith(first_name#12, M))
+- FileScan csv [first_name#12,last_name#13,country#14]
Batched: false,
Format: CSV,
Location: InMemoryFileIndex[file:/Users/powers/Documents/tmp/blog_data/people.csv],
PartitionFilters: [],
PushedFilters: [IsNotNull(country), IsNotNull(first_name), EqualTo(country,Russia),
StringStartsWith(first_name,M)],
ReadSchema: struct](https://image.slidesharecdn.com/matthewpowers-191031205821/85/Optimizing-Delta-Parquet-Data-Lakes-for-Apache-Spark-31-320.jpg)
![Filtering partitioned lake
35
== Physical Plan ==
*(1) Project [first_name#662, last_name#663, country#664]
+- *(1) Filter (isnotnull(first_name#662) && StartsWith(first_name#662, M))
+- *(1) FileScan parquet [first_name#662,last_name#663,country#664]
Batched: true,
Format: Parquet,
Location: TahoeLogFileIndex[file:/…/tmp/europe_partitioned1],
PartitionCount: 1,
PartitionFilters: [isnotnull(country#664), (country#664 = Russia)],
PushedFilters: [IsNotNull(first_name), StringStartsWith(first_name,M)],
ReadSchema: struct<first_name:string,last_name:string>](https://image.slidesharecdn.com/matthewpowers-191031205821/85/Optimizing-Delta-Parquet-Data-Lakes-for-Apache-Spark-35-320.jpg)
![Comparing physical plans
36
Unpartitioned
Project [first_name#12, last_name#13, country#14]
+- Filter (((isnotnull(country#14) && isnotnull(first_name#12))
&& (country#14 = Russia)) && StartsWith(first_name#12, M))
+- FileScan csv [first_name#12,last_name#13,country#14]
Batched: false,
Format: CSV,
Location: InMemoryFileIndex[….],
PartitionFilters: [],
PushedFilters: [IsNotNull(country), IsNotNull(first_name),
EqualTo(country,Russia), StringStartsWith(first_name,M)],
ReadSchema: struct
Partitioned
Project [first_name#662, last_name#663, country#664]
+- Filter (isnotnull(first_name#662) && StartsWith(first_name#662, M))
+- FileScan parquet [first_name#662,last_name#663,country#664]
Batched: true,
Format: Parquet,
Location: TahoeLogFileIndex[file:/…/tmp/europe_partitioned1],
PartitionCount: 1,
PartitionFilters: [isnotnull(country#664), (country#664 =
Russia)],
PushedFilters: [IsNotNull(first_name),
StringStartsWith(first_name,M)],
ReadSchema: struct<first_name:string,last_name:string>](https://image.slidesharecdn.com/matthewpowers-191031205821/85/Optimizing-Delta-Parquet-Data-Lakes-for-Apache-Spark-36-320.jpg)
Optimizing Delta/Parquet Data Lakes for Apache Spark
- 1. WIFI SSID:Spark+AISummit | Password: UnifiedDataAnalytics
- 2. Matthew Powers, Prognos Health Optimizing Delta / Parquet Data Lakes #UnifiedDataAnalytics #SparkAISummit
- 3. Agenda • Why Delta? • Delta basics and transaction log • Compacting Delta lake • Vacuuming old files • Partitioning Delta lakes • Deleting rows • Persisting transformations in columns 3
- 4. About 4 MungingData • Time travel • Compacting • Vacuuming • Update columns
- 5. Contact me • GitHub: MrPowers • Email: [email protected] • Delta Slack channel • Open source hacking 5
- 6. What is Delta lake? • Parquet + transaction log • Provides awesome features for free! 6
- 7. Delta Lake =!= Databricks Delta 7 https://github.com/delta-io/delta/issues/49
- 8. #UnifiedDataAnalytics #SparkAISummit TL;DR • 1 GB files • No nested directories 8
- 9. #UnifiedDataAnalytics #SparkAISummit 9 Delta Lake Slack says 1GB files
- 11. Why does compaction speed up lakes? • Parquet: files need to be listed before they are read. Listing is expensive in object stores. • Delta: Data is read via the transaction log. • Easier for Spark to read partitioned lakes into memory partitions. 11
- 12. Sample Data 12
- 13. Create Delta Data Lake 13
- 14. Delta Lake on Disk 14
- 16. Code examples 16
- 17. Compact Delta Data Lake 17
- 20. Compacting Delta lakes without breaking downstream apps 20 https://github.com/delta-io/delta/issues/146
- 21. 21
- 22. Delta Lake Vacuum • Files marked for removal older than the retention period • Default retention period is 7 days • Not going to improve performance 22
- 23. Vacuum Delta Data Lake 23
- 25. Optimal number of partitions (delta) 25
- 28. Optimal number of partitions (parquet) 28 https://github.com/MrPowers/spark-daria/blob/master/src/main/scala/com/github/ mrpowers/spark/daria/utils/DirHelpers.scala
- 29. Why partition data lakes? • Data skipping • Massively improve query performance • I’ve seen queries run 50-100 times faster on partitioned lakes 29
- 30. Sample data 30
- 31. Filtering unpartitioned lake 31 == Physical Plan == Project [first_name#12, last_name#13, country#14] +- Filter (((isnotnull(country#14) && isnotnull(first_name#12)) && (country#14 = Russia)) && StartsWith(first_name#12, M)) +- FileScan csv [first_name#12,last_name#13,country#14] Batched: false, Format: CSV, Location: InMemoryFileIndex[file:/Users/powers/Documents/tmp/blog_data/people.csv], PartitionFilters: [], PushedFilters: [IsNotNull(country), IsNotNull(first_name), EqualTo(country,Russia), StringStartsWith(first_name,M)], ReadSchema: struct
- 32. Partitioning the data lake 32
- 33. Partitioned lake on disk 33
- 35. Filtering partitioned lake 35 == Physical Plan == *(1) Project [first_name#662, last_name#663, country#664] +- *(1) Filter (isnotnull(first_name#662) && StartsWith(first_name#662, M)) +- *(1) FileScan parquet [first_name#662,last_name#663,country#664] Batched: true, Format: Parquet, Location: TahoeLogFileIndex[file:/…/tmp/europe_partitioned1], PartitionCount: 1, PartitionFilters: [isnotnull(country#664), (country#664 = Russia)], PushedFilters: [IsNotNull(first_name), StringStartsWith(first_name,M)], ReadSchema: struct<first_name:string,last_name:string>
- 36. Comparing physical plans 36 Unpartitioned Project [first_name#12, last_name#13, country#14] +- Filter (((isnotnull(country#14) && isnotnull(first_name#12)) && (country#14 = Russia)) && StartsWith(first_name#12, M)) +- FileScan csv [first_name#12,last_name#13,country#14] Batched: false, Format: CSV, Location: InMemoryFileIndex[….], PartitionFilters: [], PushedFilters: [IsNotNull(country), IsNotNull(first_name), EqualTo(country,Russia), StringStartsWith(first_name,M)], ReadSchema: struct Partitioned Project [first_name#662, last_name#663, country#664] +- Filter (isnotnull(first_name#662) && StartsWith(first_name#662, M)) +- FileScan parquet [first_name#662,last_name#663,country#664] Batched: true, Format: Parquet, Location: TahoeLogFileIndex[file:/…/tmp/europe_partitioned1], PartitionCount: 1, PartitionFilters: [isnotnull(country#664), (country#664 = Russia)], PushedFilters: [IsNotNull(first_name), StringStartsWith(first_name,M)], ReadSchema: struct<first_name:string,last_name:string>
- 37. Directly grabbing the partitions is faster for Parquet lakes… 37 Directly grabbing partitions was 83 times faster than relying on partition filters for a simple query
- 38. Real partitioned data lake • Updates every 3 hours • Has 5 million files • 15,000 files are being added every day • Still great for a lot of queries 38
- 39. Creating partitioned lake (2/3) 39
- 40. Partitioned lake on disk (2/3) 40
- 41. Creating partitioned lake (3/3) 41
- 42. Incrementally updating partitioned lakes • Small file problem grows quickly • Compaction is hard 42
- 43. Filtering data from a lake 43
- 44. We can delete rows in Delta lakes 44
- 45. Deleting under the hood 45
- 46. Append a column on the fly 46
- 48. Append a column in Delta 48
- 49. Delta lake downsides… not many 49
- 50. Contact me • GitHub: MrPowers • Email: [email protected] • Delta Slack channel • Open source hacking 50
- 51. DON’T FORGET TO RATE AND REVIEW THE SESSIONS SEARCH SPARK + AI SUMMIT