Delta: Building Merge on Read
1 like•549 views
The document discusses the implementation of Merge on Read (MoR) in Delta Lake, contrasting it with Copy on Write (CoW) and detailing the architecture, strategies, and testing methods associated with MoR. It highlights the effectiveness of MoR in scenarios with high random updates and the necessity for periodic rematerialization to maintain read performance. The presentation also outlines challenges faced, solution strategies, and the importance of view selection in optimizing read and write operations.
![Delta Lake Merge - Under the hood
▪ source: new data, target: existing data (Delta table)
▪ Phase 1: Find the input files in target that are touched by the rows that
satisfy the condition and verify that no two source rows match with the
same target row [innerJoin]
▪ Phase 2: Read the touched files again and write new files with updated
and/or inserted rows
▪ Phase 3: Use the Delta protocol to atomically remove the touched files and
add the new files (write stuff to object/blob store)](https://image.slidesharecdn.com/11breesekarpov-201124182935/85/Delta-Building-Merge-on-Read-8-320.jpg)
![Testing [normally] takes a long time
▪ Things to consider:
▪ How many tests are sufficient?
▪ How can I make them as even as possible?
▪ What do you actually want to test?
▪ Why is this part so hard and manual?
▪ Databricks has a `/runs/submit` API - starts a fresh cluster for each run
▪ Databricks notebooks have widgets which act as params
▪ Let’s do 3 tests for each viewType (method) and each operation
(read/write) ⇒ 3 * 4 * 2 = 24 tests!
But it doesn’t have to!](https://image.slidesharecdn.com/11breesekarpov-201124182935/85/Delta-Building-Merge-on-Read-20-320.jpg)
![Configuring the API
Check out my gitHub [https://github.com/justinbreese/databricks-gems#perftestautomationpy]
Made a simple script that leverages the Databricks runs/submit API
Run info
Cluster info
Here is what we will create:
Run Operation Method
0 Read leftJoinUnionInserts
1 Read leftJoinUnionInserts
2 Read leftJoinUnionInserts
0 Read outerJoined
1 Read outerJoined
2 Read outerJoined
0 Read antiJoinLeftJoinUnion
1 Read antiJoinLeftJoinUnion
2 Read antiJoinLeftJoinUnion](https://image.slidesharecdn.com/11breesekarpov-201124182935/85/Delta-Building-Merge-on-Read-23-320.jpg)
![Calling the API
Check out gitHub [https://github.com/justinbreese/databricks-gems#perftestautomationpy]
Made a simple script that leverages the Databricks runs/submit API
python3 perfTestAutomation.py -t <userAccessToken> -s 0 -j artifacts/perfTest.json](https://image.slidesharecdn.com/11breesekarpov-201124182935/85/Delta-Building-Merge-on-Read-24-320.jpg)
Delta: Building Merge on Read
- 1. Building Merge on Read on Delta Lake Justin Breese Senior Solutions Architect Nick Karpov Resident Solutions Architect
- 2. Who are we? Justin Breese [email protected] | Los Angeles Senior Strategic Solutions Architect I pester Nick with a lot of questions and thoughts Nick Karpov [email protected] | San Francisco Senior Resident Solutions Architect History & Music
- 3. Agenda ▪ Background: Copy on Write (COW) & Merge on Read (MOR) ▪ Use case, challenges, & MOR strategies ▪ Testing: choosing the right MOR strategy ▪ Rematerialization?
- 4. Problem statement(s) ▪ Dealing with highly random and update heavy CDC streams ▪ Wanting to be able to get fresh data at any given time Summary ▪ Using MOR allows for faster writes and still get reads that can meet SLAs
- 5. Building Merge on Read on Data Lake ▪ What is Merge on Read (MOR) and Copy on Write (COW)? ▪ What is the use case? ▪ Why did we build? ▪ What is the architecture? ▪ How to test and verify it?
- 6. Copy on Write (COW) and Merge on read (MOR)
- 7. Copy on Write (COW) ▪ TL;DR the merge is done during the write ▪ Default config for Delta Lake ▪ Data is “merged” into a Delta table by physically rewriting existing files with modifications before making available to the reader ▪ In Delta Lake, merge is a three-step process ▪ Great for write once read many scenarios ▪
- 8. Delta Lake Merge - Under the hood ▪ source: new data, target: existing data (Delta table) ▪ Phase 1: Find the input files in target that are touched by the rows that satisfy the condition and verify that no two source rows match with the same target row [innerJoin] ▪ Phase 2: Read the touched files again and write new files with updated and/or inserted rows ▪ Phase 3: Use the Delta protocol to atomically remove the touched files and add the new files (write stuff to object/blob store)
- 9. COW: What is Delta Lake doing under the hood? Phase 2: Read the touched files again and write new files with updated and/or inserted rows. The type of join can vary depending on the conditions of the merge: ▪ Insert only merge (e.g. no updates/deletes) → leftAntiJoin on the source to find the inserts ▪ Matched only clauses (e.g. when matched) → rightOuterJoin ▪ Else (e.g. you have updates, deletes, and inserts) → fullOuterJoin Phase 2 double click
- 10. Merge on Read (MOR) ▪ TL;DR the “merge” is done during the read ▪ Common strategy: don’t logically merge until you NEED the result ▪ Implementation? Two tables and a view ▪ Materialized table ▪ Changelog table (can be a diff, Avro, parquet, etc.) ▪ View that acts as the referee between the two and is the source of truth
- 11. Which one do you pick? Well it depends... or write many read less write less read many
- 12. Use case
- 13. Use case info ▪ 100-200/second (6k-12k/minute) ▪ CDC data coming from Kafka ▪ usually 1-3 columns are changing ▪ partial updates ▪ Each row has a unique ID ▪ 200GB active files; growing at a small rate ▪ SLA: read updates to point lookups in <5 min ▪ Currently doing daily batch overwrites; data can be up to 24 hours stale
- 14. Initial observations and problems encountered ▪ Lots of updates: 96% of events ▪ Matching condition is uniformly distributed across the target ▪ No natural partitioning keys ▪ Sample of 50k events could have 2k different days of updates ▪ Default Delta Lake Merge configs were not performing well ▪ Ended up rewriting almost the entire table each merge
- 15. Architecture: what did we settle on? MOR This is what we will talk about
- 16. Snapshot & Changeset ▪ Snapshot: base table ▪ Changset: append only Primary Key id Most recent data fragno Partitioning optional (depends on use case) many data columns …. Primary Key id Most recent data fragno Partitioning Structured Streaming batchId (this is important!) many data columns ...
- 17. Changeset ▪ Get the unique values in the changeset - primaryKey and latest ▪ As I have partial updates, I need to coalesce(changes, baseline) ▪ Check to understand if the dataframe can be broadcasted? ▪ If I believe I can broadcast 1GB data and each row is 364b, then I can broadcast anything up to 2.8M rows. If the changeset is > 2.8M rows ⇒ do not broadcast -- because memory! * if your changeset is small enough
- 18. View: Methods to join rankedChangeset into the baseline doubleRankOver fullOuterJoin leftJoinAntiJoin broadcastable! leftJoinUnionInserts: broadcastable! Great if you are guaranteed that your inserts are not upserts! ▪ Now that we have our changeset… we still need to compare these values to the baseline table to get the latest by id ▪ There are several methods to do this
- 19. How to pick the right view - perfTesting!
- 20. Testing [normally] takes a long time ▪ Things to consider: ▪ How many tests are sufficient? ▪ How can I make them as even as possible? ▪ What do you actually want to test? ▪ Why is this part so hard and manual? ▪ Databricks has a `/runs/submit` API - starts a fresh cluster for each run ▪ Databricks notebooks have widgets which act as params ▪ Let’s do 3 tests for each viewType (method) and each operation (read/write) ⇒ 3 * 4 * 2 = 24 tests! But it doesn’t have to!
- 21. Create the widgets in your Notebooks Create your results payload (note: we are calling the widgets as params)
- 22. Create a timer function Save to Delta table (note: payload) Operation to test Case statement to match the method and supply the correct view - send it to the stopwatch utility
- 23. Configuring the API Check out my gitHub [https://github.com/justinbreese/databricks-gems#perftestautomationpy] Made a simple script that leverages the Databricks runs/submit API Run info Cluster info Here is what we will create: Run Operation Method 0 Read leftJoinUnionInserts 1 Read leftJoinUnionInserts 2 Read leftJoinUnionInserts 0 Read outerJoined 1 Read outerJoined 2 Read outerJoined 0 Read antiJoinLeftJoinUnion 1 Read antiJoinLeftJoinUnion 2 Read antiJoinLeftJoinUnion
- 24. Calling the API Check out gitHub [https://github.com/justinbreese/databricks-gems#perftestautomationpy] Made a simple script that leverages the Databricks runs/submit API python3 perfTestAutomation.py -t <userAccessToken> -s 0 -j artifacts/perfTest.json
- 25. View the results leftJoinUnionInserts in the winner for the view
- 26. Recap thus far Now we will talk about this part
- 28. Periodic Rematerialization ▪ If changes are getting appended consistently, then you’ll have more and more rows to compare against ▪ This makes your read performance degrade over time ▪ Therefore, you need to do a periodic job that will reset your baseline table ▪ And yes, there are some choices that you have for this: Because you need to reset your baseline table for read perf Method Consideration(s) Merge Easy, very helpful if you have many larger partitions and only a smaller subset of partitions need to be changed, and built into Delta Lake Overwrite Easy, great if you do not have or cannot partition, or if all/most partitions need to be changed replaceWhere Moderate, only can be used if you have partitions, built into Delta Lake
- 29. Periodic Rematerialization ▪ Now that we’ve materialized the new changes into the baseline, we want to delete those batches that we don’t need ▪ Since we partitioned by batchId, when we delete those previous batches, this is a metadata only operation and super fast/cheap - line 68 ▪ We do this so we don’t duplicate changes and because we don’t need them anymore ▪ Remember: we have an initial bronze table that has all of our changes so we always have this if we ever need them Code! Remember that we said that the batchId is important?
- 30. Periodic Rematerialization ▪ Yes, you can even do some perfTesting on this to understand which method fits your use case best ▪ Our use case ended up using overwrite as it was a better fit ▪ Changes happened very randomly; going back up to 2000+ days ▪ Dataset was ~200GB; partitioning was not able to be effective ▪ 200GB is small and we can overwrite the complete table in <10 min with 80 cores
- 31. Final recap ▪ Talked about the use case ▪ Introduced the MOR architecture ▪ Talked about the two tables ▪ Different views and understanding their differences ▪ How to test the different view methods ▪ Periodic rematerialization
- 32. This wouldn’t have been possible without help from: Chris Fish Daniel Tomes Tathagata Das (TD) Burak Yavuz Joe Widen Denny Lee Paul Roome
- 33. Feedback Your feedback is important to us. Don’t forget to rate and review the sessions.