Correctly Loading Incremental Data at Scale
0 likes63 views
The document discusses the concept of incremental data loading, identifying types of data that can be loaded incrementally, such as clicks and views. It outlines different methods for reading data incrementally, including maximal timestamps and time partitions, along with their pros and cons. The document also compares various techniques such as merge and insert overwrite for efficiently handling late arriving data and maintaining data integrity.
Correctly Loading Incremental Data at Scale
- 1. Correctly Loading Incremental Data at Scale April 16, 2024 Copyright © 2024 Tobiko Data, Inc.
- 2. About Me ● Co-Founder & CTO at Tobiko Data ● Creator of SQLGlot ● Previously Airbnb, Netflix ● Experience with rec sys, xp, semantics layer
- 3. What kind of data is incremental? ● Facts ○ Clicks, views, etc... ● Previous events immutable ● Can be quite large
- 4. What is not incremental? ● Dimensions ○ Users, billing info, etc... ● Rows can change over time ● Usually smaller
- 5. Should I avoid incremental loading?
- 6. Incremental loading exists for a reason
- 7. How do you read data incrementally? maximal timestamp time partitions
- 8. Maximal timestamp ● If the table doesn’t exist, compute the whole history in one shot ● If the table does exist, query it to find the last processed timestamp and then use that to filter the upstream source data.
- 9. Maximal timestamp pros / cons ● Doesn't require extra state ● It assumes that you’re able to load the entire table in one go ● The query is more complicated to write and maintain ● Custom SQL is often needed to handle incremental models differently in development ● You cannot detect or fix data gaps
- 10. Time partitions ● Scheduler tracks what time ranges need to run and passes it to the query
- 11. Time partitions pros / cons ● Requires extra state ● The queries are simpler ● Backfills are more scalable and reliable ● You can easily compute just one day of data or manually fix gaps
- 12. Initial load maximal timestamps vs partitions
- 13. ● Requires a primary key ● Simple to ensure consistency and no duplicates ● Can have performance issues Merge - incremental by unique key
- 14. ● Very efficient ● Doesn't handle updates or late arriving data easily Insert overwrite
- 15. Merge vs Insert overwrite ● Merge is expensive without partition pruning ● Merge is more effective when only a few records need to be updated ● Insert overwrite is inefficient when only a few rows have changed ● Insert overwrite doesn't need to match rows so can be more efficient
- 16. Reading late arriving data
- 17. Writing late arriving data ● Avoiding data leakage with insert overwrite ○ Filter your results to the expected range and insert the complete range ● Use predicate push down with merge ○ Avoid full scanning your data
- 18. Thank you!