Streaming is a Detail
0 likes179 views
The document discusses the current landscape of streaming in analytics, highlighting the distinctions between transactional and analytical approaches, as well as the shift from ETL to ELT. It emphasizes the importance of modern tools like dbt for maintaining data integrity through testing and version control while fostering collaboration among data engineers and analysts. Key considerations for using materialized views versus batched incremental models are also outlined, focusing on data platform capabilities and latency requirements.
Streaming is a Detail
- 1. Streaming is a Detail Current 2023 Amy Chen & Florian Eiden 1
- 2. 2 Introductions Amy Chen Staff Partner Engineer Fun fact: Iʼve made dbt soap 🧼 Florian Eiden Staff Product Manager Fun fact: I live on an 🏝
- 3. Pieces of the puzzle - Transactional vs Analytical - Analytics Engineering vs Data Engineering - Personas - ELT vs ETL - Streaming in the ELT world (bring streaming to the database, or the other way around) - Operational Analytics - Why does it have to be Batch vs Streaming? - Is streaming the biggest trend in analytics? - What is the next big milestone for streaming? - How do we solve CI/CD? Testing? Replayability? - Is Flink a database now? - Is pipeline the right way to bundle logic? What about DAGs? - Logic: plumbing vs business logic
- 5. dbt is ELT 5
- 6. The dbt viewpoint: Build data like developers build applications 6 dbt uses testing, version control, reusable code, and documentation to get to the right answer, faster. Work like engineers Pairing code-based transformation with your favorite git provider means flexibility without chaos. Code Reigns
- 7. Write reusable & referenceable logic with SQL + Jinja
- 8. Infer lineage for automated dependency management.
- 9. 9 SQL-friendly + version control safely expands participants Reusable code speeds development Built-in CI/CD increases pipeline reliability Dependency management speeds troubleshooting 01 Visible lineage increases data understanding Testing and documentation increase data trust 02 03 04 05 06 Data Engineers, Analysts, and Data Scientists Collaborative Code Dashboard A = Dashboard B Automatic Documentation Analysts and Business Users
- 10. Leverage your existing cloud data platform, with out-of-the-box adapters to all major warehouses. Benefit from partnerships across the Modern Data Stack. 10 Data Quality Orchestration Data Ingestion Other Cloud Data Platform Analysis & Visualization Data Catalog & Active Metadata Operational Analytics Develop Test & Document Deploy
- 12. The questions on when to use MVs What are the costs associated with running the materialized view versus a batched incremental model? (this will vary depending on your data platform as some will require different compute nodes) Does your data platform support joins, aggregations, and window functions on MVs if you need them? What are the latency needs of your development environment? In production? (If not near real time, you can make the choice between a batch incremental model or a MV with a longer refresh schedule.) How often do your upstream dependencies update? If your answer is not frequent, you may not need a MV. How large is your dataset?(It might be cheaper to use MVs for extremely large datasets) How often do you need your query refreshed? What are your downstream dependencies and their stakeholders? (If near real time is important, MVs might be the right choice). Do you have real time machine learning models training or applications using your transformed dataset?