3D: DBT using Databricks and Delta
3 likes2,713 views
The document discusses dbt (data build tool) and its integration with Delta Lake and Azure Databricks, focusing on building data pipelines using principles of DataOps. It highlights the capabilities of dbt in transforming data, managing tables and views, and enabling incremental data ingestion while detailing its features such as testing, documentation, and metrics tracking. Additionally, it emphasizes the importance of observability and provides links to resources for further exploration.
3D: DBT using Databricks and Delta
- 1. 3D: DBT, Databricks and Delta Fokko Driesprong Principal Code Connaisseur
- 2. whoami ▪ Fokko Driesprong ▪ Master Distributed Systems & Software Engineering ▪ Code Connaisseur at GoDataDriven ▪ Mostly doing {data,software} engineering ▪ Open source enthousiast ▪ ASF Member ▪ Committer + PMC member on Apache {Airflow, Avro, Druid} ▪ Committer on Apache Parquet
- 3. GoDataDriven ▪ Amsterdam based consultancy ▪ Data {Engineer,Science,Strategy} ▪ And now also analytics engineering! ▪ Around 50 consultants ▪ Used to do Hadoop, now Cloud
- 4. Agenda What is DBT? DBT + Delta Lake DBT + Azure Databricks
- 5. What’s DBT? And why I ❤ it so much
- 6. Data Build Tool ▪ Tool for building data pipelines following the DataOps principles ▪ Simple tool to build complex pipelines ▪ Best practices from software engineering ▪ Linting / Peer reviews / DRY principle / Data testing ▪ SQL First ▪ Encodes organisational knowledge into the pipeline Try it yourself: https://godatadriven.com/blog/tutorial-for-dbt-analytics-engineering-made-easy/ But everyone just calls it DBT
- 7. Data Build Tool ▪ Main focus on integration with DWH platforms ▪ Postgres, Redshift, Snowflake and Bigquery ▪ Support for Spark / Databricks ▪ Created by Fishtown Analytics ▪ Huge open source community ▪ Apache 2.0 Open Source license But everyone just calls it DBT
- 8. DBT The T in Extract-Transform-Load (ELT) Analysts using dbt can transform their data by simply writing select statements, while dbt handles turning these statements into tables and views in a data warehouse.
- 9. Let’s go through at a small example Combine orders and order_lines into a revenue table
- 10. SQL with some Ninja2 sauce Revenue table
- 11. DBT as a SQL Runner Executes the pipeline from the command line
- 12. Seamless integration with the Databricks Metastore ▪ Requires a Hive Metastore ▪ Analyize table
- 13. DBT as a SQL Compiler Compiled SQL: target/compiled/dbtpreprocessing/models/revenue.sql
- 14. Next to the SQL there is documentation Give meaning to the columns and add constraints
- 15. Looking at the docs dbt docs generate dbt docs serve ▪ Columns including types ▪ Test constraints ▪ Statistics ▪ The compiled query
- 16. Testing dbt test ▪ Not-null ▪ Uniqueness ▪ Accepted values ▪ Referential constraints ▪ Custom tests
- 17. How does DBT communicate with Spark? ▪ SQL Over HTTP ▪ Authenticate using the token ▪ Parallel execution
- 18. DBT with Delta Lake
- 19. Switch to incremental ingestion Using the Delta format ▪ ACID dataformat by Databricks ▪ Linux Software Foundation ▪ Allows MERGE INTO ▪ Enabled incremental imports
- 20. Switch to incremental Delta If the table doesn’t exists (yet)
- 21. Switch to incremental Delta Incremental MERGE INTO if the table exists
- 23. In practice Incremental imports ▪ Watermark column ▪ Only load the changed orders ▪ Also interesting for Users table
- 24. DBT Macro’s Running it a second time ▪ Don’t Repeat Yourself ▪ Write a Macro instead
- 25. DBT with Azure Databricks
- 26. Observability is king Keeping track of your pipelines ▪ Building trust ▪ Track aggregated metrics computed by Spark ▪ Application insights ▪ Centralized system
- 27. Very simple Hive UDF Keep track of stats over time
- 28. Small snippet of Scala Sends the metrics to Application Insights
- 29. Use the UDF in DBT Sends the metrics to Application Insights ▪ Register the UDF ▪ Keep track of ▪ Seconds since last order ▪ Number of orders
- 30. Be proactive Before there are angry managers at your desk ▪ Keep track of the metric ▪ Send alerts on business rules ▪ Outlier based on historical distributions
- 31. Feedback Your feedback is important to us. Don’t forget to rate and review the sessions. ▪ Code available at: ▪ https://github.com/godatadriven/dbt-data-ai-summit ▪ https://github.com/godatadriven/azure-dbt-logger
- 33. Code example
- 34. Two Columns ▪ Bulleted list format ▪ Bulleted list format ▪ Bulleted list format ▪ Bulleted list format ▪ Bulleted list format ▪ Bulleted list format ▪ Bulleted list format ▪ Bulleted list format Headline FormatHeadline Format
- 35. Attribution Format Second line of attribution This is a template for a quote slide. This is where the quote goes. Attribute the source below…
- 36. Databricks simplifies data and AI so data teams can innovate faster
- 37. Logos
- 38. Databricks Logos