Data Quality Patterns in the Cloud with Azure Data Factory
Download as PPTX, PDF6 likes4,750 views
This document discusses data quality patterns when using Azure Data Factory (ADF). It presents two modern data warehouse patterns that use ADF for orchestration: one using traditional ADF activities and another leveraging ADF mapping data flows. It also provides links to additional resources on ADF data flows, data quality patterns, expressions, performance, and connectors.
Data Quality Patterns in the Cloud with Azure Data Factory
- 1. Data Quality Patterns in the Cloud with Azure Data Factory Azure Data Week
- 2. ADF: Simple and Productive ETL in the Cloud at any Scale
- 6. Modern Data Warehouse Pattern Today Applications Dashboards Business/custom apps (structured) Logs, files, and media (unstructured) r Ingest storage Azure Storage/ Data Lake Store Data Loading Azure Data Factory Load flat files into data lake on a schedule Data processing Read data from files using DBFS Orchestration Azure Data Factory Extract and transform relational data Azure Databricks Serving storage Azure SQL DW Load processed data into tables optimized for analytics Clean and join with stored data Load to SQL DW Databases
- 7. Modern Data Warehouse Pattern with ADF Mapping Data Flows Applications Dashboards Business/custom apps (structured) Logs, files, and media (unstructured) r Azure Storage/ Data Lake Store Azure Data Factory Load files into data lake on a schedule Azure Data Factory Extract and transform relational data Azure SQL DW Load processed data into tables optimized for analytics Clean and join disparate data Databases Azure Databricks
- 13. list of connectors full list of compute environments data
- 16. Data Flow
Editor's Notes
- #3: Let’s take a look at the process through the visual user interface Orchestration. Directs other services to execute actions as part of the transformation process. Mapping Data Flows. Develop graphical data transformation logic at scale without writing code using Mapping Data Flows (preview). Mapping Data Flows. Develop graphical data transformation logic at scale without writing code using Mapping Data Flows (preview). Monitoring: Monitor pipeline and activity runs with a simple list view interface