Understanding Azure Data Factory: The What, When, and Why (NIC 2020)
3 likes1,853 views
The document is a presentation on Azure Data Factory that discusses what it is, when and why it would be used, and how to work with it. It defines Azure Data Factory as a data integration service that can copy and transform data. It demonstrates how to use Azure Data Factory to copy data between cloud and on-premises data stores, transform data using mapping and wrangling data flows, and schedule data pipelines using triggers. Common data architectures that use Azure Data Factory are also presented.
Understanding Azure Data Factory: The What, When, and Why (NIC 2020)
- 1. © 2020 Cathrine Wilhelmsen ([email protected])
- 2. Understanding Azure Data Factory The What, When, and Why Cathrine Wilhelmsen NIC · February 6th, 2019
- 3. © 2020 Cathrine Wilhelmsen ([email protected]) Understanding Azure Data Factory
- 5. © 2020 Cathrine Wilhelmsen ([email protected]) Data Warehousing Business Intelligence Artificial Intelligence Big Data and Analytics Machine Learning Data Science
- 6. © 2020 Cathrine Wilhelmsen ([email protected]) Data Warehousing Business Intelligence Artificial Intelligence Big Data and Analytics Machine Learning Data Science
- 7. © 2020 Cathrine Wilhelmsen ([email protected]) What? When? Why?
- 8. © 2020 Cathrine Wilhelmsen ([email protected]) Collect Store Transform Integrate Prepare
- 9. © 2020 Cathrine Wilhelmsen ([email protected])
- 11. © 2020 Cathrine Wilhelmsen ([email protected]) What is Azure Data Factory?
- 12. © 2020 Cathrine Wilhelmsen ([email protected]) What can you do in Azure Data Factory? Copy Data Transform Data
- 13. © 2020 Cathrine Wilhelmsen ([email protected]) What is inside Azure Data Factory? Pipelines Activities Datasets Linked Services Integration Runtimes Triggers Templates
- 14. DEMO Let's look inside Azure Data Factory!
- 15. © 2020 Cathrine Wilhelmsen ([email protected]) What can you do in Azure Data Factory? Copy Data Transform Data
- 16. © 2020 Cathrine Wilhelmsen ([email protected]) What is the Copy Data Activity? * * Cathrine's opinion :)
- 17. © 2020 Cathrine Wilhelmsen ([email protected]) Copy Data Process: Binary Files Source Sink
- 18. © 2020 Cathrine Wilhelmsen ([email protected]) Copy Data Process: Complex Files Source Sink Serialization Deserialization Compression Decompression Column Mapping
- 19. © 2020 Cathrine Wilhelmsen ([email protected]) Copy Data Process: Complex Files Source Sink Serialization Deserialization Compression Decompression Column Mapping
- 20. © 2020 Cathrine Wilhelmsen ([email protected]) Copy Data Process: Complex Files Source Sink Serialization Deserialization Compression Decompression Column Mapping
- 21. © 2020 Cathrine Wilhelmsen ([email protected]) Copy Data Process: Complex Files Source Sink Serialization Deserialization Compression Decompression Column Mapping
- 23. © 2020 Cathrine Wilhelmsen ([email protected]) what if my systems are on-premises?
- 24. © 2020 Cathrine Wilhelmsen ([email protected]) Hybrid Azure Data Factory
- 25. © 2020 Cathrine Wilhelmsen ([email protected]) What are Integration Runtimes?
- 26. © 2020 Cathrine Wilhelmsen ([email protected]) Azure Integration Runtime
- 27. © 2020 Cathrine Wilhelmsen ([email protected]) Self-Hosted Integration Runtime
- 28. © 2020 Cathrine Wilhelmsen ([email protected]) Copy Data Scenarios
- 29. DEMO Let's connect to an on-prem SQL Server!
- 30. © 2020 Cathrine Wilhelmsen ([email protected]) Ok, so we can copy data… Copy Data Transform Data
- 31. © 2020 Cathrine Wilhelmsen ([email protected]) …what about transforming data? Copy Data Transform Data
- 32. © 2020 Cathrine Wilhelmsen ([email protected]) Mapping or Wrangling
- 33. © 2020 Cathrine Wilhelmsen ([email protected]) What are Mapping Data Flows?
- 34. © 2020 Cathrine Wilhelmsen ([email protected]) How do Mapping Data Flows work?
- 35. © 2020 Cathrine Wilhelmsen ([email protected]) What are Wrangling Data Flows?
- 36. © 2020 Cathrine Wilhelmsen ([email protected]) How do Wrangling Data Flows work?
- 37. © 2020 Cathrine Wilhelmsen ([email protected]) Mapping Data Flows Wrangling Data Flows
- 39. © 2020 Cathrine Wilhelmsen ([email protected]) how do we schedule data pipelines?
- 40. © 2020 Cathrine Wilhelmsen ([email protected]) Trigger pipelines…
- 41. © 2020 Cathrine Wilhelmsen ([email protected]) Triggers: Schedule
- 42. © 2020 Cathrine Wilhelmsen ([email protected]) Triggers: Tumbling Window
- 43. © 2020 Cathrine Wilhelmsen ([email protected]) Triggers: Event Based
- 44. © 2020 Cathrine Wilhelmsen ([email protected]) Triggers: Now
- 45. © 2020 Cathrine Wilhelmsen ([email protected]) Monitoring Triggers
- 48. © 2020 Cathrine Wilhelmsen ([email protected]) Advanced Analytics on Big Data https://azure.microsoft.com/en-us/solutions/architecture/advanced-analytics-on-big-data/
- 49. © 2020 Cathrine Wilhelmsen ([email protected]) Real-time Analytics https://azure.microsoft.com/en-us/solutions/architecture/real-time-analytics/
- 50. © 2020 Cathrine Wilhelmsen ([email protected]) Modern Data Warehouse https://azure.microsoft.com/en-us/solutions/architecture/modern-data-warehouse/
- 51. © 2020 Cathrine Wilhelmsen ([email protected]) Ingest Azure Data Factory Serve Azure Synapse Analytics Visualize Power BI Store Azure Data Lake Storage Data Pipeline Orchestration and Monitoring Azure Data Factory Sources Cloud SaaS Prepare Wrangling Data Flows Transform Mapping Data Flows On-Premises
- 52. © 2020 Cathrine Wilhelmsen ([email protected]) Serve Azure Synapse Analytics Visualize Power BI Sources Cloud SaaS On-Premises
- 53. © 2020 Cathrine Wilhelmsen ([email protected]) Good luck!
- 54. © 2020 Cathrine Wilhelmsen ([email protected]) @cathrinew cathrinew.net [email protected] thank you!