Pipelines and Packages: Introduction to Azure Data Factory (Techorama NL 2019)
3 likes1,488 views
This document discusses Azure Data Factory (ADF) and how it can be used to build and orchestrate data pipelines without code. It describes how ADF is a hybrid data integration service that improves on its previous version. It also explains how existing SSIS packages can be "lifted and shifted" to ADF to modernize solutions while retaining investments. The document demonstrates creating pipelines and data flows in ADF, handling schema drift, and best practices for development.
Pipelines and Packages: Introduction to Azure Data Factory (Techorama NL 2019)
- 2. Pipelines and Packages: Introduction to Azure Data Factory Cathrine Wilhelmsen Techorama NL · Oct 2, 2019
- 3. Pipelines and Packages: Introduction to Azure Data Factory As Data Engineers and ETL Developers, our main responsibilities are to move, transform, integrate and prepare data for our end users as quickly and efficiently as possible. With the ever-increasing volume and variety of data, this can easily start to feel like a daunting task. Azure Data Factory (ADF) is a hybrid data integration service that lets you build, orchestrate and monitor complex and scalable data pipelines - without writing any code. The first version of Azure Data Factory may not have lived entirely up to its nickname "SSIS in the Cloud", but the second version has been drastically improved and expanded with new capabilities. But wait, what's that? You have already invested years and millions in a comprehensive SSIS solution, you say? No problem! You can lift and shift your existing SSIS packages into Azure Data Factory to start modernizing your solution while retaining the investments you have already made. In this session, we will first go through the fundamentals of Azure Data Factory and see how easy it is to build new data pipelines or migrate your existing SSIS packages. Then, we will explore some of the major improvements in Azure Data Factory v2, including the new Mapping Data Flows. Finally, we will look at design patterns and best practices for development to speed up productivity while keeping costs down.
- 5. Data Warehousing Business Intelligence Artificial Intelligence Big Data and Analytics Machine Learning Data Science
- 6. Data Warehousing Business Intelligence Artificial Intelligence Big Data and Analytics Machine Learning Data Science
- 10. 10 years ago…
- 11. SSIS SQL Server Integration Services
- 13. Then…
- 14. ADF v1 Azure Data Factory Version 1
- 17. Today…
- 18. ADF v2 Azure Data Factory Version 2
- 21. What is Azure Data Factory? Hybrid data integration service Complex and scalable pipelines No-code ETL/ELT data flows
- 22. What can you do in Azure Data Factory? Copy Data Transform Data
- 23. What is inside Azure Data Factory? Pipelines Activities Datasets Linked Services Data Flows Templates Triggers
- 24. DEMO Let's look inside Azure Data Factory!
- 25. Wait… I already have thousands of SSIS packages!
- 26. And… You told me to use this Biml thing!
- 28. What does Lift and Shift mean? Lift up existing SSIS packages Shift them to a new location
- 29. Why should you Lift and Shift SSIS? Modernize while retaining investments Continue to use familiar tools Reduce maintenance and costs (*)
- 30. How do you Lift and Shift SSIS? 1. Configure Azure-SSIS Integration Runtime 2. Deploy SSIS Packages to SSISDB in Azure 3. Orchestrate SSIS Packages in Azure Data Factory
- 31. How do you Lift and Shift SSIS? 1. Configure Azure-SSIS Integration Runtime 2. Deploy SSIS Packages to SSISDB in Azure 3. Orchestrate SSIS Packages in Azure Data Factory
- 32. Azure-SSIS Integration Runtime Managed cluster of Azure VMs dedicated to SSIS Billed while running (like all VMs) Manage cost by running when necessary
- 33. DEMO Let’s explore SSIS in Azure Data Factory!
- 34. Pipeline Linked Service Source Sink Activity Data Flow ≈ ≈ ≈ ≈ ≈ ≈ Package Connection Manager Source Destination Control Flow Task Data Flow ADF vs SSIS
- 35. Pipeline Linked Service Source Sink Activity Data Flow ≈ ≈ ≈ ≈ ≈ ≈ Package Connection Manager Source Destination Control Flow Task Data Flow ADF vs SSIS
- 37. What are Mapping Data Flows? Data transformation at scale Runs on Azure Databricks Visual editor, no-code experience
- 38. How do Mapping Data Flows work?
- 39. Why use Mapping Data Flows? Transform Data Upsert Data Load a Data Warehouse Handle schema drift
- 40. What is Schema Drift? Rapidly changing source files and metadata: • Added / Removed Columns • Renamed Column Names • Changed Data Types If not handled properly, Schema Drift can (and most likely will) cause problems in the upstream pipeline
- 41. Schema Drift in SSIS
- 42. Schema Drift in ADF
- 43. Oh no!
- 44. DEMO Let's create some mapping data flows!
- 45. Lessons Learned In ADF, everything has a price SSIS best practices != ADF best practices Learn how to learn and adapt
- 46. Good luck!