Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse Analytics (Data Toboggan 2021)
1 like1,014 views
The document introduces data integration in Azure Synapse Analytics, focusing on the importance of data for business intelligence and machine learning projects. It covers the fundamentals of data pipelines and data flows, how to ingest and transform data without coding, and the orchestration and scheduling of these processes. A practical demo is included to illustrate the building of solutions using Azure Synapse Analytics.
Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse Analytics (Data Toboggan 2021)
- 1. © 2021 Cathrine Wilhelmsen ([email protected]) Pipelines & Data Flows: Introduction to Data Integration in Azure Synapse Analytics Cathrine Wilhelmsen Data Toboggan · January 30th, 2021
- 2. © 2021 Cathrine Wilhelmsen ([email protected]) Session Abstract Do you regularly need to get data for your projects? Yep! 🙋♀️ Data is at the core of every Business Intelligence, Data Science, and Machine Learning project. You need data to understand what has happened in the past, to predict what may happen in the future, to discover patterns and anomalies, and to gain the insight necessary for making faster and better decisions. But before you can do any of those things, you need to ingest, store, transform, integrate, and prepare your data. Guess what? You can do all of those things in Azure Synapse Analytics - without having to write any code! In this session, we will cover the fundamentals of data integration in Azure Synapse Analytics. First, we will go through what Pipelines and Data Flows are. Then, we will quickly build a solution for ingesting and transforming data. Finally, we will look at how to orchestrate and schedule our pipelines, and how to monitor our solution once it has been deployed.
- 3. © 2021 Cathrine Wilhelmsen ([email protected])
- 4. © 2021 Cathrine Wilhelmsen ([email protected]) @cathrinew cathrinew.net
- 5. © 2021 Cathrine Wilhelmsen ([email protected]) Relational Data Data Warehouse Big Data Data Lake
- 6. © 2021 Cathrine Wilhelmsen ([email protected]) Relational Data Data Warehouse Big Data Data Lake
- 7. © 2021 Cathrine Wilhelmsen ([email protected]) Relational Data Data Warehouse Big Data Data Lake
- 8. © 2021 Cathrine Wilhelmsen ([email protected]) Integration Data
- 9. © 2021 Cathrine Wilhelmsen ([email protected]) Data Integration in Azure Synapse Code-Heavy Scripts, Notebooks Low-Code Pipelines, Data Flows
- 10. © 2021 Cathrine Wilhelmsen ([email protected]) Data Integration in Azure Synapse Copy Data Transform Data
- 12. © 2021 Cathrine Wilhelmsen ([email protected]) What are Pipelines?
- 13. © 2021 Cathrine Wilhelmsen ([email protected]) What are Activities?
- 14. © 2021 Cathrine Wilhelmsen ([email protected]) What are Datasets?
- 15. © 2021 Cathrine Wilhelmsen ([email protected]) What are Linked Services?
- 16. © 2021 Cathrine Wilhelmsen ([email protected]) What are Data Flows?
- 17. © 2021 Cathrine Wilhelmsen ([email protected]) What are Triggers?
- 19. © 2021 Cathrine Wilhelmsen ([email protected])
- 20. © 2021 Cathrine Wilhelmsen ([email protected])
- 21. © 2021 Cathrine Wilhelmsen ([email protected])
- 22. © 2021 Cathrine Wilhelmsen ([email protected])
- 23. © 2021 Cathrine Wilhelmsen ([email protected])
- 24. © 2021 Cathrine Wilhelmsen ([email protected])
- 25. © 2021 Cathrine Wilhelmsen ([email protected])
- 26. © 2021 Cathrine Wilhelmsen ([email protected])
- 27. © 2021 Cathrine Wilhelmsen ([email protected]) Code-Heavy Scripts, Notebooks Low-Code Pipelines, Data Flows
- 28. © 2021 Cathrine Wilhelmsen ([email protected]) @cathrinew cathrinew.net [email protected] cathrinew.net/adf