Visually Transform Data in Azure Data Factory or Azure Synapse Analytics (PASS Data Community Summit 2023)
0 likes53 views
The document discusses visual data transformation using Azure Data Factory and Azure Synapse Analytics, highlighting data flows as a low-code tool for large data manipulation. It covers capabilities, use cases, orchestration, and pricing models for data flows, comparing them to Power Queries. The session aims to simplify data processing without requiring extensive programming knowledge, offering demos and lessons learned.
More Related Content
Similar to Visually Transform Data in Azure Data Factory or Azure Synapse Analytics (PASS Data Community Summit 2023) (20)
Visually Transform Data in Azure Data Factory or Azure Synapse Analytics (PASS Data Community Summit 2023)
- 1. Visually Transform Data in Azure Data Factory or Azure Synapse Analytics Cathrine Wilhelmsen
- 2. #PASSDataSummit Data Warehousing Big Data and Analytics
- 3. #PASSDataSummit Data Warehousing Big Data and Analytics
- 4. #PASSDataSummit Data Warehousing Big Data and Analytics
- 5. Visually Transform Data in Azure Data Factory or Azure Synapse Analytics Cathrine Wilhelmsen She / Her Solutions Architect Evidi
- 6. #PASSDataSummit Learning Pathway: The Battle of the Data Transformation Tools Visually Transform Data in Azure Data Factory or Azure Synapse Analytics Cathrine Wilhelmsen Power up Your Transformations Game with Power Query in Power BI and Fabric Marthe Moengen Azure Databricks and Notebooks in Fabric – A Transformation Dream Come True? Emilie Rønning The Battle of the Data Transformation Tools Cathrine Wilhelmsen, Marthe Moengen, Emilie Rønning
- 7. #PASSDataSummit Session Description Do you need to clean, convert, aggregate, prepare, or transform large amounts of data, but don't want to spend your time learning a new programming language or writing lots of code? If so, Data Flows in Azure Data Factory or Azure Synapse Analytics could be the tool for you! By using Data Flows, you can build both simple and complex data transformations in a visual editor. These Data Flows are executed on an underlying spark cluster for optimal scale-out performance for big data analytics, without you having to worry about any nitty-gritty details. We will look at the capabilities and use cases for Data Flows, where they best fit into your architecture, and how they compare to Power Queries (called Dataflows Gen2 in Microsoft Fabric). Then, we will work through a few different Data Flows demos to dig deeper into the various transformations available, as well as the expression language and how to use the visual expression builder. Finally, we will cover how to orchestrate and monitor our Data Flows, discuss lessons learned, and explain the pricing model.
- 8. Cathrine Wilhelmsen She / Her Solutions Architect Evidi [email protected] cathrinew.net @cathrinew I love data and coding, as well as teaching and sharing knowledge Microsoft Data Platform MVP Organizing Fabric February Renovating a house
- 10. #PASSDataSummit What is Azure Data Factory? Standalone service for: • Data Integration • Workflow Orchestration • Scheduling
- 11. #PASSDataSummit What is Azure Synapse Analytics? Unified analytics platform: • Data Integration • Data Lake • Data Warehousing • Big Data Analytics • Time-Series Analytics • Data Science
- 16. #PASSDataSummit Copy Data Activity The core activity * Supports 100+ connectors Powerful built-in capabilities * Cathrine's opinion
- 17. #PASSDataSummit Copy Data Activity: Binary Files Source Sink
- 18. #PASSDataSummit Copy Data Activity: Complex Data Source Sink Serialization Deserialization Compression Decompression Column Mapping
- 19. #PASSDataSummit Copy Data Activity: Complex Data Source Sink Serialization Deserialization Compression Decompression Column Mapping Convert file formats
- 20. #PASSDataSummit Copy Data Activity: Complex Data Source Sink Serialization Deserialization Compression Decompression Column Mapping Zip or unzip files
- 21. #PASSDataSummit Copy Data Activity: Complex Data Source Sink Serialization Deserialization Compression Decompression Column Mapping Map columns implicitly or explicitly
- 26. #PASSDataSummit What are Data Flows? • Data transformation at scale • Visual editor, low-code experience • Runs on serverless, managed Spark clusters
- 27. #PASSDataSummit Why use Data Flows? Transform big data without writing code Modify complex structures using the expression language instead of Python, Scala, etc.
- 28. #PASSDataSummit Why use Data Flows? Optimized for data warehousing scenarios Slowly changing dimensions, fact table loading, fuzzy lookups, data quality validation etc.
- 29. #PASSDataSummit Why use Data Flows? Can handle flexible schemas and schema drift Column pattern matching, rule-based mappings, byNames, byPosition, etc.
- 31. #PASSDataSummit What are transformations? • One step in the data flow • Executed sequentially • Order generally doesn’t matter for performance
- 32. #PASSDataSummit Which transformations exist? • Inputs / outputs (Blue) • Multiple inputs / outputs (Purple) • Schema modifiers (Green) • Row modifiers (Orange) • Formatters (Teal) • Flowlets (Turquoise)
- 35. #PASSDataSummit What does orchestration mean? Defining Workflows Which activities to run in which order? Configuring Alerts and Error Handling How to handle unexpected results and failures? Adding Triggers When to execute pipelines?
- 40. #PASSDataSummit Activity Dependencies: Logical AND AND AND AND
- 42. #PASSDataSummit Triggers Execute last published pipeline: On a set Schedule In a Tumbling Window When Event happens Now
- 43. #PASSDataSummit Triggers: Schedule Execute one or more pipelines on a set schedule: • Every Wednesday at 06:00 • Last day of the month at 18:00 • Every Monday at 04:00 and Friday at 20:00
- 44. #PASSDataSummit Triggers: Tumbling Window Execute a single pipeline for each time slice: • For every 15 minutes • For every 1 hour • For every 24 hours
- 45. #PASSDataSummit Triggers: Storage or Custom Events Execute one or more pipelines when: • Blob is Created • Blob is Deleted • Custom Event Happens
- 46. #PASSDataSummit Triggers: Now Execute a single pipeline immediately
- 48. Pricing
- 49. #PASSDataSummit Azure Data Factory Data Flows Basic: $0.274 per vCore-hour General Purpose Standard: $0.343 per vCore-hour Memory Optimized * Prices in USD from November 2023. All activities are prorated by the minute and rounded up. The minimum cluster size is 8 vCores.
- 50. #PASSDataSummit Azure Synapse Analytics Data Flows Basic: $0.257 per vCore-hour General Purpose Standard: $0.325 per vCore-hour Memory Optimized * Prices in USD from November 2023. All activities are prorated by the minute and rounded up. The minimum cluster size is 8 vCores.
- 51. #PASSDataSummit Data Flows: Rounding Up Basic: $0.257 per vCore-hour General Purpose Standard: $0.325 per vCore-hour Memory Optimized * Prices in USD from November 2023. All activities are prorated by the minute and rounded up. The minimum cluster size is 8 vCores.
- 52. #PASSDataSummit Data Flows: Cluster Size Basic: $0.257 per vCore-hour General Purpose Standard: $0.325 per vCore-hour Memory Optimized * Prices in USD from November 2023. All activities are prorated by the minute and rounded up. The minimum cluster size is 8 vCores.
- 53. #PASSDataSummit Data Flows: Cluster Size Basic: $0.257 per vCore-hour General Purpose Standard: $0.325 per vCore-hour Memory Optimized * Prices in USD from November 2023. All activities are prorated by the minute and rounded up. The minimum cluster size is 8 vCores.
- 54. #PASSDataSummit Data Flows: Cluster Size Basic: $0.257 per vCore-hour General Purpose Standard: $0.325 per vCore-hour Memory Optimized * Prices in USD from November 2023. All activities are prorated by the minute and rounded up. The minimum cluster size is 8 vCores.
- 55. #PASSDataSummit Data Flows: Cluster Size Basic: $0.257 per vCore-hour General Purpose: 4 (+4 Driver Cores) Standard: $0.325 per vCore-hour Memory Optimized: 4 (+4 Driver Cores) * Prices in USD from November 2023. All activities are prorated by the minute and rounded up. The minimum cluster size is 8 vCores.
- 56. #PASSDataSummit Data Flows: Cluster Size Basic: $0.257 per vCore-hour = $2.056 per hour General Purpose: 4 (+4 Driver Cores) Standard: $0.325 per vCore-hour = $2.6 per hour Memory Optimized: 4 (+4 Driver Cores) * Prices in USD from November 2023. All activities are prorated by the minute and rounded up. The minimum cluster size is 8 vCores.
- 57. #PASSDataSummit Data Flows: Cluster Size Basic: $0.257 per vCore-hour = $69.9 per hour General Purpose: 256 (+16 Driver Cores) Standard: $0.325 per vCore-hour = $88.4 per hour Memory Optimized: 256 (+16 Driver Cores) * Prices in USD from November 2023. All activities are prorated by the minute and rounded up. The minimum cluster size is 8 vCores.
- 59. #PASSDataSummit Kamil Nowinski’s Cheat Sheet github.com/Azure-Player/CheatSheets
- 60. Keeping Up with Data Flows Resources: aka.ms/dflinks Videos: aka.ms/dataflowvideos
- 61. Lessons Learned
- 62. #PASSDataSummit «Overkill for small data» - Cathrine
- 63. #PASSDataSummit «Smaller datasets for testing will save your butt» - Cathrine
- 64. #PASSDataSummit «Turn. It. Off. When. Finished.» - Cathrine
- 66. Session evaluation Your feedback is important to us! PASSDataCommunitySummit.com/evaluation Evaluate this session at:
- 67. #PASSDataSummit Coming up next in our Learning Pathway: The Battle of the Data Transformation Tools Visually Transform Data in Azure Data Factory or Azure Synapse Analytics Cathrine Wilhelmsen Power up Your Transformations Game with Power Query in Power BI and Fabric Marthe Moengen Azure Databricks and Notebooks in Fabric – A Transformation Dream Come True? Emilie Rønning The Battle of the Data Transformation Tools Cathrine Wilhelmsen, Marthe Moengen, Emilie Rønning
- 68. Thank you! Special thanks to Mark Kromer Cathrine Wilhelmsen [email protected] cathrinew.net @cathrinew