Getting Started: Data Factory in Microsoft Fabric (Microsoft Fabric Community Conference 2024)
1 like535 views
The document provides an overview of Microsoft Fabric's Data Factory, detailing its no-code/low-code data integration capabilities and features such as data ingestion, transformation, and orchestration. It includes descriptions of dataflows, pipelines, and compliance services, emphasizing the importance of securing sensitive data. The agenda outlines various sessions that cover topics from getting started to advanced usage and compliance strategies.
Getting Started: Data Factory in Microsoft Fabric (Microsoft Fabric Community Conference 2024)
- 2. Getting Started: Data Factory in Microsoft Fabric Shireen Bahadur Product Manager, Microsoft Azure Data Cathrine Wilhelmsen Lead Consultant, Evidi Tess Kassier Senior Product Manager, Delphix
- 3. Shireen Bahadur Product Manager, Azure Data Microsoft /shireen-bahadur Cathrine Wilhelmsen Lead Consultant Evidi /cathrinewilhelmsen Tess Kassier Senior Product Manager Delphix /tessmaggio
- 4. Agenda 01 Introduction to Data Factory 02 Ingest and Transform using Dataflows 03 Ingest and Orchestrate using Data Pipelines 04 Lineage and Monitoring 05 Delphix Compliance Services 06 Q&A
- 6. What is Data Factory in Microsoft Fabric? No-code / low-code data integration experience with: • Scale and power of Azure Data Factory • Ease-of-use of Power Query • Intelligence of Copilot
- 7. The Evolution of Data Factory 2014 2017 2019 2015 2023
- 8. Ingest Data Transform Data
- 9. Orchestrate Workflows Ingest Data Transform Data
- 10. Orchestrate Workflows Ingest Data Transform Data
- 11. Orchestrate Workflows Pipelines Ingest Data Dataflows / Pipelines Transform Data Dataflows
- 12. Orchestrate Workflows Pipelines Ingest Data Dataflows / Pipelines Transform Data Dataflows
- 14. What are Dataflows Gen 2? Self-serve Data preparation Familiar experience with Power Query + M Language File, relational, multi-dimensional, SaaS data experiences Embedded into Microsoft citizen experiences – Excel, Power BI, Power Apps & Automate, Dynamics 365 Next Generation of Power BI Dataflows Shorter authoring experience New output Data Destinations New refresh history and monitoring Integration with Data pipelines
- 15. Components of a Dataflow 1. Get Data (connectors) 2. Queries Pane 3. Diagram View 4. Data Preview pane 5. Query settings pane
- 16. Demo: Ingest and Transform using Dataflows
- 17. Orchestrate Workflows Pipelines Ingest Data Dataflows / Pipelines Transform Data Dataflows
- 18. Orchestrate Workflows Pipelines Ingest Data Dataflows / Pipelines Transform Data Dataflows
- 19. Ingest and Orchestrate using Data Pipelines
- 20. What are Data Pipelines? Data Pipelines are what you execute or run They define workflows: what to do in which order
- 21. What are Activities? Activities are individual steps inside pipelines They can be chained or run in parallel
- 22. Copy Data Activity: Powerful Capabilities Source Destination
- 23. Copy Data Activity: Complex Data Source Convert File Formats Convert Data Types Map Columns Flatten Hierarchies Split & Merge Files Compress & Decompress Destination
- 24. What are Connections? Connections define how to connect and authorize Choose between built-in connection to workspace or 30+ external sources
- 25. What are Runs? Runs are individual executions of pipelines Run on-demand or on a defined schedule
- 26. Demo: Ingest and Orchestrate using Data Pipelines
- 27. Decision Guide for Ingest and Transform Copy Activity Dataflow Use cases: Data Ingestion Data Conversion Data Ingestion Data Transformation User interface: Wizard, Canvas Power Query Sources: 30+ connectors 180+ connectors Destinations: 18+ connectors Workspace: Lakehouse / Warehouse External: Azure SQL Database, Azure Data Explorer, Azure Synapse Analytics Transformations: Lightweight conversions 300+ transformation functions
- 29. Solving for Sensitive Data in Microsoft Fabric Enabling analysts to innovate at speed Power BI Run Secure Analytics Fabric Data Factory Delphix Data Masking API Delphix Data Discovery API 170+ Data Sources
- 30. UAT BACKUP AI/ML BI / ANALYTICS QUALITY ASSURANCE DEVELOPMENT REPORTING DWH INTEGRATION TESTING SANDBOX PERF TESTING DATA SCIENCE FUNCTIONAL TESTING DATA LAKE
- 31. G D P R P D P L L G P D D P P A D P A P O P I A P I P L / C L S P I P A A P P I A P P S PRIVACY ACT 2020 C C P A H I P A A C O P P A G L B A C C P A
- 32. Sensitive Data Exposed in Data Factory 170+ Targets 170+ Sources Sensitive Production Data ! Sensitive data is extracted and loaded to target. Fabric Data Factory Non-compliant Data Poses Risks ! Unprotected Pipeline / Dataflow US SSN: CCPA FIRST NAME: GDPR, LGPD, CCPA. DISEASE: HIPAA Analyst gets access to sensitive data
- 33. Compliant Data Delivered in Data Factory 170+ Targets 170+ Sources Sensitive Production Data ! Compliant Data Sensitive data is extracted and loaded to target. D I S C O V E R Y AUTOMATED SENSITIVE DATA DISCOVERY Fabric Data Factory M A S K REFERENTIAL INTEGRITY L O A D Timely access to quality compliant data D I S C O V E R AUTOMATED SENSITIVE DATA DISCOVERY â—Ź Data Sampling â—Ź Metadata Match â—Ź Pattern Match â—Ź Data Type â—Ź List Checks Employees SSN 324-23-1920 450-22-3204 529-003-2314 â—Ź Name Match â—Ź Nine Digit Match â—Ź No Entries Over/ Under 9 Digits â—Ź Data Type: VARCHAR â—Ź Occasional Delimiter M A S K REFERENTIAL INTEGRITY Siloed Production Data Secure Non-Production Data Employees last_name Haas Silverman Yang Employees last_name Lee Rogers Johnson Payroll SSN 291-24-5523 450-22-3204 529-03-2314 Payroll SSN 324-23-1920 992-35-6523 857-42-4367 Orders SSN last_name 291-24-5523 Haas 450-22-3204 Yang 857-432-4367 Silverman Employee_Details SSN last_name 324-23-1920 Lee 992-35-6523 Johnson 529-003-2314 Rogers
- 35. Questions?
- 36. Getting Started with Data Factory Speakers: Shireen Bahadur, Cathrine Wilhelmsen (MVP) TUESDAY @ 11:30am Using Azure AI Services with Data Factory Speakers: Abhishek Narain, Joroen Luitwieler THURSDAY @ 1:30pm Connecting to the World’s data using Data Factory Speakers: Matt Masson, Miguel Escobar, Jianlei Shen TUESDAY @ 11:30am From Data to Decisions: Leveraging Microsoft 365 with Data Factory Speakers: Wilson Lee, Karan Shah, Rishi Girish TUESDAY @ 3:15pm Empowering Self-service BI on SAP Data with Microsoft Fabric Speakers: Abhishek Narain, Joroen Luitwieler THURSDAY @ 2:45pm Performance Tuning Secrets for Data Factory Speakers: Sid Jayadevan, Mark Kromer, Matt Masson WEDNESDAY @ 11:15am Implement Enterprise Data Integration Patterns with Data Factory Speakers: Abhishek Narain, Miquella de Boer, Noelle Li WEDNESDAY @ 1:45pm Upgrade Pathways and Best Practices for Data Factory Speakers: Mark Kromer, Miguel Escobar, Mike Carlo (MVP) THURSDAY @ 11am Behind the Design: Crafting Data Factory Experiences in Fabric Speakers: Cristin Ford, Arian Martinez, Vichita Jianjitlert TUESDAY @ 2pm Modern Data Integration with Microsoft Fabric Data Factory Speakers: Wee Hyong, Shabnam Watson (MVP), Penny Zhou WEDNESDAY @ 8am Data Factory in Microsoft Fabric Technical Deep Dive Speakers: Mohan Sankaran, John Welch, Erwin de Kreuk (MVP) WEDNESDAY @ 9:15am WEDNESDAY THURSDAY Customer Stories: Data Integration Speakers: Andre Fomin, Tom Peplow WEDNESDAY @ 8am TUESDAY
- 37. s GENERAL AVAILABILITY PREVIEW SNEAK PEAK Modern Get Data – browse Azure Connections Dataflow output destinations – Support for schema changes for Lakehouse & Azure SQL DB Incremental Refresh for Dataflows Data Pipelines access on- premises data using “On Premises Data Gateway” (OPDG) Fast Copy for Dataflows 40 to 80 activity limit in Data Pipelines Semantic Model Refresh CI/CD in Data Pipelines Cancel Dataflow Refresh SPN support for VNET Data Gateway VNET Data Gateway support with Private Links for Dataflows Gen 2 in Fabric
- 38. Microsoft Fabric Community Conference 2024 • Shireen Bahadur, Cathrine Wilhelmsen & Tess Kassier Interested in Connecting with the Product Group? Please email us at [email protected] for any questions! Community Lounge Meet Ups Check Whova for official meetups with user group leaders, MVPs, Super Users and more! Meet Speakers & the Product Group Check Whova for the full schedule of speaker Q&A and PG meet & greets in the Community Lounge. aka.ms/FabricCommunity Ask and answer questions in the Fabric Community forum aka.ms/FabricUserGroups Find a user group in your area or to match your interests
- 39. Thank you! Shireen Bahadur Product Manager, Microsoft Azure Data Cathrine Wilhelmsen Lead Consultant, Evidi Tess Kassier Senior Product Manager, Delphix