Microsoft Ignite AU 2017 - Orchestrating Big Data Pipelines with Azure Data Factory
- 1. DA332 Lace Lofranco Orchestrating Big Data Pipelines with Azure Data Factory
- 2. Session Objective Learn to design, build and manage big data orchestration pipelines using Azure Data Factory
- 3. Agenda
- 7. Types of analytics What happened? Why did it happen? Top-down (Deductive) Bottom-up (Inductive)
- 8. Big Data Pipelines Examples Velocity, Variety, Volume
- 9. Big Data Pipelines Ingest Analyze PublishData Sources Data Lake
- 10. Lamda Architecture Data architecture for designing Big Data applications Three layers: Batch, Speed, Serving Popularized by Nathan Marz
- 12. Lamda Architecture HDInsight IoT Hub Event Hubs Data Factory Storm Streaming PowerBiSQL databaseHDInsight (LLAP/Spark)
- 13. Lamda Architecture HDInsight Data Factory PowerBiSQL databaseHDInsight (LLAP/Spark) SPEED LAYER Storm Streaming
- 14. Predicting tram load based on foot traffic Data sources: https://data.melbourne.vic.gov.au https://www.ptv.vic.gov.au
- 15. Sensor locations
- 16. Sensor locations
- 17. Solution Architecture Data Factory HDInsight Machine Learning Storage blobBatch HTTP data.melbourne .vic.gov.au PowerBI Data Lake SQL DW
- 19. Data Factory Concepts Data Store
- 20. Data Factory Concepts Data Store Linked Service
- 21. Data Factory Concepts Data Store Linked Service Dataset
- 22. Data Factory Concepts Data Store Linked Service Dataset Activity
- 23. Data Factory Concepts Data Store Linked Service Dataset Dataset Data Store Activity Linked Service
- 24. Data Factory Concepts Data Store Linked Service Dataset Dataset Data Store Activity Linked Service
- 25. Scheduling and Execution Sink Dataset Sink Dataset
- 26. Scheduling and Execution Source Dataset Sink Dataset Source Dataset Sink Dataset
- 28. Data Factory HDInsight Machine Learning Storage blobBatch HTTP data.melbourne .vic.gov.au PowerBI Data Lake SQL DW Demo: Copy Activity
- 29. Data Factory Design Considerations • Ideal for time series data • Design repeatable activity windows • Handle ‘late’ / out-of-order runs • Try to finalize pipeline schedules in advance • Currently no first class support for on-demand / event driven pipelines and control flow activities
- 30. Agenda Monitor pipeline health Developer tools Data Factory vs OozieLamda Architecture Intro to Data Factory Considerations
- 32. Data Movement Activity: Copy Cloud to Cloud On-premises to Cloud
- 33. Data Management Gateway Client agent installed on-premises environment to copy data between cloud and on-premises data stores
- 34. Category Data store Source Sink File File System* ✔ ✔ HDFS* ✔ Amazon S3 ✔ FTP ✔ Others Salesforce ✔ Generic ODBC* ✔ Generic OData ✔ Web Table (table from HTML) ✔ GE Historian* ✔ Category Data store Source Sink Azure Azure Blob storage ✔ ✔ Azure Data Lake Store ✔ ✔ Azure SQL Database ✔ ✔ Azure SQL Data Warehouse ✔ ✔ Azure Table storage ✔ ✔ Azure DocumentDB ✔ ✔ Azure Search Index ✔ ✔ Databases SQL Server* ✔ ✔ Oracle* ✔ ✔ MySQL* ✔ DB2* ✔ Teradata* ✔ PostgreSQL* ✔ Sybase* ✔ Cassandra* ✔ MongoDB* ✔ Amazon Redshift ✔ Supported Data Stores Data stores with * can be on-premises or on Azure IaaS, and require you to install Data Management Gateway on an on-premises/Azure IaaS machine.
- 35. Copy Wizard
- 36. Considerations • Copy service typically runs at region closest to sink • Performance & Tuning
- 39. Stored Procedure Activity Azure SQL Database Azure SQL Datawarehouse Use Polybase for loading large datasets SQL Server Database (on-premises or IAAS) Needs Data Management Gateway
- 40. Polybase and CTAS Polybase to access non-relational sources ‘CREATE TABLE AS’ is fully parallelized operation to create and load tables from a SELECT statements Super-charge SELECT INTO statement
- 41. Data Factory HDInsight Machine Learning Storage blobBatch HTTP data.melbourne .vic.gov.au PowerBI Data Lake SQL DW Demo: Stored Procedure Activity
- 44. Hive Activity Advanced Properties Property Hadoop config coreConfiguration core-site.xml hBaseConfiguration hbase-site.xml hdfsConfiguration hdfs-site.xml hiveConfiguration hive-site.xml mapReduceConfiguration mapred-site.xml oozieConfiguration oozie-site.xml stormConfiguration storm-site.xml yarnConfiguration yarn-site.xml
- 45. Data Factory HDInsight Machine Learning Storage blobBatch HTTP data.melbourne .vic.gov.au PowerBI Data Lake SQL DW Demo: Hive Activity
- 47. AzureML Activities Batch Execution Activity Scoring datasets Retraining models Update Resource Activity Updating ML models
- 48. Batch Execution Activity 1. Use Web service inputs/outputs 2. Use AzureML import/export modules
- 49. Retraining your model Machine Learning Predictions .iLearner file Updated Scoring web service Training web service Data for scoring Data for training Batch Execution Service Produces Update Trained Model Publish web sevices Produces Batch Execution Service
- 50. Retraining your model Machine Learning Predictions .iLearner file Updated Scoring web service Training web service Data for scoring Data for training Batch Execution Service Produces Update Trained Model Publish web sevices Produces Batch Execution Service Batch Execution Activity
- 51. Retraining your model Machine Learning Predictions .iLearner file Updated Scoring web service Training web service Data for scoring Data for training Batch Execution Service Produces Update Trained Model Publish web sevices Produces Batch Execution Service Update Resource Activity
- 52. Data Factory HDInsight Machine Learning Storage blobBatch HTTP data.melbourne .vic.gov.au PowerBI Data Lake SQL DW Demo: AzureML Batch Scoring Activity
- 54. .NET Custom Activity Use if data source/sinks not supported by ADF Compute: Azure Batch or HDInsight
- 55. public class ApiDownloadActivity : IDotNetActivity { IDictionary<string, string> IEnumerable<LinkedService> IEnumerable<Dataset> Activity IActivityLogger //your code }
- 56. .NET Custom Activity on Batch Slice 1 Slice 2 Slice 4 Slice 3 Slice 5 Activity Run Activity Run Activity Run Activity Run Activity Run Task 1 Task 2 Task 5 Task 3 Task 4 Compute node Compute node Data Factory Batch
- 57. Demo: Custom Activity on Batch Data Factory HDInsight Machine Learning Storage blobBatch HTTP data.melbourne .vic.gov.au PowerBI Data Lake SQL DW
- 62. Agenda Data Movement Data Transformation Data Factory vs OozieBig data pipelines Lamda Architecture Data Factory Concepts
- 64. Monitor and Manage Dashboard Pause and resuming pipelines Creating alerts Re-running failed pipelines
- 66. Developer tools Azure Portal VS Data Factory Extension PowerShell Azure Resource Manager Templates C# SDK
- 67. Agenda Data Movement Data Transformation Monitor pipeline health Developer tools Big data pipelines Lamda Architecture Data Factory Concepts
- 69. What is Oozie? Oozie is a workflow scheduler system to manage Apache Hadoop jobs De-facto workflow scheduler of the Hadoop stack Out-of-the-box integration with Hadoop jobs Java map-reduce, Streaming map-reduce, Pig, Hive, Sqoop and shell
- 71. Data Factory vs Oozie Data Factory Oozie Sources and Sinks Azure Hadoop Stack Numerous 3rd party Hadoop Stack Relational through Sqoop Shell, Distcp Hybrid pipelines Data Management Gateway None Tooling Visual Studio, Portal, PS Hue, Eclipse plugin, 3rd party Extensibility C# Custom Activity Java Action Node Control flow Limited Fork and Join, Decision Control, Kill (on error) Event-based No first class support. Some workarounds Dataset polling Performance Designed for big data workloads Designed for big data workloads Maturity Preview in Oct 2014, GA in August 2015 Developed since 2008, Open sourced since 2010
- 72. Data Factory vs Oozie Use Oozie when… Pipelines are exclusively on the Hadoop stack Use Data Factory when… Pipelines use Azure services along with the Hadoop stack
- 73. Overall Summary
- 74. Complete your session evaluation on MyIgnite for your chance to WIN one of many daily prizes. (image of prizes tbc) Session evaluation
- 75. Visit Channel 9 to access a wide range of Microsoft training and event recordings https://channel9.msdn.com/ Head to the TechNet Eval Centre to download trials of the latest Microsoft products http://Microsoft.com/en-us/evalcenter/ Visit Microsoft Virtual Academy for free online training visit https://www.microsoftvirtualacademy.com Continue your Ignite learning path
- 76. Microsoft Ignite