DataOps for the Modern Data Warehouse on Microsoft Azure @ NDCOslo 2020 - Lace Lofranco
- 6. LOB Graph CRM Social Relational Images Speech IoT INGEST EXPLORE PREP & TRAIN SERVE STORE Data Orchestration and Monitoring Query All Data Analytics Engine Data Warehouse Data Lake
- 7. LOB Graph CRM Social Relational Images Speech IoT INGEST EXPLORE PREP & TRAIN SERVE STORE Data Factory Databricks, HDInsight, Synapse Databricks, HDInsight, Synapse Synapse, Analysis Services, AzureSQL Azure Data Lake Gen2
- 11. Learnings
- 17. Validate BUG !
- 18. Learnings
- 20. BUG !
- 21. Replay Replay
- 22. Replay Replay
- 26. #RubDevOpsOnIt
- 27. People. Process. Products. DevOps is the union of people, process, and products to enable continuous delivery of value to your end users. “ ” Build & Test Continuous Delivery Deploy Operate Monitor & Learn Plan & Track Develop
- 28. DataOps is NOT Just DevOps for Data, Data Kitchen Data Data Pipeline Value
- 29. DataOps is NOT Just DevOps for Data, Data Kitchen Data Production Value
- 30. Cloud-hosted pipelines for Linux, Windows and macOS, with unlimited minutes for open source Azure Pipelines Multi language, platform, and cloud support Extensible Best-in-class for open source https://azure.com/pipelines➔ Containers and Kubernetes
- 31. Deliver value to your users faster using proven agile tools to plan, track, and discuss work across your teams. Build, test, and deploy with CI/CD that works with any language, platform, and cloud. Connect to GitHub or any other Git provider and deploy continuously. Get unlimited, cloud-hosted private Git repos and collaborate to build better code with pull requests and advanced file management. Test and ship with confidence using manual and exploratory testing tools. Create, host, and share packages with your team, and add artifacts to your CI/CD pipelines with a single click. Azure Boards Azure ReposAzure Pipelines Azure Test Plans Azure Artifacts https://azure.com/devops ➔
- 38. Working Branches (Dev Branches) Collaboration Branch (ei. master) GIT PULL REQUEST
- 39. Working Branches (Dev Branches) Collaboration Branch (ei. master) GIT PULL REQUEST PUBLISH AZURE
- 40. Working Branches (Dev Branches) Collaboration Branch (ei. master) Publish branch (adf_publish) GIT PULL REQUEST PUBLISH CREATED / UPDATED BY ADF AZURE
- 41. Working Branches (Dev Branches) Collaboration Branch (ei. master) Publish branch (adf_publish) Release Definition GIT PULL REQUEST PUBLISH CREATED / UPDATED BY ADF RELEASE MANAGEMENT AZURE ARTIFACT
- 42. Working Branches (Dev Branches) Collaboration Branch (ei. master) Publish branch (adf_publish) Release Definition Release GIT PULL REQUEST PUBLISH CREATED / UPDATED BY ADF RELEASE MANAGEMENT CREATE RELEASE AZURE ARTIFACT DEPLOY DEPLOY
- 43. Learnings
- 48. Run unit tests, linting, DACPAC build
- 51. Continuous Integration Continuous Delivery
- 53. Integration tests Continuous DeliveryContinuous Integration
- 55. Pipeline Stage Stage Stage Job Step: task Step: script Job Job trigger: - master variables: - foo: bar pool: vmImage: ubuntu-18.04 stages: - stage: jobs: - job: steps: - script: echo $foo ENV: Prod
- 57. Learnings
- 58. Metrics Logs Application Container VM Monitoring Solutions Insights Dashboards Views Power BI Workbooks Visualize Metrics Explorer Log Analytics Analyze Alerts Autoscale Respond Event Hubs Ingest & Export APIs Logic Apps Integrate Azure Monitor Custom Sources Application Operating System Azure Resources Azure Subscription Azure Tenant