5. iED Cloud Services.pdf
- 1. © 2018 iED 1 INTEGRATE. ENABLE. DRIVE.
- 2. © 2018 iED 2 Agenda Industry Challenges Reusable Assets 1 2 4 5 6 Solution Architecture 7 8 3 Platform Supported DevOps Value Proposition iED
- 3. © 2018 iED 3 Strategize, Assess & Plan Design, Build & Validate Run & Optimize CLOUD OPERATIONS SERVICES DATA LAKE CONSULTING SERVICES DATA MIGRATION, DATA LAKE DESIGN & BUILD iED | Your Trusted Technology Partner • Cloud strategy • Fit Gap Assessment • Migration Strategy • Data Lake & Analytical Roadmap • Data Lake Design & Build on Cloud • Data Migration, Transformation, & Validation • Data Security Implementation • Data Management & Monitoring • Service Management • Security and Identity Management • Capacity Planning and Optimization • Performance and Troubleshooting iED was founded after several decades of the learning by industry leaders to create production grade data lakes that accelerate advanced analytics and machine learning to play a critical role in transforming enterprises.
- 4. © 2018 iED 4 Industry Challenges & Highlights Data certification Inadequate “CloudFirst” focus Rapidly evolving technology choices Bringing agility and scalability Fit-gap analysis leading to technology selection Rule driven data quality engine • Pre- orchestrated ingestion, storage and consumption engine • Automated monitoring and scheduling • Configurable, functional, and business rules • Functional and technical validation • Layer based balancing • Responsive data architecture • Intelligent data management • Delivery at scale Current Challenges Our Solution Key Highlights • Searchable meta data and lineage • Enterprise semantic layer • Domains: Multiple Self service * Time to value Pre-built domain taxonomy Automation and asset based implementation • Integration Framework • Audit, Balancing & Control • Data lineage * • Continuous integration/deployment * Future road map
- 5. © 2018 iED About VISVA Quickly and easily set up massive ingestion with an intuitive GUI Metadata driven data ingestion Maximize throughput using parallel threading Web-based control features including restart-ability, throughput monitoring and notifications Ingestion Validation Transformation Consumption MD5 Checksum based validations Multi-level data validation report (row-count, row-level, cell level) Optimized data comparator using custom code (spark, native) Metadata validation Data Profiling and Standardization Dynamic creation of Logical views Point In Time view of source data Support for multiple file formats – AVRO, ORC, etc. Analytical data sets Augmented Data Discovery Enterprise semantic layer Data Visualization and self service 5 “VISVA is iED’s Cloud Data platform that facilitates modernization of enterprise Data Assets to maximize performance and reduce cost. VISVA frameworks brings 4x performance improvement and more than 60% effort savings.”
- 6. © 2018 iED VISVA | Implementation 6 Platform supported Compute Storage Sources HDInsight Hadoop Data Lake Store Mainframe Databricks SQL Database File Systems Data Lake Analytics SQL Data Warehouse RDBMS Native Spark Blob Cloud Azure Batch Analysis Services Web Hadoop Salesforce Packages Basic (option to select one of the four below) Advanced Premium Ingestion Transformation Validation Consumption 70% Automated Data Ingestion 50% Faster Time to Deployment 60% Automated Schema Generation 99% Data Accuracy
- 7. © 2018 iED 7 Solution Architecture
- 8. © 2018 iED 8 Data Accquisiton Perform Data Standardization to create staging layer Landing Layer Authentication, Authorization, Accounting & Data Protection Data Processing Layer AES-256 Consumption Layer Source Layer Appliance RDBMS Flat Files Audit Balance and Control Framework Monitoring and Administration Active Copy Layer Staging Layer Batch Data Consumption Azure Data Lake Store Azure Active Directory Load Curated data to ADW using Polybase Azure Monitor Express Route AES-256 AES-256 Azure Data Lake Store Azure Data Lake Store Azure Data Lake Store AES-256 Curation Layer Create point in time view of data in active copy layer Apply ETL transformation to created curated layer Web Real Time Express Route ADF V2 Integration Framework 1 2 3 4 5 6 7 Data Comparator Tool Azure SQL DB Azure SQL Data Warehouse 8 VISVA | Solution Architecture Excel BI Power BI SSRS Data Zen SharePoint BI Visualization
- 9. © 2018 iED 9 VISVA | Architecture Highlights • Data Integration Framework for extracting and loading large volume of data from a variety of data sources onto Azure/AWS • Dynamic Metadata Synchronization between On-premise and Azure/AWS • Audit, Balance and Control - framework is capable of providing detailed Audit Reports including process ran, success rates and failures along with functional/business rule validation • Monitoring & Scheduling - The Monitoring Workbench provides the facility to automate the workflows involved in the enterprise operational process through its scheduler
- 10. © 2018 iED 10 Reusable Assets
- 11. © 2018 iED 11 Azure WebApp - WebJob 1 CDI Framework Configuration Database Azure WebJob to execute the Integration Framework Executable Integration Framework reads the Configuration Database for Data movement and Orchestration configurations 2 3 Integration Framework generates the ADF V2 pipelines for Data Movement and Orchestration activities Data Movement ADF V2 Pipelines Workflow orchestration activity Transform Data in Active Copy Zone using Databricks activity Datahub Landing Zone ADLS On-Premise Sources Netezza, DB2, SQL Server, Flat Files, Web Datahub Staging Zone ADLS Datahub Active Copy Zone ADLS a b c Provisioning of Sources in Azure Data Lake Store 4 Apply ETL transformation and build data marts 5 Databricks Spark Activity Datahub Curation Zone ADLS Datahub Consumption Zone 6 a b Polybase Activity c VISVA | Integration Framework Metadata Driven Integration Framework to Provision a Data Source from On-Premise to Azure/AWS
- 12. © 2018 iED 12 Asp .NET Azure Web App Custom Scheduler Azure Automation Runbook Runbook polls to check if schedule has arrived for a workflow to execute Custom Job Status Portal to monitor the status of workflow Azure Data Factory V2 ZEA Pipeline Runbook triggers the ADF V2 pipelines Validate the workflow schedule details from Schedule DB Workflow Schedule Table Workflow Status Table ADF Pipeline Status Table ADF Activity Status Table Production Support Team Support team generates Reports/metric by querying operational Audit tables Customer reviews Status report Support team publish health report to customer 1 2 3 4 ADF V2 pipeline executes workflow and updates the operational audit details in Audit Database Operational Audit Database 7 8 9 Functional Rule Database Transformed and prepared data is validated against functional rules defined in Rule database Functional Reconciliation Framework 5 6 Functional Reconciliation activity is triggered post workflow execution VISVA | Audit Balance & Control Automated Configurable ABC Framework which governs and certifies the data across the ecosystem
- 13. © 2018 iED 13 Integration framework reads the config. database and generates data lineage pipelines Data Engineers ETL workflow captured in SQL DB template Integration Framework configuration database SQL DB project template in Visual Studio ADF V2 pipelines are deployed in to Data Factory VSTS GIT VSTS CI/CD Pipeline Visual Studio Team Services Azure Data Factory V2 Pipeline CI/CD process deploys the SQL DB project changes to config. database 1 6 5 4 3 2 Integration Framework ETL workflow SQL DB project checked-in to VSTS GIT Azure Cosmos Graph DB ADF V2 pipelines updates the Graph DB with lineage details 7 Data Lineage Data Lineage captured through custom API or enterprise lineage tools VISVA | Data Lineage (Future Roadmap)
- 14. © 2018 iED 14 VISVA Demo
- 15. © 2018 iED 15 VISVA | UI
- 17. © 2018 iED 17 Plan + Track Release Defect Management Track Progress Manage work Plan work Project Starts Automation functional testing environment Automation Integration testing environment Pre-production Environment Staging environment Track defect Manage defect Defect matrix Build Version control Unit testing Write code Develop + Test Report status Defect Management Track Defects, tasks, and blocking issues Project Management Project plan by creating a backlog of user stories that represent the work Release Automation Automated release pipelines in which you can reliably test and release on much shorter cycles. Development Develop, unit test and build automation VISVA |Continuous Integration/Deployment