"Implementing data quality automation with open source stack" - Max Martynov, CTO of Grid Dynamics
Download as PPTX, PDF0 likes438 views
The document discusses the implementation of real-time data quality enforcement using machine learning within cloud data lakes and analytics environments. It emphasizes the challenges of data corruption and the importance of trust in data for agile decision-making, outlining methods for data validation, anomaly detection, and governance. The proposed solutions aim to enhance data accuracy, mitigate risks, and promote data compliance across enterprise applications.
"Implementing data quality automation with open source stack" - Max Martynov, CTO of Grid Dynamics
- 1. 2 Data Quality Automatic enforcement in real-time with machine learning. Max Martynov, CTO
- 2. Introducing Grid Dynamics technology services Digital transformation Big data, real time analytics, ML & AI Microservices replatforming DevOps & cloud enablement Open Source Cloud-ready Scalable Automated
- 3. 12 years of experience in digital transformation.
- 5. 9.8 9.3 9.6 9.4 10.1 17.5 16.9 8.9 9.6 9.2 10.3 9.8 5.1 4.1 3.9 4 3.9 4.5 7.5 7.1 3.8 4.1 4 4.5 4.2 2.3 0 5 10 15 20 7/1/19 7/2/19 7/3/19 7/4/19 7/5/19 7/6/19 7/7/19 7/8/19 7/9/19 7/10/19 7/11/19 7/12/19 7/13/19 7/14/19 Retailer X, Daily Sales – Executive Summary Revenue, $M Gross Profit, $M weekend weekend
- 6. DBDBDB EDW
- 8. Cloud Data Lake EDW DBDBDB FileFileFile MQ CloudCloud AppAppAPI
- 9. Cloud EDW DBDBDB FileFileFile MQ CloudCloud AppAppAPI AppAppApp Data Lake
- 10. 1 0 1 1 0 1 1 0 1 0 1 0 1 0 0 1 1. Trust is hard to build and easy to lose. 2. Distrust in data slows down decisions. 3. Slow decisions prevent agility.
- 11. Data corruption reasons 1. Code 2. Data Sources 3. Infrastructure
- 12. Test environment Input Actual Expected ETL code compareTest suite run test Traditional approach to testing
- 13. Production data quality goals Detect data corruption Prevent it from spreading Alert support team
- 14. Production data lake Data quality enforcement in production DBDBDB FileFileFile MQ AppAppAPI data processing job
- 15. Production data lake DBDBDB FileFileFile MQ AppAppAPI data processing job data quality job Data quality enforcement in production
- 16. Production data lake DBDBDB FileFileFile MQ AppAppAPI data processing job data quality job alert & stop pipeline alert & continue x Data quality enforcement in production
- 17. Data Lake Data source data 1. Compare with SoR 2. Validate business rules 3. Data profiling and anomaly detection Main data processing pipeline confidence data confidence
- 18. 1. Control divergence from SoR Data Lake Data source Imported dataset Compare data in SoR and data lake 1. Validate correctness of import. 2. Prevent stale data. 3. Prevent corruption accumulation in stream processing use cases. 4. Check data before it gets in the lake.
- 19. 2. Validate business rules Data Lake Dataset Check for nulls and data ranges 1. Enforce schema. 2. Check for nulls. 3. Validate data ranges. 4. Specify and enforce data invariants.
- 20. 3. Anomaly detection Data Lake Dataset 1. Fully automatic data quality enforcement. 2. Collect data profile, metrics and statistics. 3. Train ML models. 4. Find anomalies in data. Data profiling and anomaly detection
- 21. Catalog Inventory Orders Data Lake Data Quality Reporting & Alerting Data Profile Demo setup
- 22. 23 Live demo
- 24. Anomaly detection example: zooming in
- 25. Capabilities for enterprise data quality and governance Enables widespread adoption. Enforces enterprise-level controls and data usage policies. Increases consistency and confidence in decision making. Decreases the risk of regulatory fines. Improves data security. Facilitates accountability for information quality. Minimizes or eliminates efforts duplication. Data Governance Platform Metadata Management Full-text Search Data Quality Status Schema / Summary Data Profiling Mapping to Glossary Change Log Dependency Detection Consumers Flow Visualization Glossary Portal Knowledge Base Fields Fingerprinting Data Catalog Dataset Profile Lineage Dashboard Data Glossary Data Quality Access and Security Business Rules Anomaly Detection Alerting Access Rules Compliance Policies Policy Engine
- 28. Dataproc cluster
- 29. Aifrlow pipeline
- 30. Griffin measures
- 31. Anomaly detection: normal data
- 33. Anomaly detection: return to normal
- 34. Anomaly detection: historical view
- 35. Anomaly detection (counts): anomaly & return
- 38. Uniqueness: return to normal
- 41. Nulls: anomaly
- 42. Nulls: return to normal