Dynamic Talks: "Implementing data quality automation with open source stack" -Max Martynov
Download as PPTX, PDF1 like699 views
The document discusses the implementation of real-time data quality enforcement using machine learning technologies at Grid Dynamics. It highlights the importance of trust in data for effective decision-making and outlines various methods for detecting and preventing data corruption. Additionally, it emphasizes the need for automated data profiling, anomaly detection, and governance to enhance data security and regulatory compliance.
Dynamic Talks: "Implementing data quality automation with open source stack" -Max Martynov
- 1. 2 Data Quality Automatic enforcement in real-time with machine learning. Max Martynov, VP of Technology
- 2. Introducing Grid Dynamics technology services Digital transformation Big data, real time analytics, ML & AI Microservices replatforming DevOps & cloud enablement Open Source Cloud-ready Scalable Automated
- 3. 12 years of experience in digital transformation.
- 5. 9.8 9.3 9.6 9.4 10.1 17.5 16.9 8.9 9.6 9.2 10.3 9.8 5.1 4.1 3.9 4 3.9 4.5 7.5 7.1 3.8 4.1 4 4.5 4.2 2.3 0 5 10 15 20 7/1/19 7/2/19 7/3/19 7/4/19 7/5/19 7/6/19 7/7/19 7/8/19 7/9/19 7/10/19 7/11/19 7/12/19 7/13/19 7/14/19 Retailer X, Daily Sales – Executive Summary Revenue, $M Gross Profit, $M weekend weekend
- 6. DBDBDB EDW
- 8. Cloud Data Lake EDW DBDBDB FileFileFile MQ CloudCloud AppAppAPI
- 9. Cloud EDW DBDBDB FileFileFile MQ CloudCloud AppAppAPI AppAppApp Data Lake
- 10. 1 0 1 1 0 1 1 0 1 0 1 0 1 0 0 1 1. Trust is hard to build and easy to lose. 2. Distrust in data slows down decisions. 3. Slow decisions prevent agility.
- 11. Data corruption reasons 1. Code 2. Data Sources 3. Infrastructure
- 12. Test environment Input Actual Expected ETL code compareTest suite run test Traditional approach to testing
- 13. Data quality goals Detect data corruption Prevent it from spreading Alert support team
- 14. Production data lake Real-time data quality enforcement DBDBDB FileFileFile MQ AppAppAPI data processing job
- 15. Production data lake Real-time data quality enforcement DBDBDB FileFileFile MQ AppAppAPI data processing job data quality job
- 16. Production data lake Real-time data quality enforcement DBDBDB FileFileFile MQ AppAppAPI data processing job data quality job alert & stop pipeline alert & continue x
- 17. Data Lake Data source data 1. Compare with SoR 2. Validate business rules 3. Data profiling and anomaly detection Main data processing pipeline confidence data confidence
- 18. 1. Control divergence from SoR Data Lake Data source Imported dataset Compare data in SoR and data lake 1. Validate correctness of import. 2. Prevent stale data. 3. Prevent corruption accumulation in stream processing use cases. 4. Check data before it gets in the lake.
- 19. 2. Validate business rules Data Lake Dataset Check for nulls and data ranges 1. Enforce schema. 2. Check for nulls. 3. Validate data ranges. 4. Specify and enforce data invariants.
- 20. 3. Anomaly detection Data Lake Dataset 1. Fully automatic data quality enforcement. 2. Collect data profile, metrics and statistics. 3. Train ML models. 4. Find anomalies in data. Data profiling and anomaly detection
- 21. Catalog Inventory Orders Data Lake Data Quality Reporting & Alerting Data Profile Demo setup
- 22. 23 Live demo
- 24. Anomaly detection example: zooming in
- 25. Capabilities for enterprise data quality and governance Enables widespread adoption. Enforces enterprise-level controls and data usage policies. Increases consistency and confidence in decision making. Decreases the risk of regulatory fines. Improves data security. Facilitates accountability for information quality. Minimizes or eliminates efforts duplication. Data Governance Platform Metadata Management Full-text Search Data Quality Status Schema / Summary Data Profiling Mapping to Glossary Change Log Dependency Detection Consumers Flow Visualization Glossary Portal Knowledge Base Fields Fingerprinting Data Catalog Dataset Profile Lineage Dashboard Data Glossary Data Quality Access and Security Business Rules Anomaly Detection Alerting Access Rules Compliance Policies Policy Engine
- 28. Dataproc cluster
- 29. Aifrlow pipeline
- 30. Griffin measures
- 31. Anomaly detection: normal data
- 33. Anomaly detection: return to normal
- 34. Anomaly detection: historical view
- 35. Anomaly detection (counts): anomaly & return
- 38. Uniqueness: return to normal
- 41. Nulls: anomaly
- 42. Nulls: return to normal