Transform Your Downstream Cloud Analytics with Data Quality
Download as PPTX, PDF0 likes123 views
The document outlines the significance of data quality in cloud analytics, emphasizing how poor data quality leads to stalled AI/ML projects and impacts corporate reputation. It discusses effective data profiling techniques, including establishing context, reviewing relevant measurements, and defining actionable rules to ensure data fit for purpose. Additionally, it highlights a case study of a European telecommunications company that improved its data governance to enhance decision-making and customer service.
Transform Your Downstream Cloud Analytics with Data Quality
- 1. Understand Your Data Transform your downstream cloud analytics with data quality Marco de Jong | Sales Engineering Director
- 2. What you will learn today • Why Data needs Data Quality • How Data Profiling helps understanding your data • The top 5 steps needed for effective data profiling • How another company saw success through data profiling • What you can do in the next 90 days to take action on DQ
- 3. Data needs data quality “Societal trust in business is arguably at an all-time low and, in a world increasingly driven by data and technology, reputations and brands are ever harder to protect.” EY “Trust in Data and Why it Matters”, 2017. 80% of AI/ML projects are stalling due to poor data quality Dimensional Research, 2019 90% of executives are concerned about how misused data can impact corporate reputation PWC, 22nd Annual Global CEO Survey, 2019 64% of IT executives have trouble finding and cleaning the right data for strategic data projects Sierra Venture, 2020 The importance of data quality in the enterprise: • Decision making • Customer centricity • Compliance • Machine learning & AI
- 4. Understanding your data Data Profiling • The set of analytical techniques that evaluate actual data content (vs. metadata) to provide a complete view of each data element in a data source. • Provides summarized inferences, and details of value and pattern frequencies to quickly gain data insights. • Business Rules • The data quality or validation rules that help ensure that data is “fit for use” in its intended operational and decision- making contexts. • Covers the accuracy, completeness, consistency, relevance, timeliness and validity of data.
- 5. Five Key Steps to effective Data Profiling These are not new, but good to reiterate 1. How do you want to analyze the data? 2. What should you review? (there's a lot of stuff) 3. What should you look for? (based on data “type”) 4. When should you build rules? (laser-focus; CDE’s) 5. What needs to be communicated?
- 6. 1. How do you want to analyze the data?
- 7. “Never lead with a data set; lead with a question.” Anthony Scriffignano, Chief Data Scientist, Dun & Bradstreet Forbes Insights, May 31, 2017, “The Data Differentiator”
- 8. Universal DQ best practices: Understand the End Goal • How does the business intend to use the data (i.e. what’s the use case)? • Empower users (“Who”) to gain new clarity into the core problem (“Why”) • What will the data be used for? • What defines the Fitness for your Purpose? Establish Scope • Ask the “right questions” about the use case and the data (not just “what” and “how”) • What data is relevant to the effort? • Big Data or other, you need to set boundaries for the work Understand Context • How does the business define the data? • What are the important characteristics and context of the data? • What are the Critical Data Elements? • What qualities will you need to address, or leave alone? • “High-quality data” definition will vary by business problem “If you don’t know what you want to get out of the data, how can you know what data you need – and what insight you’re looking for?” Wolf Ruzicka, Chairman of the Board at EastBanc Technologies, Blog post: June 1, 2017, “Grow A Data Tree Out Of The “Big Data” Swamp”
- 9. 2. What do you want to review?
- 10. Common data quality measurements What measures can we take advantage of? 1. Completeness – Are the relevant fields populated? 2. Integrity – Does the data maintain an internal structural integrity or a relational integrity across sources 3. Uniqueness – Are keys or records unique? 4. Validity – Does the data have the correct values? • Code and reference values • Valid ranges • Valid value combinations 5. Consistency – Is the data at consistent levels of aggregation or does it have consistent valid values over time? 6. Timeliness – Did the data arrive in a time period that makes it useful or usable?
- 11. New data quality problems New data, new data quality challenges • 3rd Party and external data with unknown provenance or relevance • Bias in the data – whether in collection, extraction, or other processing • Data without standardized structure or formatting • Continuously streaming data • Disjointed data (e.g. gaps in receipt) • Consistency and verification of data sources • Changes and transformation applied to data (i.e. does it really represent the original input)
- 12. Let data profiling guide you • Contextual visualizations • Value and pattern distributions • Attribute summaries and metadata • Sort and filter to quickly find data of interest • Detail drilldowns to any content
- 13. 3. What should you look for?
- 14. Common data types What do you need to be aware of? 1. Identifiers – data that uniquely identifies something 2. Indicators – data that flags a specific condition 3. Dates – data that identifies a point in time 4. Quantities – data that identifies an amount or value of something 5. Codes – data that segments other data 6. Text – data that describes or names something
- 15. 4. When do you build rules?
- 16. Build rules for defined conditions Focus on: • Critical Data Elements (data quality dimensions) • Policy-based conditions (e.g. regulatory compliance) • Correlated data conditions (e.g. If x, then y) • Filtering and segmenting data (refining evaluations; investigating root cause)
- 17. Benefits of business rules • Validate critical requirements within or across data sources • Build common rules that can be readily tested and shared • Evaluate and remediate issues • Take action on incorrect data and defaults • Create flags for subsequent use in marking or remediating data • Filter result sets and export for additional use
- 18. 5. What should you communicate?
- 19. Communicate! Culture of Data Literacy • “Democratization of Data” requires cultural support Program of Data Governance • Provide the processes and practices necessary for success Center of Excellence/Knowledge Base • Where do you go to find answers? • Who can help show you how?
- 20. Annotate results with findings
- 21. Large European Telco Leveraging data as a critical asset • Business Rules Goal • Ensure accurate data to support customer service, marketing, retention and loyalty • Implement enterprise-wide data governance Challenge • Data from multiple sources/systems, stored in many different formats • No enterprise standard for data quality • Moving to the cloud Benefits Achieved • Trusted data for faster, better strategic and operational decision making • More effective marketing and better customer service Solution • Precisely Trillium Discovery • Precisely Trillium Quality
- 22. Looking at the Next 90 Days… • Make profiling actionable (you don’t know what you don’t know until you profile) • Keep the 5 key questions top of mind! Visit us to learn more about Cloud Transformation: https://www.precisely.com/campaigns/cloud-transformation
- 23. QA