Big data and the data quality imperative
1 like2,598 views
The document discusses the challenges of big data and data quality. It defines big data as large volumes of data that are difficult to process using traditional database systems. Big data comes from various sources like social media, machines, and open data. It highlights that poor data quality will undermine the value of big data investments, and that data quality foundations are needed to build successful big data and analytics programs. Effective data integration, profiling, standardization, and governance are critical to addressing the data quality imperative of big data.
Big data and the data quality imperative
- 1. TRILLIUM SOFTWARE 2013 CUSTOMER CONFERENCE (Who’s Afraid of…) The Big Bad Data Wolf? The Big Bad Data Challenge – Big Data & the Data Quality Imperative Presented By: Nigel Turner VP Information Management Strategy 1
- 2. The tale of the Three Little Pigs 2© Copyright 2013, Trillium Software, Inc. All rights reserved.
- 3. Big Data – what is it? Set of new concepts, practices & technologies to manage & exploit digital data Can be defined as: “Data that exceeds the processing capability of conventional database systems. The data is too big, moves too fast, or doesn’t fit the strictures of your database architecture” (Source: Ed Dumbill – O’Reilly Community) Its key premise is that all data has potential value if it can be collected, analysed and used to generate actionable insight 3 3 3© Copyright 2013, Trillium Software, Inc. All rights reserved.
- 4. Where does Big Data come from? SOCIAL MEDIA & SOCIAL NETWORKS MACHINE GENERATED WIDELY KNOWN SOURCES 4 4 4 © Copyright 2013, Trillium Software, Inc. All rights reserved.
- 5. What’s different about Big Data? New technologies which enable distributed & highly scalable MPP (Massively Parallel Processing), e.g. Apache Hadoop MapReduce NoSQL databases Strong emphasis on analytical approaches Emergence of “data science” Predictive Analytics Data Mining The “democratisation” of data Data made available to all (cf Cloud Computing) Business and not IT led BI 5
- 6. Big Data & Data Quality – parallel worlds? 6 BIG DATA DATA QUALITY © Copyright 2013, Trillium Software, Inc. All rights reserved.
- 7. Parallel worlds… or are they (1)? 7 Shared with 100,000+ others and counting…
- 8. Parallel worlds… or are they (2)? 8 “ I spend the vast majority of my time cleaning data systems…cleaning and preparing data sets makes everything I do better … it’s the highest value activity I do” Josh Wills Senior Director of Data Science Cloudera (From “Training a new generation of Data Scientists” – Cloudera video)
- 9. When Big Data & Data Quality worlds collide… 9 Big Data will expose Data Quality shortcomings Poor Data Quality will undermine the value of Big Data investments
- 10. Big Data – building on solid foundations BIG DATA / ANALYTICS DATA QUALITY FOUNDATION 10
- 11. The 3Vs and the DQ challenge • Exponential growth of data – predicted 40-60% per annum • 2.5 quintillion bytes of data are created every day • 90% of all digital data created in the last two years • Data generated more varied and complex than before: – Text, Audio, Images, Machine Generated etc. • Much of this data is semi-structured or unstructured • Traditional IT techniques ill equipped to process & analyse it • Data often generated in real time • Analysis and response needs to be rapid, often also real time • Traditional BI / DW environments cannot cope – new approaches are needed 11 11
- 12. Big Data – Foundations of Success Identifying the right data to solve the business problem or opportunity The ability to integrate & match varied data from multiple data sources structured, semi-structured, unstructured Building the right IT infrastructure to support Big Data applications Having the right capabilities & skills to exploit the data 12 12
- 13. Big Data – some vertical applications Retail: using point of sale & social media data to supplement & enrich traditional CRM / Marketing data Insurance & Banking: fraud detection Health: holistic patient analysis Utilities: consumption peaks & troughs & capacity planning Telcos: call routing optimisation & customer churn Manufacturing: predictive fault identification & supply chain optimisation Research: particle analysis, genomics etc. 13
- 14. Example Big Data benefit: The Open Big Data Cloud 14 SOURCE: LINKED OPEN DATA (LOD) COMMUNITY
- 15. Big Data in practice - Volvo Every Volvo vehicle has hundreds of microprocessors / sensors Data generated used within the car itself but also captured for analysis by Volvo and its dealers All data is loaded into a centralised analysis hub & integrated with CRM, dealership, product & social network data Used to optimise design & manufacturing, enhance customer interaction, improve safety & act on customer feedback 15
- 16. Big Data – Barriers & Pitfalls The sheer volume of data – what’s worth using? Data extraction challenges The ability to match data from disparate sources / formats / media The time taken to integrate new data sources The risks of mismatching and incorrect identification of individuals Legal & regulatory pitfalls Security concerns – corporate & individual Lack of skills & expertise 16 16
- 17. Big Data – the data integration challenge SOCIAL MEDIA SENSORS OPEN DATA EMAIL MOBILES EXTERNALDATASOURCES INTERNALDATASOURCES CRM BILLING OPS SALES PRODS ANALYTICS PLATFORM 1 ANALYTICS PLATFORM 2 ANALYTICS PLATFORM 3 ANALYTICS PLATFORM n ACTIONABLE INSIGHT & KNOWLEDGE 17
- 18. Big Data – the Data Quality Imperative (1) Need to profile external and internal data sources Need to classify data to define what data really matters Need to assure the quality of internal (and some external) data sources for accuracy, completeness, consistency Need to define & apply business rules & metadata management to how the data will be defined and used Need for a data governance framework to ensure consistency & control 18
- 19. Big Data – the Data Quality Imperative (2) Need processes & tools to enable: Source data profiling Data integration Data parsing Data standardisation Business rule creation & management Metadata management & a shared business / IT glossary Data de-duplication Data normalisation Data matching Data enrichment Data audit Many of these functions must be capable of being carried out in real time with zero lag 19
- 20. Big Data – DQ as the key enabler SOCIAL MEDIA SENSOR S OPEN DATA EMAIL EXTERNALDATASOURCES INTERNALDATASOURCES CRM BILLING OPS SALES PRODS ANALYTICS PLATFORM 1 ANALYTICS PLATFORM 2 ANALYTICS PLATFORM 3 ANALYTICS PLATFORM n ACTIONABLE INSIGHT & KNOWLEDGE PROFILE PARSE STANDARDISE MATCH ENRICH DATA QUALITY PLATFORM PROFILE PARSE STANDARDISE MATCH ENRICH MOBILES 20
- 21. Big Data – some algorithms 1. BIG DATA + POOR DATA QUALITY = BIG PROBLEMS 2. DATA DEMOCRITISATION – DATA GOVERNANCE = ANARCHY 3. DATA MASH UPS – DATA QUALITY = DATA MESS 4. BIG DATA ANALYTICS + POOR DQ = WRONG RESULTS 5. BIG DATA – DATA ASSURANCE = JAIL 6. 3V + DATA QUALITY = 4V (VALIDITY) 21
- 22. Big Data & Data Quality – summary • Big Data will depend on data quality to reap its claimed benefits – the GIGO truism • The democratization of data will expose poor DQ • The need for Data Governance increases as data becomes more accessible • Data skills will become more valued for ‘data science’ • Big Data will increase the 3Vs of data • Control of data becomes more difficult – scope and variety of use increases • Data standards & business rules become more complex • Potential legal & regulatory minefield 22 22
- 23. What action should we take as data management / DQ professionals? Identify and get involved in any current or planned Big Data initiatives within our organisations Ensure that the Data Quality and Data Governance implications & imperatives of these initiatives are understood Plan for the new Data Quality and Data Governance challenges that these trends will pose 23 23
- 24. So who’s afraid of the Big Bad Data Wolf? 24
- 25. Questions (Who’s Afraid of…) The Big Bad Data Wolf? The big Bad Data challenge – Big Data & the Data Quality imperative 25