EPF-datagov-part1-1.pdf
- 1. Data governance Dr, Cédrine Madera Executive Information Architect
- 2. Introduction • Who I’m • Course objectives • Introduce data governance principles, challenges and impacts • Share enterprise use cases • Maximize interactions with the students • Your expectations
- 3. Data governance program • Part 1 : Introduction to data governance • Why data governance discussion today : the enterprise challenges • Data governance principles • Introduction to a data governance framework • Part 2 : Data governance strategy impacts on enterprise organization • Organization – process – roles • Data steward • Chief Data Officer • Part 3 : Data governance in action • From dataOps to data observability • Tools overview • Use cases • Practise data governance : maturity assessment session
- 4. Part 1.1 Introduction to data governance Why data governance discussion today : the enterprise challenges
- 5. Information Data Report Analyze Dashboard Transaction and application data Machine, Log sensor data Enterprise content Image, geospatial, video Social data Third-party data Open data Data Integr ation Data Organi zation busine ss data Archive Predictive ETL ETL ETL Traditional information driven Extract Transform and Load To control To implement business rules Data Mart Data Ware house Operat ional data store Business User Analytics (Data Scientist/Eng) Access to ALL data Sandbox area for experimentation What if analysis Build new models CDO IT/Operations Risk Compliance Complexity Cost Who sees what? What to keep? Where? Meet SLAs Leverage new tech
- 6. Information Data Transaction and application data Machine, Log sensor data Enterprise content Image, geospatial, video Social data Third-party data Open data Business User Analytics (Data Scientist/Eng) CDO IT/Operations Traditional information driven C a t a l o g u e ID 1,2,3 Name 1,2 Addr 1,2 X-REF Privacy Cust Segmentation ID Name Addr SPOUSE ID 2 Name 2 Addr 1 ID 1 Name 1 Addr 1 ID 3 Name 2 Addr 2 ID 4 Name 3 Addr 2 Same Cust? Which Add? Rel? MDM Keep governance
- 7. Information Data Transaction and application data Machine, Log sensor data Enterprise content Image, geospatial, video Social data Third-party data Open data Business User Analytics (Data Scientist/Eng) CDO IT/Operations Traditional information driven C a t a l o g u e MDM Keep governance Data Lake Data sources driven Access to ALL data Sandbox area for experimentation
- 8. Information Data Transaction and application data Machine, Log sensor data Enterprise content Image, geospatial, video Social data Third-party data Open data Business User Analytics (Data Scientist/Eng) CDO IT/Operations Traditional information driven Keep – Data - governance Data Lake Data sources driven Data warehouse Data Lake house Data Lab Sandbox
- 9. What has changed in data? • Data Governance – Data, Data, Everywhere Yet only 15% of organizations have the capability to leverage data and advanced analytics across their organization. The advent of AI The re-invention of the world in code A world awash with data
- 10. Business expectations : Align Data and AI to the desired speed of the business • Data Governance – Data, Data, Everywhere Data at rest Terabytes to Zettabytes of data to process Data in motion Event-based data, streaming data, milliseconds to seconds to respond Data in many forms Structured, unstructured, text, documents, images, speech, video Data in doubt Uncertainty due to data inconsistency & incompleteness, ambiguities, latency, deception, model approximations • To accommodate the specialized data needs of the business, and individuals within its organizations. • Gone are the days of a single, structured, data-at-rest architecture.
- 11. 11 Organizations lack correct and accessible data for efficient data analysis projects and scalable data-based solutions. Missing organizational structures. Disconnect between those who create and those who analyze data. Legacy architecture with point-to-point connections. Optimizing data sources for one analysis might impact other analyses. Poor data quality and standards. Manual steps required for data collection and preparation. Distributed data sources or data swamps. Often results in error-prone workarounds based on insufficient data. Problem Statement
- 12. Data Governance – The Data Science Perspective Data Scientists spend too much time preparing data Reclaim 79% of wasted time with effective Data Governance • Less Data Scientist time spent manipulating data/searching for truth • More Data Scientist time wringing value out of data 79% of their time is spent preparing data 13% of their time is spent extracting value from data vs
- 13. What we are hearing now… • Data Governance – Data Problems Maturity • I don’t trust my Data • We can’t find our Data • No one is responsible for the Data • We have Data we aren’t using • We never get rid of Data • How are organizations managing data. • What are the journeys. Strategy •No one knows what’s the system of record for this •Each division has its own way of storing and reporting •Too much emphasis on Excel and MS Access •We have no training plan for Analytics •Importance of Metadata is not recognized •We don’t know who owns the Data •What do we do with regulations? Quality • We don’t know Technical vs. Business Data Quality • Our CxO thinks we have Data Quality issues • Our Data Governance program needs automate Data correction • Our Auditors have questioned our Data Quality • Garbage In – Garbage Metadata •We don’t know Technical vs. Business Data Quality •Our CxO thinks we don’t understand what our data means •Our Data Governance program needs a place to create “gold source definitions” •Our Auditors have questioned where our data comes from and how it is processed Monetization •We have Data we should sell •There are no other (or few) products like this in the marketplace •We have the platforms or capabilities to provide these services •We have more specialized Data than anyone else in this space •Departments all believe that their Data is best •We should be able to use our Data make better decisions
- 14. Data need to be managed…. In order …… to stop data chaos to enable data assets to create data driven insights The objectives of Data governance…
- 15. What is Data Governance? Data Governance is the orchestration of people, process and technology to enable an organization to leverage data as an enterprise asset DATA GOVERNANCE Executive-Level Data Governance Bodies Line of Business Stewardship Community Data Quality Reporting Team Project Teams Virtual Teams Executive Sponsorship Risk Data Council Data Governance Program Manager Technical Liaisons (4) Business Liaisons (4) Metadata Liaison (1) Data Governance PMA Risk Data Governance Office (DGO) Data Quality Reporting Liaison (1) Data Definition Stewardship Function Data Production Stewardship Function Data Usage Stewardship Function Quality Measurement Stewardship Function Lead Steward Executive-Level Data Governance Bodies Line of Business Stewardship Community Data Quality Reporting Team Project Teams Virtual Teams Executive Sponsorship Risk Data Council Data Governance Program Manager Technical Liaisons (4) Business Liaisons (4) Metadata Liaison (1) Data Governance PMA Risk Data Governance Office (DGO) Data Quality Reporting Liaison (1) Data Definition Stewardship Function Data Production Stewardship Function Data Usage Stewardship Function Quality Measurement Stewardship Function Lead Steward 3. Identify Domain SMEs and Stakeholders 8. Mentor Stewards 6. Recognize Data Definer, User, and Producer Stewards 2.1.1 Build Team Data Quality Team- Business & Data Analysts Data Domain Steward LOB/Functional Area Data Steward Coordinator Data Steward Committee Data Governance Council 4. Identify SMEs for Applications Resource Checklist Template Process Manual 5. Note Potential Data Stewards During Domain Definition Resource Checklist Template Steady State: DG Council and Data Steward Committee* are Established 9. Mobilize Data Stewards 2.1.2.7 End 1. Select Data Domain Steward Resource Checklist Template 2. Map Data Domain to Lines of Business * If Data StewardCommittee is not yet Established, LOB Coordinators (who will eventually be on it) will serve this function 7. On-Board Stewards Resource Checklist Template 2.1.2 Build Common Definition (Continued) LOB/ Functional Area Data Steward Coordinators Data Quality Team- Modeler Data Quality Team- Business Analyst Data Domain Steward No Yes 16. Update Glossary of Terms with CLDM Terms 2.1.3 14. Validate Data Element List and Conceptual Model 8. Conduct Subject Area-Focused JAD Session 11. Update Conceptual Data Model 10. Document Data Standards & Rules Findings 2.1.2.4 17. Validate Glossary of Terms Glossary of Terms Template DQ Rules & Standards Template 9. Document Business Definition Findings Glossary of Terms Template DQ Rules & Standards Template Glossary of Terms Template 1.3.1 12. Update Subject Area List 15. Initiate CLDM Maintenance Process 2.1.2.7 JAD Session Guide 13. Have All Subject Areas Been Sufficiently Explored? 15. Initiate CLDM Maintenance Process 6. Create JAD Session Guide and Draft Element List 2.1.2 Build Common Definition LOB/ Functional Area Data Steward Coordinators Data Quality Team- Modeler Data Quality Team- Business Analyst Data Domain Steward 5. Obtain Participant Time JAD Session Guide 3.Validate Domain Scope Scope Summary Template 2.1.1.3 2. Determine Domain Boundaries 4. Create Conceptual Data Model Scope Summary Template Data Governance Council Initiates Domain Definition 2.1.2.8 CLDM 7. Prepare Pre- JAD Session Communications 1. Create Domain Boundaries Draft Subject Area List 2.1.2.11 7. Verify DQ Rules and Standards 4. Create Draft DQ Rules and Standards 2. Gather Information on Data Elements 8. Validate DQ Rules & Standards 1. Review Elements in DQ Rules & Standards 2.1.3 Build Data Quality Rules and Standards Data Quality Team- Business Analyst 9. Capture Dashboard/ Scorecard/Reporting Requirements/ Scope LOB/ Functional Area Data Steward Coordinators Data Domain Steward Data Quality Scorecard Team Yes No Yes No DQ Rules and Standards Template 6. Conduct Additional JAD Sessions or Meetings DQ Rules and Standards Template DQ Rules and Standards Template 10. Create Data Quality Dashboard Mock-Up 2.1.2.14 DQ Rules and Standards Template DQ Rules and Standards Template 1.3.1 2.3.2 DQ Rules and Standards Template 12. Mock-Up Meets Needs? 5. Is More SME Input Needed? 3. Determine Application Instances of Data Elements DQ Rules and Standards Template 11. Validate Dashboard Mock- Up DQ Rules and Standards Template People Process Technology The core objectives of a governance program are: § Guide information management decision-making § Ensure information is consistently defined and well understood § Increase the use and trust of data as an enterprise asset Extract Extract Extract Extract Extract Extract
- 16. Data Governance in a Nutshell Orchestration of people, organization, and technology that enables an organization to leverage data as an enterprise asset Organization Empowered by C-level management Top-down enforcement of data standards/ rules People Empowered to make decisions (based on information) Distributed throughout organization (roles might already exist) Technology Monitor data quality and standards in IT systems Business glossary to define and describe data and its usage Empower Decide Organize Head of DG/ CDO CxO DG Office DG Council Ops. IT R&D Finance ... Business Owner Data Steward Data/ IT Custodian Business processes and requirements Data and data quality requirements IT system and interface requirements Data Sources Data Target Data Repository*/ Data Lake* *Optional, but makes life much easier
- 17. Apply Data Governance to Data Issues Identify and empower people before expecting them to solve data issues (not vice versa) Business Value Use Cases Methods Enablers Data Sources Data Governance is introduced for prioritized use cases (unless you want to spend the next 2 years on “cleaning” your data) “As a product portfolio manager, I need standardized product master data, so I can compare product costs and margins.” 1 Someone in the organization has a data issue 3 Fully understand the reason for the as-is situation and why it’s a data issue 4 Mitigate data issue, define new data standards, update data definition, ... Don’t start here! 2a Identify affected data, business processes and IT systems 2c Make sure that roles are empowered and willing to make decisions Important! 2b Identify respective business owners, data stewards and data custodians Iterative process in beginning
- 18. How organizations interact • Data Governance is the forum where business and technical decisions are made that set the foundation for getting the right data into the right hands at the right time • Information Technology implements the decisions, creates the designs, manages the environment and implements the technologies that enable getting the data into the right hands at the right time. In a modern data organization this can include Data Engineers and Data Wranglers • Business Users are the hands and the minds that enter the data in a way that can be understood by other business users. In a modern data organization this can include Data Curators and Data Scientists • Data Governance – What is it? Business Users Information Technology Data Governance
- 19. The Excessive Cost of Bad Data • Data Governance – Data Value Proposition Data Governance creates Trusted and Higher Quality Data
- 20. Considerations of a Data Governance Strategy 20 A good Data Governance Strategy: ü Focuses on delivering value to the organization by developing a value proposition that maps Business Goals to the development of Information as a corporate asset ü Recognizes cultural change as a critical component of a good Data Governance Strategy ü Develops Data Governance policies to guide the organization with procedures customized to the business domains ü Defines the structure and roles of organizational elements which are key to defining, producing, and using high quality data supporting business initiatives ü Identifies tools as important enablers to profiling, defining, migrating, reporting, and managing processes and elements critical to Data Quality management ü Develops measurements to ensure standards of quality are defined, understood, reported, managed, and met ü Develops processes for proactively and reactively addressing Data Quality including prioritization approaches to identify specific problem areas or “hot spots” ü Provides scalable framework for Data Quality metrics and reporting ü Primes and instructs the enterprise for continuous process improvement to deliver sustained value delivery Data Governance – strategy
- 21. Data Governance Strategy Core Objectives Take away for this part 1.1 21 Data Governance is the orchestration of people, process, and technology to enable an organization to leverage data as an enterprise asset. The core objectives of a Data Governance Strategy are: –Build a sustaining Data Governance foundation including infrastructure, technology and supporting organization –Define processes and business rules for ongoing Data Governance –Develop common and standard data domain definitions –Monitor and improve data quality
- 23. Part 1.2 Introduction to data governance Data governance principles Next session