You Need a Data Catalog. Do You Know Why?
1 like•1,383 views
The document discusses the importance of a modern data catalog in addressing data management challenges such as governance, trust, and data quality. It outlines the features and benefits of a data catalog, including its role as a central repository for metadata and its integration with data governance to ensure compliance and accountability. The document emphasizes that effective data governance is essential for leveraging data effectively to meet business objectives.
You Need a Data Catalog. Do You Know Why?
- 1. Fern Halper, Ph.D. VP and Senior Director, TDWI Advanced Analytics March 2, 2022 You Need a Data Catalog. Do You Know Why? Copyright © 2022 TDWI
- 2. SPONSOR 2
- 3. FERN HALPER VP, Senior Research Director for Advanced Analytics TDWI
- 4. What we will talk about today • Data management challenges and priorities • The modern data catalog – what it is and why it is important • The role of the modern data catalog in your data quality and governance programs • The kinds of information that should be in your data catalog and why Copyright © 2022 TDWI
- 5. 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Video data Audio data Clickstream data Image still data Machine generated data (e.g., from sensors, devices) Real-time event streams External text data Geospatial data Semi-structured data Internal text data Time series data Log data Demographic data Transactional data Structured data What kind of data is your organization managing now? Looking to manage in the next year? Manage now Manage in next year Organizations are collecting diverse data for use in analytics Copyright © 2022 TDWI
- 6. And, they are running into challenges, especially around governance and trust • “It’s a mess” • “Finding the right data” • “Data variety and volumes” • “Metadata reconciliation, participation by stakeholders in data governance” • “Lack of built-in metadata within new Data Lakehouse platform relating to business and technical metadata around individual files/tables/elements/columns.” • “Dispersed data environment and lack of governance” • “So many disparate systems and sources of truth” • “Misunderstanding of the data” • “Understanding it. Do we have the SME's, or data lineage or metadata to really know what's there? So end users can leverage it, so technical professionals can support it, so QA analysts can test it.” Copyright © 2022 TDWI
- 7. Organizations want to see improvements in data environment Copyright © 2022 TDWI
- 8. Their priorities reflect the challenges Copyright © 2022 TDWI
- 9. Data Catalog: a central repository that contains metadata for describing data sets, how they are defined, and where to find them. Copyright © 2022 TDWI
- 10. Modern enterprise data catalogs offer new features AI infused • Search-based interface – For finding data – NL Query for basic stats questions • Automation- metadata extraction, tagging, cleansing, classification, access, syntax, lineage, etc. Collaboration/Crowdsourcing to provide rating and certification Copyright © 2022 TDWI
- 11. • Completeness • Accuracy • Integrity • Consistency • Data usage • Data storage • Data transfer • Data disposal • Data compliance Data awareness Accountability and responsibility Data quality Data audit and compliance Data protection • Organizational structure • Data owners • Data stewards • Discovery • Classification • Treatment • Access controls • Other controls • Encryption • Masking Data Governance Data Governance Principles Copyright © 2022 TDWI
- 12. Data governance and the data catalog • Provides an inventory of data to be controlled • A collaboration hub for governance • Provides information needed for governance (automated) – Metadata extraction – Data quality – Sensitive data – Data lineage • Provide quality metrics • Can integrate to other governance tools • Respond faster to governance regulations and audit Copyright © 2022 TDWI
- 13. • Completeness • Accuracy • Integrity • Consistency • Data usage • Data storage • Data transfer • Data disposal • Data compliance Data awareness Accountability and responsibility Data quality Data audit and compliance Data protection • Organizational structure • Data owners • Data stewards • Discovery • Classification • Treatment • Access controls • Other controls • Encryption • Masking Data Governance Areas where data catalog can help in governance Copyright © 2022 TDWI
- 14. THANK YOU!
- 16. CHRISTOPHER REED Sales Engineering Manager Precisely
- 17. Data catalog drivers 17 Today’s challenges • We stood up a data lake, it’s too much, now what? • Profusion of valued analytics. Driving need for shareable data • Regulatory compliance • You’ve been told you need a data catalog • My spreadsheet and email repositories are not keeping up
- 18. Data catalogs and data governance 18 Data Internet Webpages Marketplace Appliances Cosmetics Groceries Business Technical Positional Fixed length record Delimited XML JSON Unstructured AVRO, PARQUET, ORC MS Excel Files Databases & data lakes Applications Books eBooks Magazines Library Reports Analytics ETLs and technologies Cloud
- 19. A data catalog enables business-ready data 19 Data that is trusted Data that is ready to deliver outcomes Data that is easy to find and understand • Business objectives framework • Enterprise governance & ownership • Metrics & scoring • Balancing, reconciliation & controls • Stewardship & workflow • Case management • Data catalog & smart glossary • Data lineage & impact analysis • Data acquisition & analysis
- 20. What are the components of a “modern” data catalog? 20 Contains extensive information about your data Has business context Intuitive user interface Turnkey automation, administration, and integration Promotes governance and stewardship
- 21. What is governance? Governance is often influenced by perspective
- 22. Governance activities 22 • In a well-managed data environment data governance is well integrated with the business strategy • Modern data governance does this proactively and passively • Best in class solutions incorporate AI and ML to facilitate recommendation and automation engines Data strategy Data governance Data management / operations Strategy drives actions The data “W”s drive awareness (metadata) How well the data strategy is working (metric metadata) Identifies data that is important to strategy Business strategy Impact of data strategy on KPIs Explicit alignment to business goals & objectives Business alignment & impact metrics
- 23. Data governance is the keystone for your data catalog 23 Outcome Business objectives Measures & metrics Processes & stages People Data Governance Catalog = + Reporting & compliance Analytics & insights Operational excellence
- 24. Governance focuses on critical data 24 All available data 100% of data Data we use 40% of data Data we should govern 10% of data Data of high value 100-200 data elements CRITICAL DATA Data and metadata Selection of data at the system and source level (tables and fields) Information As required to develop a common language for important data Business process excellence To monitor the effectiveness of our processes design and execution Business goals & objectives Focus on critical data elements required to support value drivers and key initiatives The evolution has been towards greater reliance on rich metadata as it captures the data’s findability, usability and appropriateness within a particular context – implies well cataloged data
- 27. Data has too many relationships to be handled in spreadsheets 27 Answers need to be a few clicks away – not a few pivot tables away!
- 28. Compliance 28
- 29. Catalogs for compliance… If it is not cataloged – it is not governed! Contained in catalog: 1. How and where used 2. Data is labelled to show where it is in the lifecycle 3. Data is linked to a data package 4. Data has security classification 5. Personal information “type” label flags this data as falling under privacy regulations Do we know enough about the data to know it is being managed correctly?
- 30. Audit control model has transparency and accountability 30 Catalog contains the audit “view” of data: A. Task owner B. Task detail C. The data required D. The standard that guides the execution E. The control rule that enforces the standard F. The metric that measures compliance … which provides data level accountability Accountability is defined for each control point… Task Owner A Tasks B Data C Standard D Control Rule E Metric F
- 32. Data exchange with external manufacturing MDM Finance Source Pian Make Quality Deliver SAP P01 JDE 7.3 PC4 PC4 PC4 MBox PIP Quality Prisym 360 Wm Mbox Mbox MDS SAP P02 Laser etch SQLDB at SAP forms DocuSphere ATP Bank Sterling NRP Excel LIDO/ PAGO Neptune SAP GRC BODS SOLMAN SMI purchase order Basic data and classification Data load script Data load script Basic data and classification A/P check details (end state) Inventory Open purchase orders Sales orders Open purchase orders Advanced shipping Notification SMI Delivery note/ Sale order, ASN Doc. Print Requests Label data Material, serial code, batch data Production order – Lot master data Manual load email Production order – Lot master data Picking, manufacturing logistics transactions Manual interface with Sap P02 Lot master data Agile client (JnJ network) Company For data Conversion Solman LPPF Print Print Supply chain lifecycle Company External manufacturer Exchange
- 33. Data exchange with external manufacturing MDM Finance Source Plan Make Quality Deliver SAP P01 JDE 7.3 PC4 PC4 PC4 MBox PIP Quality Prisym 360 Wm Mbox Mbox MDS SAP P02 Laser etch SQLDB at SAP forms DocuSphere ATP Bank Sterling NRP Excel LIDO/ PAGO Neptune SAP GRC BODS SOLMAN SMI purchase order Basic data and classification Data load script Data load script Basic data and classification A/P check details (end state) Inventory Open purchase orders Sales orders Open purchase orders Advanced shipping Notification SMI Delivery note/ Sale order, ASN Doc. Print Requests Label data Material, serial code, batch data Production order – Lot master data Manual load email Production order – Lot master data Picking, manufacturing logistics transactions Manual interface with Sap P02 Lot master data Agile client (JnJ network) Company For data Conversion Solman LPPF Print Print Set up Order request Shipping notice Master data reference data Transaction data Billing notice shipping detail
- 34. Cataloging captures issue PRIOR to production 34 Data quality rule: the shipping notice cannot contain any master data or reference data that was not contained in the set-up file Objects contain data Interface has 3 objects
- 35. In closing… it is not just about the data! 35 How do you manage everything that surrounds the data? (metadata) Technical metadata Platforms; applications; tables; relationships Control metadata Rules; standards; ownership; RACI The data Semantic metadata / meaning & context Hierarchies; classification; allowed values…
- 36. Let’s continue the conversation… Contact us Set up a 30-minute personalized demo Precisely.com/contact www.precisely.com Demos White Papers Case Studies
- 38. CONTACT INFORMATION If you have further questions or comments: Fern Halper, TDWI Christopher Reed [email protected] @fhalper [email protected] Shaun Connolly [email protected] tdwi.org