A New Way of Thinking About MDM
4 likes•2,007 views
This document discusses master data management (MDM) and presents a new approach using an operational data hub with streamlined MDM. It begins by defining MDM and noting the complexity of traditional MDM systems. Traditional MDM uses relational databases and lengthy processes for data modeling, ETL, and integration across siloed systems. This leads to systems that are slow, expensive, and brittle. The document then introduces an alternative approach of using an operational data hub to directly integrate transactional applications and handle various data types. It describes how streamlined MDM can load data as-is, match and merge data at the point of engagement, maintain metadata and provenance for all data, and provide a simplified and flexible architecture
A New Way of Thinking About MDM
- 1. © COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED. A NEW WAY OF THINKING ABOUT MDM Michael Doane, Solutions Director
- 2. SLIDE: 2 © COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED. “MDM is the consistent and uniform set of identifiers and extended attributes that describe the core entities of the enterprise and are used across multiple business processes. ” — Gartner “The set of disciplines and methods to ensure the currency, meaning, quality, and deployment of a company’s reference data within and across subject areas.” — Baseline Consulting “A set of disciplines, processes and technologies, for ensuring the accuracy, completeness, timeliness and consistency of multiple domains of enterprise data ‒ across applications, systems and databases, and across multiple business processes, functional areas, organizations, geographies and channels.” — Dan Power, Hub Designs What is Master Data Management? MANY DEFINITIONS
- 3. SLIDE: 3 © COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED. MDM is Complex DATA QUALITY BUSINESS RULES DATA FLOWS INTEGRATION POINTS STEWARDSHIP AUDIT SUPPORTING PROCESSES & WORKFLOW MANY COMPONENTS MASTER DATA
- 4. SLIDE: 4 © COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED. Traditional Lengthy Lifecycle of RDBMS-based MDM Analyze Data Sources | Create Data Dictionary Create Canonical List of Entities & Attributes Create Canonical Data Model Map Sources to Data Model Write & Test ETL processes Write & Test Data Source Priority Rules Write & Test Disambiguation Rules Load Data Gather DQ Metrics Data Cleansing Operations Query Writing Performance Testing Functional Testing Development Time
- 5. SLIDE: 5 © COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED. Lengthy modeling & ETL design processes = slow progress, dwindling interest, brittle to adapt to changes in goals and data Some successes in personnel and custom siloed solutions Usually owned by IT and disconnected from the impact to the enterprise’s profit and loss Chasing a truth-based model with a rigid golden definition vs. a trust-based model with a golden portion and flexibility to capture and keep all data Traditional MDM Faces Strong Headwinds
- 6. SLIDE: 6 © COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED. The Problem With the Relational Approach The Business Changes, The Requirements Change, The Source Data Changes 1 Take a Current State Snapshot Design the New Data Model Perform ETL Create the Indexes 2 3 4 Build the Application 5 Restart Process 6
- 7. SLIDE: 7 © COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED. Repeat this Again? Analyze Data Sources | Create Data Dictionary Create Canonical List of Entities & Attributes Create Canonical Data Model Map Sources to Data Model Write & Test ETL processes Write & Test Data Source Priority Rules Write & Test Disambiguation Rules Load Data Gather DQ Metrics Data Cleansing Operations Query Writing Performance Testing Functional Testing Development Time
- 8. SLIDE: 8 © COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED. Traditional Data Integration Is Complex and Fractured RDBMS for highly structured data Specialized databases for other data types ETL and integration software to connect silos …and, what about hierarchical data? …and, what about unstructured content? …maybe a data lake will help? UNFORTUNATE REALITY ETL OLTP ARCHIVES ETL ETL DATA MARTS ETL MDM REPOSITORY REFERENCE DATA ETL UNSTRUCTURED DATA ETL SEARCH ETL HADOOP MAINFRAME ETL
- 9. SLIDE: 9 © COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED. OLTP WAREHOUSE DATA MARTS ARCHIVES REFERENCE DATA OLTP WAREHOUSE DATA MARTS DATA MARTS OLTP WAREHOUSE DATA MARTS DATA MARTS WAREHOUSE DATA MARTS DATA MARTS OLTP HADOOP UNSTRUCTURED DATA REFERENCE DATA OLTP WAREHOUSE ARCHIVES Traditional Data Integration Complex – Fixed schemas and sprawling components Slow – Too long to develop, deploy, and update Expensive – High costs for software and personnel Brittle – Changes become overwhelming
- 10. SLIDE: 10 © COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED. Re-thinking Master Data Management Achieve progress incrementally Tie to business drivers and events Reduce duplicate data and lengthy ETL processes Adjust to change A NEW APPROACH MARKETING SALES CRMERP FINANCELINE-OF- BUSINESS 1 LINE-OF- BUSINESS 2 HR
- 11. SLIDE: 11 © COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED. Operational Data Hub 360 views of important entities Direct integration with transactional applications Handles volume, variety, velocity (like a data lake) FIX THE ARCHITECTURE MARKETING SALES CRMERP HR FINANCELINE-OF- BUSINESS 1 LINE-OF- BUSINESS 2
- 12. SLIDE: 12 © COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED. Streamlined Master Data Management A NEW APPROACHDATA SOURCES MATCH. MERGE & UNMERGE OPERATIONAL DATA HUB CORE JSON XML CUSTOMER ENGAGEMENT OPERATIONS FORMATS EVOLVE NEW SOURCES EASILY ADDED BUSINESS OPERATIONS HARMONIZE IN PLACE
- 13. SLIDE: 13 © COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED. Streamlined Master Data Management Schema-agnostic: Load “as-is”, minimize ETL, incrementally deliver results early on, maintain buy-in A NEW APPROACH DATA SOURCES MATCH. MERGE & UNMERGE OPERATIONAL DATA HUB CORE JSON XML CUSTOMER ENGAGEMENT OPERATIONS FORMATS EVOLVE NEW SOURCES EASILY ADDED BUSINESS OPERATIONS
- 14. SLIDE: 14 © COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED. Streamlined Master Data Management Contextual: Improved data quality through minimizing data duplication at point of engagement A NEW APPROACH DATA SOURCES MATCH. MERGE & UNMERGE OPERATIONAL DATA HUB CORE JSON XML CUSTOMER ENGAGEMENT OPERATIONS FORMATS EVOLVE NEW SOURCES EASILY ADDED BUSINESS OPERATIONS
- 15. SLIDE: 15 © COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED. Streamlined Master Data Management All the Data: Master a subset and keep all source data in original formats; reverse changes or unmerge at any time A NEW APPROACH DATA SOURCES MATCH. MERGE & UNMERGE OPERATIONAL DATA HUB CORE JSON XML CUSTOMER ENGAGEMENT OPERATIONS FORMATS EVOLVE NEW SOURCES EASILY ADDED BUSINESS OPERATIONS
- 16. SLIDE: 16 © COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED. Streamlined Master Data Management Metadata Unlimited: Maintain data provenance, bitemporal timestamps, security on every data element A NEW APPROACH DATA SOURCES MATCH. MERGE & UNMERGE OPERATIONAL DATA HUB CORE JSON XML CUSTOMER ENGAGEMENT OPERATIONS FORMATS EVOLVE NEW SOURCES EASILY ADDED BUSINESS OPERATIONS
- 17. SLIDE: 17 © COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED. Streamlined Master Data Management Simplified Architecture: Database, Search, Application Services, Security and more in one QA’d platform A NEW APPROACH DATA SOURCES MATCH. MERGE & UNMERGE OPERATIONAL DATA HUB CORE JSON XML CUSTOMER ENGAGEMENT OPERATIONS FORMATS EVOLVE NEW SOURCES EASILY ADDED BUSINESS OPERATIONS
- 18. SLIDE: 18 © COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED. Streamlined Master Data Management Logical Data Model: Entities represent naturally as Documents, not shredded. Semantic Triples to relate data A NEW APPROACH DATA SOURCES MATCH. MERGE & UNMERGE OPERATIONAL DATA HUB CORE JSON XML CUSTOMER ENGAGEMENT OPERATIONS FORMATS EVOLVE NEW SOURCES EASILY ADDED BUSINESS OPERATIONS
- 19. SLIDE: 19 © COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED. Streamlined Master Data Management Secure: Support for operational apps data access control and data governance A NEW APPROACH DATA SOURCES MATCH. MERGE & UNMERGE OPERATIONAL DATA HUB CORE JSON XML CUSTOMER ENGAGEMENT OPERATIONS FORMATS EVOLVE NEW SOURCES EASILY ADDED BUSINESS OPERATIONS
- 20. SLIDE: 20 © COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED. Streamlined Master Data Management Full Auditing: Capture who, what, when and why for all user and automated data changes and actions A NEW APPROACH DATA SOURCES MATCH. MERGE & UNMERGE OPERATIONAL DATA HUB CORE JSON XML CUSTOMER ENGAGEMENT OPERATIONS FORMATS EVOLVE NEW SOURCES EASILY ADDED BUSINESS OPERATIONS
- 22. SLIDE: 22 © COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED. Operational Data Hub with Streamlined MDM STATE DHHS SOLUTION Operational MDM with fuzzy match to find duplicates and correct or merge POC APPLICATIONS JSON XML OPERATIONAL CASE WORK ADULT BENEFITS DETERMINATION INCREMENTAL ADDITION OF NEW DATA SOURCES CHILDREN’S SOCIAL SERVICES JUVENILE SERVICES ASYLUM & REFUGEE ASSISTANCE RESOURCE ELIGIBILITY SYSTEM HOME ENERGY ASSISTANCE ASYLUM & REFUGEE ASSISTANCE OTHER (RESTful SERVICES) REST API
- 23. SLIDE: 23 © COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED. Operational Mastering HEALTHCARE.GOV Operational Mastering at Point of Engagement Reduces Duplicates and Need for Traditional MDM CUSTOMER SERVICE REPS JSON XML CONSUMER WEB ACCESS OPERATIONAL MASTERING STATE BENCHMARKS MULTIPLE SOURCES & TYPES INSURANCE POLICIES USER FINANCIAL DATA INSURANCE PLANS & RATES EVENTS CHANGE UTILITY CASE EDITING MEDICAID TRANSFERS OPERATIONAL MASTERING OPERATIONAL MASTERING HEALTH INSURANCE MARKETPLACE MIDAS FINANCIALS (HADOOP) IRS SSA TRICARE PEACE CORPS DHS OPM ELIGIBILITY DETERMINATION STATE EXCHANGES DATA SERVICES HUB
- 24. SLIDE: 24 © COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED. An Operational and Transactional Enterprise NoSQL Database Makes Streamlined MDM Possible The MarkLogic Alternative Data ingested as is (no ETL) Structured and unstructured data Data and metadata together Adapts to changing data and changing data structures EASY TO GET DATA IN Flexible Data Model Index once and query endlessly Real-time and lightning fast Query across JSON, XML, text, geospatial, and semantic triples in one database EASY TO GET DATA OUT Ask Anything Universal Index Reliable data and transactions (100% ACID compliant) Out-of-the-box automatic failover, replication, and backup/recovery Enterprise-grade security and Common Criteria certified 100% TRUSTED Enterprise Ready
- 25. Questions?