SlideShare a Scribd company logo
Data Quality Technical Architecture Components
Contents Data Quality Best-in-Class Properties Data Quality Technical Architecture Core Functional Components Interfacing Components Non-functional Components Data Quality Invocation Flow Example Data Quality Technical Architecture Foundational Systems Data Quality Technical Architecture Products / Technologies Mapping of architecture concepts to Data Quality Technical Architecture
Data Quality Best-in-Class Properties High-performing Fast response times for human users and high throughput for non-human users within given service level agreements Reliable and recoverable All data and process changes are transactional secured and can be recovered to the last consistent state after any type of failure Uniform and consistent All similar processing is done in the same way and all processing leads to a consistent system state (including data and processes) Dynamically changeable and extensible Changes can be applied dynamically to accommodate new business as well as system requirements Minimal and simple As few as possible foundational systems and concepts as necessary to implement the overall functionality Conceptually sound All functionality is based on a conceptual model that reflects the business architecture goals and requirements
Data Quality Architecture Concepts Architecture Concept Service Workflow Business and Data Rules Data and Data Flow External services and workflows External Services Architecture Component Web Service Management Workflow Management Business Rule Management Data Management Packaged Application Management External Invocation Management The 1:1 relationship of architectural concepts and architectural  components is intentional
DQ Technical Architecture –  Core Functional Components Data Storage Data Replication Reference Data Data Flow Data Management Data Access Workflow Access Business Rule Access Packaged Application Management Service Definition Service Execution Service Choreography (Web) Service Management Service Registry Workflow Definition Workflow Execution Workflow Management Business Rule Definition Business Rule Automation Business Rule Management Gateway External Invocation Management Public Service Registry DQ External Interface Management Service Publication Case Management Policy Management
DQ Technical Architecture –  Core Functional Components Data Storage Data Replication Reference Data Data Flow Data Management Data Access Workflow Access Business Rule Access Packaged Application Management Service Definition Service Execution Service Choreography (Web) Service Management Service Registry Workflow Definition Workflow Execution Workflow Management Business Rule Definition Business Rule Automation Business Rule Management Gateway External Invocation Management Public Service Registry DQ External Interface Management Service Publication Case Management Policy Management Fundamental Pillars of Production Systems
DQ Technical Architecture –  Web Service Management System
DQ Technical Architecture – Workflow Management System
DQ Technical Architecture –  Business Rule Management System
DQ Technical Architecture – Data Management System
DQ Technical Architecture –  Packaged Application Management Integration of Crystal Report Software Integration of BO/DQ Software
DQ Technical Architecture –  External Invocation Management
DQ Services -  Components Member Provider Claim Subject Area: Other Service Area: Enrichment Profiling  Other De-duping Data Area: Address Name Phone SSN Other TIN Tool Area: BO/DQ Project Crystal Report Other Rules Area: Mandated Rule Business Rule Technology  Rule Industry Standard  Rule Software Area: Shell Script JAVA Wrapper Other Method Area: Set() Throw() Get() Run()
DQ Rules – Mandated Rules At least one of  member ids to access person id if the person id is related to more than one member id. The valid coverage contract (policy holder)  must be establish before any member added for coverage. Provider can have only one  license in a state. Provider must have TIN. If  Paper Claim then Paper Claim  = (Member) ID
DQ Rules – Business Rules Grievances are to be handled professionally and expeditiously to maintain member satisfaction Provider can only be par if he has official agreement with Delta If the coverage code changes from Family to Subscriber and Child (from some coverage code that includes spouse to one that eliminates the spouse), then the dropping of that spouse member should occur automatically with the termination date equal to the effective date for the new coverage.
DQ Rules – Technology Rules VSAM Structure as multiple occurrence of same record to accommodate record size can be one record in relational db. In short, duplicate record can stored as unique record in relational db. History Cross-check (HCC) Rules are rules that define if a procedure has frequency limits within a specific time period.
DQ Rules – Industry Standard Rules  Dental Service of Massachusetts, Inc., (DSM) defines the privacy practices to protect personal health information.  DSM is required by law to:  Maintain the privacy of patient health information.  Provide this notice of company  duties and privacy practices.  Abide by the terms of the notice currently in effect DSM Account Information is password-protected for patient privacy and security.
DQ Services –  EEP E1 Invocation (As-is) requests executes using processes for has applies invokes invokes Conceptual Member Subject Area: Enrichment Service Area: Data Area: Address Method Area: Run() Tool Area: BO/DQ Project Tool End User Workspace Driver Rules Area: Mandated Rule Software Area: JAVA Wrapper Software Area: Shell  Script
DQ Services – Address Standardization (To-be) Address standardization for member data  to match zip code with city and state. Conceptual
DQ Services –  Address Standardization (To-be) Address standardization for member data  to match zip code with city and state. executes using processes for has applies requests invokes Conceptual Member Subject Area: Profiling Service Area: Data Area: Address Method Area: Get() Tool Area: Crystal Report Tool End User Workspace Driver Rules Area: Industry Standard Rule Software Area: Other (SQL)
DQ Services –  Mixed Case (To-be) Convert all upper case name  to mixed case name  Conceptual Draft
DQ Services –  Mixed Case (To-be) Convert all upper case name  to mixed case name executes using processes for has applies requests invokes Conceptual Draft Provider Subject Area: Enrichment Service Area: Data Area: Name Method Area: Set() Tool Area: BO/DQ Project Tool End User Workspace Driver Rules Area: Business Rules Software Area: Shell/JAVA
DQ Services – Enrichment services Address – Street Address, City, State, Zip, Location Contact –  Name, Phone Number Person – Last Name, First Name, Middle Name and Name Suffix, Social Security Number, TIN Other – TBD Data Cleansing Data Normalization For
Data Acquisition Type – Member  Web – B2C EDI – B2C
DQ Services – Profiling services Exception Reporting Pre Cleansing  Post Cleansing
DQ Services – De-duping services Address –  Street Address, City, State, Zip, Location Contact –  Name, Phone Number Person –  Last Name, First Name, Middle Name and Name Suffix, Social Security Number, TIN, etc Other – TBD

More Related Content

PDF
Data Governance Best Practices
DATAVERSITY
 
PPTX
Strategic Business Requirements for Master Data Management Systems
Boris Otto
 
PDF
Top 10 Artifacts Needed For Data Governance
First San Francisco Partners
 
PDF
Why data governance is the new buzz?
Aachen Data & AI Meetup
 
PDF
The Role of Data Governance in a Data Strategy
DATAVERSITY
 
PDF
Metadata Strategies - Data Squared
DATAVERSITY
 
PDF
Data Quality Management - Data Issue Management & Resolutionn / Practical App...
Burak S. Arikan
 
PDF
Building a Data Strategy – Practical Steps for Aligning with Business Goals
DATAVERSITY
 
Data Governance Best Practices
DATAVERSITY
 
Strategic Business Requirements for Master Data Management Systems
Boris Otto
 
Top 10 Artifacts Needed For Data Governance
First San Francisco Partners
 
Why data governance is the new buzz?
Aachen Data & AI Meetup
 
The Role of Data Governance in a Data Strategy
DATAVERSITY
 
Metadata Strategies - Data Squared
DATAVERSITY
 
Data Quality Management - Data Issue Management & Resolutionn / Practical App...
Burak S. Arikan
 
Building a Data Strategy – Practical Steps for Aligning with Business Goals
DATAVERSITY
 

What's hot (20)

PDF
The Non-Invasive Data Governance Framework
DATAVERSITY
 
PDF
RWDG Slides: Building a Data Governance Roadmap
DATAVERSITY
 
PDF
Building a Data Strategy – Practical Steps for Aligning with Business Goals
DATAVERSITY
 
PDF
The Importance of Master Data Management
DATAVERSITY
 
PDF
Strategic Placemat 2016 Final
Lisa Tran
 
PDF
Data Quality Strategies
DATAVERSITY
 
PDF
Big Data Strategies – Organizational Structure and Technology
DATAVERSITY
 
PDF
A Comparative Study of Data Management Maturity Models
Data Crossroads
 
PPTX
Capability Model_Data Governance
Steve Novak
 
PDF
Data Catalog as the Platform for Data Intelligence
Alation
 
PDF
Data Quality Best Practices
DATAVERSITY
 
PPT
Data Architecture for Data Governance
DATAVERSITY
 
PDF
Data Governance Program Powerpoint Presentation Slides
SlideTeam
 
PDF
Data Quality Strategy: A Step-by-Step Approach
FindWhitePapers
 
PDF
Becoming a Data-Driven Organization - Aligning Business & Data Strategy
DATAVERSITY
 
PDF
CDO - Chief Data Officer Momentum and Trends
Jeffrey T. Pollock
 
PPT
Building a Data Quality Program from Scratch
dmurph4
 
PDF
Building a Data Governance Strategy
Analytics8
 
PPTX
Digital transformation roadmap 16th july 2018 v1.1
Chandresh Adhiya
 
PDF
Data Modeling Fundamentals
DATAVERSITY
 
The Non-Invasive Data Governance Framework
DATAVERSITY
 
RWDG Slides: Building a Data Governance Roadmap
DATAVERSITY
 
Building a Data Strategy – Practical Steps for Aligning with Business Goals
DATAVERSITY
 
The Importance of Master Data Management
DATAVERSITY
 
Strategic Placemat 2016 Final
Lisa Tran
 
Data Quality Strategies
DATAVERSITY
 
Big Data Strategies – Organizational Structure and Technology
DATAVERSITY
 
A Comparative Study of Data Management Maturity Models
Data Crossroads
 
Capability Model_Data Governance
Steve Novak
 
Data Catalog as the Platform for Data Intelligence
Alation
 
Data Quality Best Practices
DATAVERSITY
 
Data Architecture for Data Governance
DATAVERSITY
 
Data Governance Program Powerpoint Presentation Slides
SlideTeam
 
Data Quality Strategy: A Step-by-Step Approach
FindWhitePapers
 
Becoming a Data-Driven Organization - Aligning Business & Data Strategy
DATAVERSITY
 
CDO - Chief Data Officer Momentum and Trends
Jeffrey T. Pollock
 
Building a Data Quality Program from Scratch
dmurph4
 
Building a Data Governance Strategy
Analytics8
 
Digital transformation roadmap 16th july 2018 v1.1
Chandresh Adhiya
 
Data Modeling Fundamentals
DATAVERSITY
 
Ad

Similar to Data Quality Technical Architecture (20)

PPT
Data Quality Services
Harshendu Desai
 
DOCX
Vinay-Resume
sagarv48
 
PPTX
Yongsan presentation 3
GovCloud Network
 
PPT
Ws Soa V6 Theory And Practice
Pini Cohen
 
PPTX
SharePoint 2010 Global Deployment
Joel Oleson
 
PDF
SoftwareONE Oracle Licensing Introduction 18.02.14
SoftwareONEPresents
 
PPT
Insurance Vertical Presentation
Murty NSN
 
PDF
chapter 2.pdf
Susan130641
 
PPT
Soa Business And Technical Overview Presentation (Reed003707)
miteshisheth
 
PPT
Soa Business And Technical Overview Presentation (Reed003707)
miteshisheth
 
PPT
Soa Business And Technical Overview Presentation (Reed003707)
miteshisheth
 
PPT
Soa Business And Technical Overview Presentation (Reed003707)
miteshisheth
 
PPT
Soa Business And Technical Overview Presentation (Reed003707)
miteshisheth
 
PPT
Soa Business And Technical Overview Presentation (Reed003707)
miteshisheth
 
PPS
STL LItigation Services
guestc7f86
 
PPT
On-Demand: Is It Right For Your Company?
Callidus Software
 
PDF
SoftwareONE Oracle Licensing Introduction 18.02.14
SoftwareONEIndia
 
PDF
A "First Time Right" Start with Data Virtualization by Bart De Groeve, Practi...
Patrick Van Renterghem
 
PPT
Service Analysis And Design
Rody Middelkoop
 
DOCX
Venkataraman Raghunathan
Venkataraman Raghunathan
 
Data Quality Services
Harshendu Desai
 
Vinay-Resume
sagarv48
 
Yongsan presentation 3
GovCloud Network
 
Ws Soa V6 Theory And Practice
Pini Cohen
 
SharePoint 2010 Global Deployment
Joel Oleson
 
SoftwareONE Oracle Licensing Introduction 18.02.14
SoftwareONEPresents
 
Insurance Vertical Presentation
Murty NSN
 
chapter 2.pdf
Susan130641
 
Soa Business And Technical Overview Presentation (Reed003707)
miteshisheth
 
Soa Business And Technical Overview Presentation (Reed003707)
miteshisheth
 
Soa Business And Technical Overview Presentation (Reed003707)
miteshisheth
 
Soa Business And Technical Overview Presentation (Reed003707)
miteshisheth
 
Soa Business And Technical Overview Presentation (Reed003707)
miteshisheth
 
Soa Business And Technical Overview Presentation (Reed003707)
miteshisheth
 
STL LItigation Services
guestc7f86
 
On-Demand: Is It Right For Your Company?
Callidus Software
 
SoftwareONE Oracle Licensing Introduction 18.02.14
SoftwareONEIndia
 
A "First Time Right" Start with Data Virtualization by Bart De Groeve, Practi...
Patrick Van Renterghem
 
Service Analysis And Design
Rody Middelkoop
 
Venkataraman Raghunathan
Venkataraman Raghunathan
 
Ad

Data Quality Technical Architecture

  • 1. Data Quality Technical Architecture Components
  • 2. Contents Data Quality Best-in-Class Properties Data Quality Technical Architecture Core Functional Components Interfacing Components Non-functional Components Data Quality Invocation Flow Example Data Quality Technical Architecture Foundational Systems Data Quality Technical Architecture Products / Technologies Mapping of architecture concepts to Data Quality Technical Architecture
  • 3. Data Quality Best-in-Class Properties High-performing Fast response times for human users and high throughput for non-human users within given service level agreements Reliable and recoverable All data and process changes are transactional secured and can be recovered to the last consistent state after any type of failure Uniform and consistent All similar processing is done in the same way and all processing leads to a consistent system state (including data and processes) Dynamically changeable and extensible Changes can be applied dynamically to accommodate new business as well as system requirements Minimal and simple As few as possible foundational systems and concepts as necessary to implement the overall functionality Conceptually sound All functionality is based on a conceptual model that reflects the business architecture goals and requirements
  • 4. Data Quality Architecture Concepts Architecture Concept Service Workflow Business and Data Rules Data and Data Flow External services and workflows External Services Architecture Component Web Service Management Workflow Management Business Rule Management Data Management Packaged Application Management External Invocation Management The 1:1 relationship of architectural concepts and architectural components is intentional
  • 5. DQ Technical Architecture – Core Functional Components Data Storage Data Replication Reference Data Data Flow Data Management Data Access Workflow Access Business Rule Access Packaged Application Management Service Definition Service Execution Service Choreography (Web) Service Management Service Registry Workflow Definition Workflow Execution Workflow Management Business Rule Definition Business Rule Automation Business Rule Management Gateway External Invocation Management Public Service Registry DQ External Interface Management Service Publication Case Management Policy Management
  • 6. DQ Technical Architecture – Core Functional Components Data Storage Data Replication Reference Data Data Flow Data Management Data Access Workflow Access Business Rule Access Packaged Application Management Service Definition Service Execution Service Choreography (Web) Service Management Service Registry Workflow Definition Workflow Execution Workflow Management Business Rule Definition Business Rule Automation Business Rule Management Gateway External Invocation Management Public Service Registry DQ External Interface Management Service Publication Case Management Policy Management Fundamental Pillars of Production Systems
  • 7. DQ Technical Architecture – Web Service Management System
  • 8. DQ Technical Architecture – Workflow Management System
  • 9. DQ Technical Architecture – Business Rule Management System
  • 10. DQ Technical Architecture – Data Management System
  • 11. DQ Technical Architecture – Packaged Application Management Integration of Crystal Report Software Integration of BO/DQ Software
  • 12. DQ Technical Architecture – External Invocation Management
  • 13. DQ Services - Components Member Provider Claim Subject Area: Other Service Area: Enrichment Profiling Other De-duping Data Area: Address Name Phone SSN Other TIN Tool Area: BO/DQ Project Crystal Report Other Rules Area: Mandated Rule Business Rule Technology Rule Industry Standard Rule Software Area: Shell Script JAVA Wrapper Other Method Area: Set() Throw() Get() Run()
  • 14. DQ Rules – Mandated Rules At least one of member ids to access person id if the person id is related to more than one member id. The valid coverage contract (policy holder) must be establish before any member added for coverage. Provider can have only one license in a state. Provider must have TIN. If Paper Claim then Paper Claim = (Member) ID
  • 15. DQ Rules – Business Rules Grievances are to be handled professionally and expeditiously to maintain member satisfaction Provider can only be par if he has official agreement with Delta If the coverage code changes from Family to Subscriber and Child (from some coverage code that includes spouse to one that eliminates the spouse), then the dropping of that spouse member should occur automatically with the termination date equal to the effective date for the new coverage.
  • 16. DQ Rules – Technology Rules VSAM Structure as multiple occurrence of same record to accommodate record size can be one record in relational db. In short, duplicate record can stored as unique record in relational db. History Cross-check (HCC) Rules are rules that define if a procedure has frequency limits within a specific time period.
  • 17. DQ Rules – Industry Standard Rules Dental Service of Massachusetts, Inc., (DSM) defines the privacy practices to protect personal health information. DSM is required by law to: Maintain the privacy of patient health information. Provide this notice of company duties and privacy practices. Abide by the terms of the notice currently in effect DSM Account Information is password-protected for patient privacy and security.
  • 18. DQ Services – EEP E1 Invocation (As-is) requests executes using processes for has applies invokes invokes Conceptual Member Subject Area: Enrichment Service Area: Data Area: Address Method Area: Run() Tool Area: BO/DQ Project Tool End User Workspace Driver Rules Area: Mandated Rule Software Area: JAVA Wrapper Software Area: Shell Script
  • 19. DQ Services – Address Standardization (To-be) Address standardization for member data to match zip code with city and state. Conceptual
  • 20. DQ Services – Address Standardization (To-be) Address standardization for member data to match zip code with city and state. executes using processes for has applies requests invokes Conceptual Member Subject Area: Profiling Service Area: Data Area: Address Method Area: Get() Tool Area: Crystal Report Tool End User Workspace Driver Rules Area: Industry Standard Rule Software Area: Other (SQL)
  • 21. DQ Services – Mixed Case (To-be) Convert all upper case name to mixed case name Conceptual Draft
  • 22. DQ Services – Mixed Case (To-be) Convert all upper case name to mixed case name executes using processes for has applies requests invokes Conceptual Draft Provider Subject Area: Enrichment Service Area: Data Area: Name Method Area: Set() Tool Area: BO/DQ Project Tool End User Workspace Driver Rules Area: Business Rules Software Area: Shell/JAVA
  • 23. DQ Services – Enrichment services Address – Street Address, City, State, Zip, Location Contact – Name, Phone Number Person – Last Name, First Name, Middle Name and Name Suffix, Social Security Number, TIN Other – TBD Data Cleansing Data Normalization For
  • 24. Data Acquisition Type – Member Web – B2C EDI – B2C
  • 25. DQ Services – Profiling services Exception Reporting Pre Cleansing Post Cleansing
  • 26. DQ Services – De-duping services Address – Street Address, City, State, Zip, Location Contact – Name, Phone Number Person – Last Name, First Name, Middle Name and Name Suffix, Social Security Number, TIN, etc Other – TBD

Editor's Notes

  • #8: EDQ Web Service Management enables to query multiple web services in a unified and transparent fashion through a SQL-like interface. Metadata Component deals with the registration of new web services, and mapping their schemas to the global view exported in the client interface. Query Processing and Optimization is to take a user-input query and find the optimal "query plan", i.e., the most efficient way to process the query by issuing calls to the relevant web services. The execution engine then executes this query plan by making actual web service calls and efficiently managing the data exchange between the WSMS and the various web services. Profiling and Statistics component profiles the web services for response time and maintain the relevant statistics over the web service data. It is used to optimize the Query processing.
  • #9: The workflow service can be called asynchronously from BPEL to perform operations like adding, updating, completing, renewing, routing and escalating tasks. User applications, on the other hand, can communicate with the Workflow Service to acquire the list of tasks for selected users, render appropriate user interfaces, and return results to the workflow service, which forwards them to the BPEL process. BPEL Process - Business Process Execution Language for Web Services provides a means to formally specify business processes and interaction protocols.
  • #10: A BRMS or Business Rule Management System is to manage and support the EDQ business rules. Policy Management - Business Rule Definition – It defines the business constraint and asserts business structure or influence the behavior of business. Business Rule Automation - It is a technology components substituting / Supplementing the manual processes to manage information flow. Business Rule Engine – A system which manage and automate business rules.
  • #11: MDM – Master Data Management Federated MDM - cross references key identifying information from participating systems to implement a registry-style solution. The main benefit of a federated solution is non-intrusiveness on participating systems that maintain their original context. Integrated MDM - stores all master data information from all participating systems in a centralized MDM repository. This centralized repository houses the "gold copy" of all master data information. The main benefit of the integrated approach is that it provides the most complete, accurate, and consistent single view of master data. Hybrid MDM - stores common data elements from participating systems creating a "light gold copy" of the master data, while disparate elements are referenced from their original system of record. The benefit and drawback of the hybrid solution is the partial combination of the federated and integrated benefits. MDM and SOA evolved separately but share many design principles. "Contract first" applies to the interfaces in MDM and the service definition in SOA "Reusability" applies to data through conformance in MDM and services through SOA principles in SOA "Discoverability" applies to data through the master data repository in MDM and services through registry in SOA "Abstraction" applies to source system complexity and MDM and underlying service complexity under SOA.
  • #12: Data Access - Workflow Access Business Rule Access