Data quality practical guide
0 likes389 views
The document discusses the critical importance of data quality across industries, highlighting significant financial losses and legal requirements tied to poor data quality. It emphasizes the necessity for collaboration between business and IT to establish clear data ownership, management, and quality improvement processes. Additionally, it addresses the challenges of ensuring data quality in an organizational setting, advocating for proactive measures rather than reactive responses to data quality issues.
Data quality practical guide
- 1. Data Quality From Theory to Practice MARCUS EVANS CONFERENCES Paul Ormonde-James Group Intelligence MBF Australia
- 2. Agenda • Why is data quality important?
- 3. The problem to be solved….. Target You are here
- 4. The problem to be solved….. Target You are here
- 5. Paul Ormonde-James – Presenter, Speaker, Author • Paul, has spent many years in the field of Change Management. He has qualifications in Cybernetics (Robotics & Artificial Intelligence), Computer Sciences, a Master of Business Administration (MBA) and post grad in Company Law. He has driven many change programs and used Business Intelligence, Competitive Intelligence and Knowledge Management to drive such cultural change to increase company value.
- 6. Business Intelligence, simply….. Corporate Survival & Growth “Growing through Knowing” More effective decision making At the To the
- 8. Business BI
- 9. Why do we need data quality? • There is a growing cross-industry recognition of the value of high-quality data. • 2001 PricewaterhouseCoopers Global Data Management Survey, 75 percent of the senior executives reported significant problems as a result of defective data. • In 2002, The Data Warehousing Institute published its Report on Data Quality in which they estimated that the cost of poor data quality to U.S. businesses exceeds $600 billion each year. • Two critical pieces of legislation passed within the past few years impose strict information quality requirements on both public corporations in the U.S. (the Sarbanes-Oxley Act of 2002) • As well as U.S. federal agencies (The Data Quality Act of 2001). • Both of these laws require organizations to provide auditable details as to the levels of their information quality.
- 10. Do Managers really understand data quality????? • How many times does a manager or executive ask where the data comes from? • How many executives understand the numerous data sources in and across an organisation • How many Executives understand the errors generated as incorrect data is summarised. • Are executives more focussed on quality data the more important the decision?
- 11. The Journey….. Self Service Model Place Logo Here Enterprise Business Reporting & Analytics (DW Strategy) Current Applications Model Customers Customer Reports, Investment Performance Reports (Quarterly, Monthly) Business Unit Classification Clearview Exit Reports (Weekly / Ad hoc) (via MS PowerPoint) Clearview Leads Reports (Weekly / Ad hoc) (via MS PowerPoint) Funds Sales Reports (Weekly / Ad hoc) (via MS PowerPoint) Misc. Clearview & MBF Life Reports Financial Services Group Marketing Risk Management & MBF Health Corporate Culture & Group Strategy & New PHI Group Finance & Board Treasury (PHI) Communication Business Development Corporate Services KPI / Balanced Score Card (via spreadsheet or flat file on J Drive) Membership & Sales Reports Claims, Provider, MBF Health KPI (via MS Excel) General, KPI & Clearview Product Reports (Ad hoc) Investment & Budget & Rate Rise Reports Health Program Reports MBF & MBFH Actuarial Reports Performance Reports Actuarial Reports P&L & OPEX Sales & Ad-hoc Reports Executive Pack Reports Membership Reports Claims Reports Ad-hoc Reports - National Claims Reports (via Excel & Word) (Daily & Monthly) In2Life Campaign Reports (Monthly) Membership Customer Matrix (Monthly) - Member Services Reports (via Excel & Word) (Daily, Weekly & Monthly) Reports - Customer Enquiry, Complaints & Compliments Reports (via Excel & Powerpoint) Sales Reports (Monthly) - Membership, Members’ Exits & Exit Interviews Reports Ad-hoc Reports - Teams’ Performance Reports (via Powerpoint) Dashboard Suite of Reports (Monthly) Membership & Sales Reports Membership Reports Customer Retention & Sales Reports External Organisations and Customer Contact Channels Ad-hoc Reports FTE Reports Reporting on Investment, Regulatory, Budgeting, Pricing, Variances, EOM Provisioning, Claims Analysis Components Staff Assisted MBF & MBFH Misc. Membership & Competitors ad-hoc Analysis & Reports Exceptional Request / Reports Financial Services Business Intelligence Group IT & Program Business & Clinical Analysis Risk Management & Group Marketing Group Finance & MBF Health PHI (Group Finance) Management (PHI) Treasury Corporate Services (PHI) Membership Report Analysis Reports KPI (via MS Excel) EOM & Ad-hoc (eg. Customer, Reports (via Excel) Investment & Actuarial / Membership, Benefits & Sales Reports & Analytics Customer Profiling & Demographic Data Membership & Sales Reports (Weekly,Monthly) Competitor) Sales Reports Dashboard Data from all MBF Business Units Satisfaction Surveys (via Excel) Call Center Management Reports (Weekly) Front End Processing Components Complete Claims History Data (Monthly) (via Flat File) Back End Processing Back Office Departments Summurisation of Revenue & Membership Data (Monthly) Components Customer Enquiry, Complaints & Compliments Reports (via Excel & Powerpoint) / Membership, Members’ Exits & Exit Interviews Reports MISC Internal Reporting (Membership, Claims, Sales Activity) (Weekly) BASE SAS MONTH END CONSOLIDATED CDATA DATABASE SPREADSHEETS Membership & Sales Reports (Weekly, Monthly) FINANCIAL Standard Detailed Claims Data Data Formatting & Revenue, Membership P&L, Cost Centre OPEX (Monthly) (via OPEX Excel Spreadsheet) REPORTING Census Data & Claims Data (MS Access Database) Various spreadsheets Consolidation used by Investments & (MS Word) Actuarial for PMP Compliance, Risk & Contract Reports (via Excel for Formatting) Analysis & Reporting Services Utilization Reports Eg GL Variances Misc Reports Calculations Services Utilization Reports Retail, Corporate & Travel Business Intelligence Misc. Data (via Dynamic Link) Claims Leakage & Exception Reports (via Excel) Misc. Clearview & Misc. Reports Month End Claims Data PHI Health Program Reports Self-Service Reports Sales eBusiness Reports (Group Finance) MBF Life Reports Clinical & Business from Web Channel (excl. Financial Services & IT) Claims Reports Executives Commentary Balanced Score Card Analysis Reports (Weekly, Monthly, Ad-Hoc) Census Data Data (Via J drive) 2 4 5 4 5 MISC. PRODUCT BUSINESS OBJECTS CRYSTAL REPORTS/ MISC AD-HOC HEALTH SAS PRODUCTS MISC WEBSITE MISC REPORTING CUSTOMER CUSTOMER FACING CRYSTAL REPORTS META CUBES XRAY SPSS FUTRIX TM1 CENTREVU ANALYSIS 5.9 CRYST. ENTERPRISE DATABASES MANAGEMENT SUITE ANALYSIS TOOLS DATABASES EXPERIENCE DB SYSTEM (CFS) REPORTING Reports Formatting Analysis Cubes. IBM. PROGRAMS Based On SAS AF. - Customer Profiling Aka Online Cubes for - Contract Modelling & Avaya IVR Reporting Data in Crystal Around 50 MS Access - Base SAS (H&B Stats) MS Excel/Access (MS Access Database) Customer Experience GTX-PROD Database Research. Data Benchmarking - Data Mining - Enterprise Miner Self-Service Usage. Reporting. Call Tracking System - Customer Analysis Decommissioned Analysis Cubes Enterprise. Databases Preventive, Asthma & Providers Against - Statistical Analysis - Self-service & Satisfaction Surveys. (Graham Technology) (Data Mining & Modelling) - Competitor Analysis in favour of & Data Extraction. Use of DW Diabetes Programs. Industry Profiles & Analysis & Reporting - Based On Excel. Used by Customer (MS Access Database) - Enterprise Guide - Investment Business Objects XI is Recommended Updated Monthly Trends - SAS Stat Service & Retention Call Center Data Performance Tracking excluding for (MS Access Database) - SAS Graph Written In SAS Used By Customer Operational Reporting Service & Retention Clinical & Business Activity & Clinical & Business Analysis Data Analysis Data Analysis Data Manipulated Clinical & Call Centre Data Customer Retention (Cancellations) & Sales Data (via MS Access) Manipulated Clinical & Business Analysis Data Business Analysis Data Customer, Brand & Product Data Service Call Survey Data Claims Data PHI Research Agencies 1 Sales Reporting By Channel Client & Product Data Tasmania Sales & EOM Provisioning (via MS Excel & Access) Prosthesis Pricing Data (Monthly) Client & Product Data Membership Data Levies, Revenue & Reinsurance Data (Monthly) Medical Pricing Data (Monthly) Misc. Reports MS Access Databases Funds Data (from Diamond) Sales & HCP Data CUSTOMER MBF LEAD OPERATIONS OBSC PORTAL MISC SALES & MISC. AD HOC TM1 MISC HOSPITAL MISC ANCILLARY MISC PROSTHESIS MISC MEDICAL MISC. MBF HEALTH IMPROMPTU MISC PRICING DATA Membership WATCHLIST REFERRAL SYSTEM TRANSFER Data MEMBERSHIP DB REPORTING DATABASES DATABASES DATABASES DATABASES REPORTING TRACKING SYSTEM Oracle (from DW) Extract, Match, Format 3 Cubes: Data Summarization - MBF Prothesis Price Retention Campaign Tracks Leads Balance Score Card - MS Access - Investment - Claims 2003 - 3 Claims - Dental - New Prices04 - Duplicate Services (MS Excel) & Reporting. List 2004 Data. Transferred From (MS Access Database) Membership Data (Monthly) Ad-hoc Data Extract - Claims 2004 - 6 Monthly Reports - Schedule 5 V2000 Tracks External Funds Web Front-End. - MS Excel - FTE Part Of Cognos Suite. - CMBS 2000 Tracks Reasons Why Member Centers To Managers’ Rollovers. - 7 State Sales DB Membership Reporting - Oracle GL - Claims Fy04 (Ancillary, Dental - File Upload, Master - Fy04 Gap Survey Data Extraction & - AMA Customer Redeem Clearview. Assirt V6 - 7 Weekly Member DB - Claims Fy04 V97 Optical, Pharmacy, - Human Tissue Items - Contract Modelling Formatting - Diagnostic Schedule Clearview Funds. (MS Access Database) (MS Access Database) - 7 Month Member DB Based On Excel. - Hospital Physio, Membercare) - Deleted / Replaced - CHC931 Medical (MS Access Database) Commonwealth (MS Access Database) To Be Phased Out. - Misc DB: Sales Rep... - Quarterly Report Items - Price Summary Claims Streamlined Department of Health Client & Product Data Membership & Claims Data Extract (from Diamond) Customer, Claims, Membership Movements, Gains and Exits Data & Ageing Client & Product Data Membership & HCP (Monthly), Ad-hoc Provider Contract / Modelling Data (from DW via Text file) Sales Data HCP (Monthly), Ad-hoc All Providers Claims, Financial, Membership Data (via Flat file from DW / Mainframe Domain) Ad-hoc Financial Data Extract (from DW / Mainframe Domain) Financials & Balanced Score Card Data Customer, Claims, Hospital Price Modelling & Ad-hoc Data Membership & Sales Data Medical & Prosthesis Pricing (Monthly) Industry & Regulatory Data (Quarterly) Integrated Data Domains Sales & Membership Report Query Data Data (via Flat File) Hospital To Be Modelled MAXIMISER Hospital Details 4 5 - Hospital Details from Hospital & FINTECHNIX SALES WEB-BASED DIAMOND RPT Benefit Management - Contract Modelling Private Health Insurance Synchronised Data REPORT SERVER OPPORTUNITY CORPORATE SAS DATABASE EAP37 Maintained By Account MANAGEMENT DOCUMENTS Replicated Diamond Hospital Details Administration Council (Weekly) Replicated Fintechnix Misc Data Environment for Summarisation & Managers. Environment For (SOM) SERVER (PHIAC) Predefined Corporate Reporting & Testing Census Data Reformatting. Reporting (Lifeprod). CRM Siebel Database Key Reports. - Prosthesis To Be Fed by Diamond Life, Superannuation, - Hospital Ancillary & Commissions Extract (Monthly) Managed Funds Data 3 Claims are Updated. - Medical - Ancillary Synchronised Data - Customer/Membership 3M GROUPER Sales & Membership Data By State excl. Tasmania (via Text File) HCP, Providers Misc. Data Misc. Data HCP & Claims Data Propensity Modelling Efficient Investment Data (via Excel) Regrouped DRG Matching & Cleansing Data Warehouse Standard Claims & Membership Data Outsourced Mainframe Domain (Subject Areas) Regrouped DRG Modelling Agency FTE (via Excel), (HCP) HOSPITAL CASE MIX GL Revenue, Claims & CORPORATE PROFITABILITY REINSURANCE WEEKLY SALES CLAIMS Reinsurance Provision Details Journal PROTOCOLS Expenses, Account Balances (Monthly) Misc Data Entry 1 Business Solutions - DW Balanced Score Card Data (via Exchange of Flat File) Synchronised Data Oracle Financial Suite (Group IT) Claims & Revenue Data ORACLE BALANCED ORACLE ITEMS DATABASE Diamond Domain (Subject Areas) SCORE CARD FINANCIALS (OBSC) Financial New Items MGM REINSURANCE Synchronised - General Ledger Item/Category Data MEMBERSHIP ACTIVITY CLAIMS Data (Provided by IT-DW) National Custodian (MEMBERSHIP GROSS MARGIN) Data - AP / AR (via ETL) To Be Matched to Services (NCS) Replacing Corporate Profitability - Asset Management Category of - Purchasing Procedures. Updated Monthly Contract Administration and Support Applications Effective Health Programs Data (Monthly) Revenue Data Clinical & Business Analysis Data (Monthly) FINTECHNIX HEALTH INSURANCE WHICS DIAMOND Hospital Provider Claims Data (Monthly) Clearview & MBF Life SYSTEM MBF Health Policies Client & Product Data Membership, Product, (HIS MAINFRAME) Ancillary Provider Claims Data (Monthly) Data Account Payable, Membership, Claims, Prosthesis Provider Claims Data (Monthly) Claims, Provider, Premium Billing, Travel, Revenue, Sales – Other, Health Claims & Medical Provider / Hospital Pricing Data (Monthly) Bemis/DDS, Customer EOM Aggregate Customer Feedback Service Rewards Hospital Data Notes Data Flow Mode Business Unit Ownership Uncategorised Internal File Legend 1 Not being used Enterprise Business Reporting & Analytics (DW Strategy) Point To Point Email GROUP IT & FINANCIAL BUSINESS GROUP FINANCE & GROUP GROUP STRATEGY BUSINESS & MBF HEALTH PRIVATE HEALTH RISK UNCATEGORISED Current Applications Model Business Unit Classification PROGRAM SERVICES INTELLIGENCE CORPORATE MARKETING AND NEW CLINICAL INSURANCE (PHI) MANAGEMENT & Version 1 Belongs to PHI Description PHYSICAL DATABASE Middleware Fax MANAGEMENT - Clearview Belongs to SERVICES Description BUSINESS ANALYSIS Description TREASURY Effective Date: 13 July, 2005 Sales Reporting DEVELOPMENT Description File: Copy (1) of MBF Business Intelligence Current Applications Model - BU (1.2).vsd File Transfer - ITS - MBF Life Group Finance & Description Belongs to PHI Eg. Investment & is around Web Contact: Paul Ormonde-James, MBF - Business Intelligence - (02) 9323 9783 - Business Solutions/ Corporate Services Description Actuarial Author: Marie-France Gara & Tamer Galil, Chartres Business Solutions (Email [email protected]) All Products Voice Hard Copy Data Warehouse (see Business - HIT, ... This model has been prepared with due care from the information made available, The Chartres Path® Behavioral Model) Screen External Memory and is a true representation of the subject area as it was presented to the Author. Commercial in Confidence Duplication, Confusion, Inefficient Federated Service Model
- 12. Questions Regarding Data Ownership • Unless there is a set of clearly defined data ownership and stewardship policies, there are bound to be some questions regarding responsibility, accountability and authority associated with auditing and reviewing the quality of data sets. • This needs to be a collaboration between business and technical IT • Needs to have the business understand the uses of the data
- 13. Application-Based Data Management • The systems and applications that comprise an enterprise environment may be structured in way that the business manager for each system has authority over the information used within that system. • Consequently, each application in isolation has its own requirements for data quality. • In the development of an enterprise architecture, it is possible that application data may be used in ways never intended by the original implementers. • This, in turn, may introduce new data quality requirements that may be more stringent than the original, yet there may be hesitation by the application teams to invest resources in addressing issues not relevant within their specific applications.
- 14. Administrative Authority • In some instances, the information used in an application originates from a source that is outside of the application manager's administrative authority. For example, in an application that is used to aggregate information from many information partners, many of the data quality issues are associated with problems at the partner level, not within the aggregated system. • Because the problems occur outside of the centralized administrative authority, even if the data is modified/corrected at the centralized repository, it does not guarantee that the next submission would not still include instances of the same problems.
- 15. Data Quality in an Advisory Role • In application-oriented organizations, another impediment to data quality coordination relates to how one deals with improving the quality of specific data used within an application when that data is sourced from an external data supplier, and is consequently managed outside of the application manager's jurisdiction. • Although in some organizations the project structures may already have an associated data qualtiy function, the more important issue is whether in practice all participants will cooperate with the data quality improvement process.
- 16. Data Quality as a Business Problem • In many organizations, business clients assume that any non compliance with expectations results from data quality issues and needs to be addressed by the technical teams. • However, in reality, the business rules with which the data appears to be noncompliant are associated with the running of the business. • Consequently, those rules should be owned and managed by the business client as opposed to the technical team members, whose subject-matter expertise is less likely to be appropriate to address the problems.
- 17. Impact Analysis • Anecdotal evidence may frequently inspire attitudes about requirements for data quality. • However, in the absence of a true understanding of the kinds of problems that take place, the scope of the problem and the impacts associated with the problems, it is difficult to determine the proper approach to fixing the problem as well as eventually measuring improvement. • There must be cooperation between business and IT to determine the scope of the problem
- 18. Reactive versus Proactive Data Quality: • Most data quality programs are designed to react to data quality events instead of determining how to prevent problems from occurring in the first place. • A mature data quality program determines where the risks are, the objective metrics for determining levels and impact of data quality compliance, and approaches to ensure high levels of quality. • Once again the business and IT must work together to deliver this approach
- 19. Personal Objections • While it is unlikely that any individual would specifically disagree with any of the data quality concepts that constitute an effective improvement program, that does not necessarily guarantee any specific individual's participation. Providing a clear business case that demonstrates how specific data quality issues impede the stated business objectives, as well as a discussion of the steps that need to be taken to address the problem, will make the decision to introduce the improvement very clear. • Must be some guiding principles for development and approval of data quality guidelines, • as well as approaches to integration of the best practices into the enterprise. Our approach has been to adapt data quality best practices within a documented guideline structure that conforms to internal policies and standards and can be approved through internal procedures. When a particular activity has been approved through the standard internal channels, it transitions from "guidance" into organizational policy, with all the compliance requirements that implies.
- 20. What is metadata….. • As the term implies, metadata is data about data. • Metadata is, of course, data as well, and has its own metadata (meta-metadata), implying that metadata exists at many levels of granularity. • Metadata is generally defined in terms of usage as technical or business metadata. – Technical metadata describes the technical aspects of the data such as field type, length, allowable values, and so on. – Business metadata describes the business context of the data, such as: “This forecast data was submitted in march 2005 and includes revenue....”
- 21. Metadata – View by pattern: • Syntactic metadata: – Data describing the syntax of the data; for example, language syntax, format • Structural metadata: – Data describing the structure of the data; for example, document structures, DTDs, XML schemas, relational schemas • Semantic metadata: – Data describing the meaning of the data in a specific domain, such as definitions of “product,” “accounting period,” “total revenues”
- 22. Metadata – Another key concept is ontologies • Data that formally specifies how to represent objects, concepts, and other entities in a domain of interest, as well as the relationships that bind them; for example, a taxonomy of typical data domains such as “product” and “market” and their relationships. • Whereas taxonomy is simply the hierarchical structure or classification of the data (e.g., “product” attributes and their structure), ontology is an overarching term that includes not just the taxonomy, but also the relationships and mapping between the attributes. • • Therefore, ontologies will include not just entity structures and relationships, but acceptable values of the entity attributes, their synonyms, word variants, abbreviations, etc. • Industry standard ontologies exist for common data domains (such as “product,” “country,” etc.) and provide a pre-built base of metadata for an enterprise. • These ontologies can be adapted for specific enterprise needs, and mappings created for data structures across enterprise entities or divisions
- 23. Why Should Enterprises Care about Metadata Integration • Data in a typical organization is scattered across multiple databases, files, and networks. To link data from disparate sources, enterprises have developed many levels of integration: business process integration, UI integration, application integration, etc. • However, what is most critical to effective decision making and innovation is that the integrated data be accurate, complete, consistent, and timely, and be insulated from changes in underlying data structures. • Metadata is what decouples integrated data from changes in granular data sources. Further, this metadata is most effective when shared across the enterprise, requiring enterprise metadata integration.
- 24. A Prerequisite for Defining an Enterprise Metadata Strategy • Before you can think of implementing metadata, you need to build an awareness of data integration needs. • Typically, an organization uses data either for operational needs or for business intelligence. For operational systems, source data may be relatively homogeneous (syntactically, semantically, and structurally) and data integration may not require extensive metadata validation. • However, in complex data flows between operational systems, or systems providing business intelligence, metadata assumes much greater significance. Data typically flows through distinct steps in the organization’s information supply chain: – · Acquisition: acquiring data from multiple sources – · Consolidation: normalizing data within a given source – · Aggregation: integrating data across multiple sources – · Dissemination: delivering data to end users • If these steps are not supported by an underlying metadata infrastructure, disseminated data will be degraded, inconsistent, inaccurate, and lack integrity. Therefore, it is imperative to identify sources of record for given data domains and optimize data platforms to serve one or more of the above functions.
- 25. A Prerequisite for Defining an Enterprise Metadata Strategy • Key Requirements for successful metadata – Checklist. Effectiveness Provide users more useful information than they currently get Allow metadata to be extended easily to reflect the addition of Extensibility business such as a new product or market Reusability Provide reusable metadata across different business functions in an organization Connect to disparate systems to glean metadata. The solution Interoperability should also provide APIs or services to access metadata by different systems Efficiency and Process a user’s metadata request in a timely and efficient manner Performance Allow metadata built around one business unit or function at a time Evolution to be promoted to the enterprise Entitlement Allow delegated administration, data entry, and access
- 26. A Prerequisite for Defining an Enterprise Metadata Strategy Allow for changes to the underlying data structures without affecting Flexibility existing business processes Distribute and segregate metadata into several sub-systems, Segregation maintained at a business-unit level User interface Provide a friendly user interface Versioning Control metadata versions Deal with different types of metadata such as syntactic, structural, Versatility semantic, and ontological Low maintenance Provide ease of administration and a self-maintenance model cost Store metadata for semi-structured or unstructured data; transform Others metadata into different formats: human readable, machine readable, etc.; and analyze patterns and detect inconsistencies in stored metadata
- 27. Usability - What and Why • Usability is the "make it or break it" of any BI initiative. Just because a BI tool is in place does not mean people will use it. If people do not use it, it does not matter how well the solution is designed or developed. • The solution should have an intuitive user interface, enabling users to get the information they need with just a click - without calls to tech support. The BI tool should make operations easier, not more complicated. Ease-of-use drives early user acceptance and early adoption. An intuitive interface minimizes training and reduces costs in the long run. Most importantly, the simple interface maximizes use, so decision-makers will be more likely to use the system on day one. • A usable BI solution will provide the ROI it should. The initiative must result in delivering something that can be used immediately to resolve today's problems and assist in accomplishing the business goals that have been set.