Downloaded 29 times
Uploaded byMark Tapley
Building a business intelligence architecture fit for the 21st century by Jon Page
The document discusses the importance of building a business intelligence (BI) architecture that aligns with the demands of the 21st century, emphasizing the need for cooperation between business and IT. It outlines critical components, including the integration of technology to manage data effectively, the development of a coherent BI strategy linked to corporate strategy, and the establishment of principles for improving data accessibility and trustworthiness. Additionally, it highlights the need for systematic frameworks and a focus on priorities in data management to address challenges faced by organizations.
More Related Content
Similar to Building a business intelligence architecture fit for the 21st century by Jon Page
Building a business intelligence architecture fit for the 21st century by Jon Page
- 1. Building a Business Intelligence Architecture Fit forthe 21st Century Jon Page
- 2. This seminar isbased on the Contents of this book Building a The premise is that: Business BI is not delivering what is Intelligence needed Architecture Fit Current BI architectures are for the 21st not appropriate Century Business and IT need to work Together on agreed strategies Technology must be used to enable the reduction of copy management and the support of real time accuracy Jon Page of critical data People and organisation are Key to success in BI
- 3. Our Objectives To recordsome history – what has happened in the past that makes the future quite challenging. To provide real examples of BI at work – good and bad. To illustrate the nature of data and why it has become so important in driving forward the business in the 21st century. To outline a way to align technology with the business so that efforts and budget are spent in a way that will enable the future rather that support the past. To propose a set of principles and ideas that can guide a company in a way to make data available to all who have the penchant to turn it into useful and valuable information. To describe the new organisation unit that will be needed to realise the dream.
- 4. The Changing BusinessClimate Globalisation Deregulation Technology CHOICE
- 5. Competition : Veryaggressive
- 6. To Be Successful– Two Critical Components Intelligence Blue Chip The Winners ! Out Of Business Out Of Business (slowly) (fast) Speed
- 7. Maximising Customer Value Full Potential Relationships Number of Current Current Customer Value Current Relationship Duration
- 8. Imagine You CouldDo This... Profitable Not Profitable Now Future
- 9. BI is anafterthought BI is reporting BI is not taken seriously Everyone does their own ‘bi’ Lots of tools, lots of platforms No-one trusts the data No-one understands the data No-one shares information No-one owns the problem Business and IT don’t work together There is no BI Strategy And on, and on, and on
- 10. HELP!! Customers Get more Technology – and fast Get some technology Time
- 11. Data Warehousing Terminology Strategic Information Data Sandpits MIS Systems E-Intelligence TNF Data Warehouse DSS OLAP Dimensional Star Schema ROLAP Business Snowflake Intelligence MOLAP Data Mart
- 12. BI technology S-Curve Data Provisioning Product Performance Platform Enterprise Data Warehouse Independent Data Mart Executive Information System Time or Engineering Effort
- 13. Executive Information System(EIS) Dimensional Proprietary PC based (if you’re lucky) Product Performance • Very few users, typically management • Standard reports, usually monthly • Often run on production system • High IT requirement • Full centralized control, minimal user flexibility • No persistent analytic data store • Minimal data integration Time or Engineering Effort
- 14. Data Warehousing: IndependentDMs Dimensional (star schema) OLAP SMP based Product Performance • Rise of the power user • Query capability and analysis added • Dimensional systems; typically departmental • Introduction of BI tools and normalization for more user flexibility • Rise of data redundancy and synchronization problems • Heavy management overhead • Data integrated from multiple sources via ETL Time or Engineering Effort
- 15. Data Warehousing: EnterpriseDW • Eliminates redundant data; single version of the truth • Ad hoc query and cross-functional analysis • Reduces ETL costs; continuous data loading possible • Re-centralizes control • Integrates data fromon a single data model requires agreement multiple sources, but • Complex to manage, not adaptable to change Product Performance Normalised ROLAP, MOLAP MPP Time or Engineering Effort
- 16. Data Provisioning? OperationalBI?? Cloud??? ??? ? Product Performance Time or Engineering Effort
- 17. Choosing the RightArchitecture
- 18. When to builddata marts? • Geographical considerations. • Specific performance requirements. • Specific availability requirements. • Departmental control over reports and queries. • Specialized applications. • The Three P’s • Performance, Politics and Packages
- 19. DM Topologies –Independent DMs Advantages • “Fast” implementation. • Quick ROI. • Departmental control. • Not reliant on IT for data. Disadvantages • Multiple data models. • No consistent corporate data model. • Multiple interfaces to manage/maintain. • No single version of the truth. • Duplication of data.
- 20. DM Topologies –Dependent DMs Advantages • Single version of the truth. • Clean/scrubbed data. • Consistent data model. • Robust data transformation. Disadvantages • Must have an existing data warehouse. • Must fit with corporate strategy. • Duplication of data.
- 21. DM Topologies -Dependent DMs If you have an EDW and you want to propagate data to Data Marts, you have a choice: 1) Propagate data to distributed Data Marts... è Original data is stored in EDW. è Propagated data is stored in different physical sites. Table 1 Table 2 Table 3 2) Propagate data to integrated Data Table 4 Table 5 Table 6 Marts... è Original data is stored in EDW. Data Mart #1 Data Mart #2 è Propagated data is stored in T1 T2 T1 T2 separate “databases” within the T3 T4 T3 T4 EDW environment
- 22. DM Topologies -Integrated DMs If you have an EDW and you want to achieve the Data Mart result without creating more databases to maintain, you can create Materialised Views within the EDW. èThe data remains stored in the EDW. èMV’s can provide superior performance. èApplications can use logical model of the data. Table 1 Table 2 Table 3 JI #1 AJI #2 Table 4 Table 5 Table 6
- 23. DM Topologies -Federated DMs IT Users Operational Data Data Transformation Enterprise Warehouse & Management Visual/Intelligent/ Magic Joiner Business Users
- 24. Question??? What do allof these architectures have in common?
- 25. Failure is AlwaysBlamed on Technology • The Computer is too slow • The Database can’t scale • SQL is not powerful enough • The Database doesn’t have the functionality I need • The data is not to be trusted (not upto date) • Nearly always technology is blamed for faults in people/organisation • There’s really nothing wrong with technology today...
- 26. Moore’s Law Illustrated 1000000 ‘Every 20 years the cost of processing a single bit of 1000 information declines by a factor of 1,000’ 1990 = 1 1 0.001 1950 1970 1990 2010
- 27. >1200 MIPS Merced 1000 Pentium II ~500 Pentium Pro ~321 Pentium 100 100 80486 20 2010 10 • 100,000 MIPS 80386 • 4 GHz., .07u 6 • 1B transistors 80286 1 8080 8088 0.9 0.1 4004 0.64 0.64 0.01 0.06 1975 1980 1985 1990 1995 2000
- 28. Questions? “Even if you’reon the right track, you’ll get run over if you just sit there.” Will Rogers
- 29. Building a Business Intelligence Architecture Fit forthe 21st Century Part 2 Jon Page
- 30. So it seemsthat we still have a problem? The trouble is that BI doesn’t stand alone – it feeds some very complex processes
- 31. Linking BI Strategyand Corp Strategy WHY??? • Raise visibility of BI • Build required organisation • Ensure BI solves REAL business issues • Qualify and Prioritise BI activities • Get funding • Get recognition for a job well done
- 32. Corp Strategy LooksSomething Like This: Vision Goals Objectives Strategies Project
- 33. Global Information TechnologyPlanning Projects can be broken down into Facets: Strategies Projects Process Technology Applications Organisation Information (Now we’re moving into BI Strategy)
- 34. Vision The Information Facet Goals can be broken down into Business Objectives Questions: Strategies Project Organisation Technology Process Application Information Business Questions
- 35. Vision Business Questions can Goals be broken down into SQL and Data Objectives Strategies Project Data SQL Organisation Technology Process Application Information Business Questions
- 36. Vision Data and SQL can be Goals regrouped into meaningful and discreet BI ‘tasks’ Objectives Pebble Pebble Pebble Pebble Strategies Project Data SQL Organisation Technology Process Application Information Business Questions
- 37. Vision Value Goals Time Objectives Pebble Pebble Pebble Pebble Strategies Project Data SQL Organisation Technology Process Application Information Business Questions
- 38. What is aPebble? They are opportunities that: can be enabled by a technology based solution need to exploit information can be realised in no more than six months are well understood by business and technology are of high priority are measurable in terms of success whether this be ROI, TCO or some other internalised mechanism have a descriptive and meaningful name. solve at least some part of the information facet of a project. Perhaps the most important characteristic of pebbles is that they support agreed and strategic projects
- 39. Real ‘Pebbles’ • Managementof churn in a specific segment of customers – let’s reduce churn by 5% in our customers who make more than 5 calls per day • Revenue recognition – lets make sure that we improve by 5 points the ratio between service usage, billing and collection across the board • Increase acquisition rate of customer in a specific segment – let’s increase rate by 2% for corporations with 50 to 300 employees. • Decrease fraud – let’s reduce subscriber fraud by 3 points across all customers • Understand product affinity – lets’ determine which products are bought in combination to allow us to more effectively deploy ‘loss leaders’.
- 40. Let’s reduce churnby 5% in our customers who make more than 5 calls per day • Questions (SQL): • Who makes more than 5 calls per day • What is the churn rate in this segment? • What do the churners have in common? • Who in the current segment looks like these churners? • How can I keep them? Bills Subscriber Contracts attributes Calling Circle Payments CDR’s
- 41. Global Information TechnologyPlanning Process Applications Organisation Technology Information Pebbles Business Questions Critical Data Supporting LDM PDM
- 42. Global Information TechnologyPlanning Prioritising Pebbles High High/Low High/High High Strategic Business Value High Strategic Business Value Low Need for IT Support High Need for IT Support Mean V Low/Low Low/High a l Low Strategic Business Value Low Strategic Business Value u Low Need for IT Support High Need for IT Support e Low Mean High Difficulty
- 43. Global Information TechnologyPlanning High Revenue Customer Assurance Customer Profitability Risk Segmentation Analysis Statutory Usage Reporting Analysis Mean Fraud Basket V Detection Analysis a l Competitor u Assessment e Sales Reporting Low Mean High Difficulty
- 44. Competitor Revenue Assessment Assurance Usage Customer Customer Risk Fraud Analysis Segmentation Profitability Analysis Detection Sales Statutory Reporting Reporting Q1 Q2 Q3 Q4 Q5 Q6
- 45. Let’s look ata worked example Workbook 1
- 46. BI Strategy Documentsand Owners Key Information Imperatives – owner Information Master Business Intelligence Roadmap – owner Information Mast BI Pebble Definitions – owner Information Manager Data Dictionary – owner Information Master Skills Matrices – owner Information Master or Information Manager as relevant. Security Policy – DBA Access Policy – DB Information Architecture – Information Architect Data Reuse Map – Database Designer Logical Data Model – Database Designer Physical Data Model – Database Designer User Requirements – Information Creator Various technical documents - various Various Service level Agreements – Information Manager/Master as appropriate
- 47. Building a Business Intelligence Architecture Fit forthe 21st Century Part 3 Jon Page
- 48. What Have WeLearned So Far? BI has not matured as fast as we maybe believe The gap between what we need and what we have is getting bigger The goal is to be able to predict and maybe change the future Business and IT MUST work together A BI Strategy linked to Corp Strategy is vital Contemporary BI platforms – Data Marts and Data Warehouses are a good start but that’s all. They’re: Full of useless inaccurate data No one knows where the data came from or how old it is It’s copy miss-management gone crazy We need something that: Is trusted Manages a single copy of master data Can provide different ‘types’ of info – strategic, regulatory, ad-hoc, DSS, history, trend Recognises that data is of many natures
- 49. Pay Receive Design Make Distribute Retail Sell Distribute Bill Pay
- 50. Pay Receive Design Make Distribute Retail Sell Distribute Bill Pay Research and Promote and Sell Collect Design Logistics Logistics Manufacture Bill
- 51. Pay Receive Design Make Distribute Retail Sell Distribute Bill Pay Procurement Projects Billing Manufacturing Fraud Detection Applications Network Optimisation Requirements Contracts Assets and Liabilities Analytics Engineering Marketing HP CRM Accounts Recievables Revenue Assurance Question 3 – What are the main differences between these two sets of data?
- 52. We need tobe able to store and manage many different types of data automatically Transactional, detailed, historical - vast volume Back-office, operational Data - on-line? Master Data - on-line Regulatory Data - accurate Summarised Data - automatically created KPI’s - automatically maintained – complex algorithms Metadata - in one place, meaningful
- 53. What We Needto do Now: • Build a single BI Platform that: • Stores and Manages Critical Data (Master Data) in a real time way and as a service to applications • Stores and makes available detailed, historical, transactional data • Manages automatically all summary data • Provided fresh KPI’s • Is the basis for trusted and repeatable regulatory reporting • Services ad-hoc complex queries and transactional queries • Is self managed
- 54. Collaborative Information Blueprint Somefundamentals: It depends on collaboration between Business and IT It needs a BI strategy as described in Part 2 It builds on what we’ve learned in Data Warehousing It is an incremental strategy It is a centralised, data driven initiative It’s a set of guiding principles rather than a methodology Depends on a centralised platform that is TIGHTLY coupled with the operational world
- 55. Steps 1 to4 of CIB Plan Define Key Information Define/Prioritise Imperatives (KIIs) BI Pebbles (BIPs) Identify Critical Data Elements (CDEs) and build Data Model Determine Data Rules Implement
- 56. Step One –Information Imperatives We must deploy all reporting across the web. We will rigorously abide by all pertinent standards at all times. We will keep a single copy of important data accessible to all. Wherever possible we will pursue a single vendor strategy. We will not use bleeding edge technologies to support core processes. We will forbid the copying of data unless suitably authorised. Non-compliant applications will only be used if a pathway to full compliance is possible within an 18 month window. We will aggressively centralise all important data.
- 57. Step Two –Determine and Prioritise Pebbles What is a Pebble? They are opportunities that: can be enabled by a technology based solution need to exploit information can be realised in no more than six months are well understood by business and technology are of high priority are measurable in terms of success whether this be ROI, TCO or some other internalised mechanism have a descriptive and meaningful name. solve at least some part of the information facet of a project. Perhaps the most important characteristic of pebbles is that they support agreed and strategic projects
- 58. Global Information TechnologyPlanning High Revenue Customer Assurance Customer Profitability Risk Segmentation Analysis Statutory Usage Reporting Analysis Mean Fraud Basket V Detection Analysis a l Competitor u Assessment e Sales Reporting Low Mean High Difficulty
- 59. Real ‘Pebbles’ • Managementof churn in a specific segment of customers – let’s reduce churn by 5% in our customers who make more than 5 calls per day • Revenue recognition – lets make sure that we improve by 5 points the ratio between service usage, billing and collection across the board • Increase acquisition rate of customer in a specific segment – let’s increase rate by 2% for corporations with 50 to 300 employees. • Decrease fraud – let’s reduce subscriber fraud by 3 points across all customers • Understand product affinity – lets’ determine which products are bought in combination to allow us to more effectively deploy ‘loss leaders’.
- 60. Step Three) IdentifyCDE and Model Customer Invoice Bill Office Payment Supplier Product Credit Employer Score BIP1 y y y Y BIP2 y y Y BIP3 y y y y y Y BIP4 y Y y y BIP5 y y Y BIP6 y y Y Total 5 4 2 1 4 2 3 1 1
- 61. Now We HaveA Clear Justifiable View on what is OUR: MASTER DATA – (working definition for this seminar!!!) The Data That Supports Most of our Critical BI Requirements (Pebbles) This is KEY – because I propose that we store this data (at least), in just one place and one place ONLY!
- 62. PAYMENT ACCOUNT Payment No Account No Account No INVOICE Payment Amt Account Type Invoice No Location Date Opened Account No Date Invoice Amt ACCOUNT OWNERSHIP PRODUCT USAGE Account No Cust No Prod No Cust Role Cust No Start date Usage Date PRODUCT CUSTOMER Prod No Category Cust No Date Created Date of Birth Retail Price Title Gender
- 63. Step 4 -Define Rules Around CDEs • Sample Business rules may include: • Account Open Date must be 12 years later than Birth Date of Account Owner • Customer Name cannot be ‘null’ • A company must be allocated a SIC code from the published list • An account is ‘open’ if any type of contact has been associated with this account in the past 20 days • Address must always have a post code • Sample Technical rules may include: • Account Number must be 6 digits • ‘Male’ will always be represented as ‘M’ • Value must be capitalised • Value must be from following list…… • All table names should be singular • Process rules may include: • If there are more than ten of these marked ‘O’ then…. • This attribute is the result of function X on data item Y • This cannot be deleted whilst…..
- 64. After Phase Onewe have: • A prioritised map for BI implementation • A set of IT Imperatives to guide architecture, infrastructure, purchasing etc • A clear definition of Master Data, including rules etc • A core Logical Data Model
- 65. Steps 5 to8 of CIB Implement Select and Deploy Critical Data Platform (CDP) Enable Compliance Implement Pebbles Incrementally Build end-user layer Automate and Secure
- 66. Step 6 -Technology must be selected to create the Critical Data Platform (CDP), and this will include at a minimum the: • Computing platform (hardware) • Operating System plus management tools • RDBMS • Communications capability • Business Intelligence tools (reporting, ad-hoc query, mining, OLAP) • Integration tools • Metadata Management tools/capability • Backup and Recovery capabilities • Security software
- 67. Question 5: WHO CANPROVIDE THE PLATFORM? One ‘computer’ One Data Base CDE’s on-line to operational systems All data feeds automatic, in and out
- 68. Step 7 –Implement Pebbles Incrementaly Competitor Revenue Assessment Assurance Usage Customer Customer Risk Fraud Analysis Segmentation Profitability Analysis Detection Sales Statutory Reporting Reporting Q1 Q2 Q3 Q4 Q5 Q6
- 69. Initial Data Load Determine Data Implement Interface Test Acceptance Source Technology Physical Database Deployment of Develop Queries Implementation Integrity rules And Reports M1 M2 M3 M4 M5 M6
- 70. Step 8) EnableCompliance • Get the right answers • Get these answers quickly and reliably • Demonstrate how numbers are calculated (audit)
- 71. • The Implementphase brings into a single database: • Key shared data for enabling Corporate strategy (often called ‘Master Data’ but actually much more. • Operational data needed by multiple operational applications • Derived data for complex algorithms and scoring • Summary data often used to form the basis of Compliance calculations as Key Performance Indicators • Detailed, historical, behavioural data for deep mining and predictive analysis.
- 72. So in ourpreferred architecture, the tables we have modelled to represent our Critical Data Elements will be implemented in a single database and as a set of relational tables. • As these tables are deemed critical to the enterprise it is imperative that: • We have only one set of these tables in the whole enterprise. • These tables are shared by all applications that need this data whether these applications are internal or external, operational or analytical. • This data is protected from miss-use by a defined layer of metadata. • The data in these tables is always up-to-date. • The data in these tables records history.
- 73. Step 9 and10 of CIB Automate and Secure Ensure 100% Automate and Availability Secure
- 74. Clients Clients Clients Clients Primary Database A Cluster for Reliability?
- 75. Clients Clients Primary Standby Site Site Data Changes Broker Agent Broker Agent Data Guard Broker Primary Standby Database Database Remote Backup?
- 76. Source Systems Critical Data Elements BI Pebble Logical Model
- 77. Critical Data Elements, SourceSystems plus other as needed BI Pebbles BI Platform
- 78. Source Systems BI Pebbles Critical Data Elements plus other as needed BI Platform
- 79. Source Systems BI Pebbles Critical Data Elements plus other as needed BI Platform
- 80. Product Bundling and Business Pricing Value Credit Risk Analysis Profitability Analysis Behavioural Analysis Information Infrastructure Additional Data
- 81. Summary • Define andprioritise requirements – already done in our BI Strategy • Define first deliverable – already done in our BI Strategy (see BI Pebble Road Map) • Build Logical Data Model for first Pebble - already done • Build Physical Model to support initial requirement • Deploy rules as triggers, procedures, macros etc. • Define preferred provider system for each • Define interface requirements of each item of data with provider system • Perform initial data load to database • Deploy the required data feed technologies • Test system
- 82. How do wepay more for less • Tailoring pre-canned applications • Building our own logical data model • Not using the technology that we have already paid for • Buying data we already have • Issuing RFIs and RFPs • Evaluating identical technologies • Copying data
- 83. Summary • Build acomprehensive (but flexible) BI strategy linked to the Corp Strategy • Develop IT Imperatives • Put in place the right organisation • Cherish Critical Data • Build a closer but more demanding relationship with your vendors • Use technology • Promote yourselves – BI is the most important facet of most internal business
- 84. Building a Business Intelligence Architecture Fit forthe 21st Century Part 4 Jon Page
- 85. BI Projects failfor many reasons, we continually see: Insufficient dedication of resources (technical and most importantly, human). People with the wrong skills. Multiple, unprioritised roles and responsibilities. Lack of decision making power. Rudimentary future planning. Conflicting commitments. Too many on-going projects
- 86. A Real DWAudit findings: Lack of business involvement/ownership No three year roadmap for expected deliverables No formal Data Architecture or Logical Data Model Poorly defined job descriptions/roles – confusion of responsibilities Non-optimal deployment of Hardware Using out of date software No consistent application of Parallel Database features Many important Oracle facilities not being used effectively Lack of documentation Non-scalable environment Critical data was missing No deep usage of data in terms of analytics/data mining etc
- 87. Operationalise Define Idea Identify Data Create and Test Report LEARN! Estimate Cost Implement Interfaces Extend Models Go or No Go
- 88. The Information Board •Creation of Corporate Strategy • Ensuring alignment of BI Strategy • Providing support for deployment of CDP • Providing guidance and support to the BI ‘team’ • Solving critical issues • Arbitrating on internal conflicts • Monitoring return on Investment and/or Total Cost of Ownership goals • Appointment of the Information Master
- 89. The Information Master: isultimately responsible for the creation and maintenance of the following documentation: Key Information Imperatives BI Roadmap Data Dictionary The complete Business Intelligence Strategy Skills Matrices for the BI team
- 90. BI Strategy Documentsand Owners Key Information Imperatives – owner Information Master Business Intelligence Roadmap – owner Information Mast BI Pebble Definitions – owner Information Manager Data Dictionary – owner Information Master Skills Matrices – owner Information Master or Information Manager as relevant. Security Policy – DBA Access Policy – DB Information Architecture – Information Architect Data Reuse Map – Database Designer Logical Data Model – Database Designer Physical Data Model – Database Designer User Requirements – Information Creator Various technical documents - various Various Service level Agreements – Information Manager/Master as appropriate
- 91. Information Master Information Information Information Manager Sales Coord Creator Information Architect Integration Manager Project Manager Data Base Admin Data Base Designer
- 92. Information Board CEO Information CIO Master Business Users Information Information Information Manager Sales Coord Creator Partners Information Architect Interface Integration Data Base Data Base Project Manager Manager Manager Admin Designer General Project Manager
- 93. Information Board CEO Information CIO Master Business Users Information Creator Interface Integration Information Data Base Manager Manager Architect Admin
- 94. How do wepay more for less • Tailoring pre-canned applications • Building our own logical data model • Not using the technology that we have already paid for • Buying data we already have • Issuing RFIs and RFPs • Evaluating identical technologies • Copying data
- 95. Summary • Build acomprehensive (but flexible) BI strategy linked to the Corp Strategy • Develop IT Imperatives • Put in place the right organisation • Cherish Critical Data • Build a closer but more demanding relationship with your vendors • Use technology • Promote yourselves – BI is the most important facet of most internal business