IRM UK - 2009: DV Modeling And Methodology
Download as PPTX, PDFâ˘2 likesâ˘1,767 views
The document provides an overview of the Data Vault modeling methodology, outlining its benefits for enterprise data warehousing through historical traceability and traceable models that support various business functions. It discusses the challenges and issues faced in current enterprise data warehouse (EDW) projects and highlights the advantages of adopting a Data Vault approach, such as scalability, flexibility, and reduced need for re-engineering. Additionally, the document offers a step-by-step guide on how to build a Data Vault and includes insights from industry experts on the methodology's potential impact on data warehousing.
IRM UK - 2009: DV Modeling And Methodology
- 2. Data Vault Modeling MethodologyA PrimerâŚÂŠ Dan Linstedt 2009-2012All Rights Reservedhttp://LearnDataVault.com
- 3. A bit about meâŚ3Author, Inventor, Speaker â and part time photographerâŚ25+ years in the IT industryWorked in DoD, US Govât, Fortune 50, and so onâŚFind out more about the Data Vault:http://www.youtube.com/LearnDataVaulthttp://LearnDataVault.comFull profile on http://www.LinkedIn.com/dlinstedtLearnDataVault.com
- 4. What IS a Data Vault? (Business Definition)Data Vault ModelDetail orientedHistorical traceabilityUniquely linked set of normalized tablesSupports one or more functional areas of business10/6/2011LearnDataVault.com4Data Vault Methodology
- 5. CMMI Level 5 Project Plan
- 7. Peer Reviews, Release Cycles
- 9. Complete with Best Practices for BI/DWBusiness KeysSpan / CrossLines of BusinessSalesContractsPlanningDeliveryFinanceOperationsProcurementFunctional Area
- 10. What Does One Look Like?10/6/2011LearnDataVault.com5Records a history of the interactionCustomerProductSatSatSatSatSatLinkCustomerProductF(x)F(x)F(x)SatSatSatSatOrderF(x)SatOrderElements:Hub
- 11. Link
- 12. SatelliteHub = List of Unique Business KeysLink = List of Relationships, AssociationsSatellites = Descriptive Data
- 14. The PAIN!!Issues in Current EDW Projects10/6/2011LearnDataVault.com7
- 15. EDW Architecture: Generation 110/6/2011LearnDataVault.com8Enterprise BI Solution(batch)SalesStaging(EDW)StarSchemasComplex Business RulesFinanceConformed DimensionsJunk TablesHelper TablesFactless FactsStaging + HistoryContractsComplex Business Rules+Dependencies
- 16. Kick-Starting Data WarehousingHR Asks IT to build the FIRST Data Warehouse / Prototype10/6/2011LearnDataVault.com91.2.IT Says⌠OK: $125k and 90 daysâŚ3.HR Says:Great! Get Started
- 17. Everyoneâs Happy!IT Delivers. On-Time & In Budget!10/6/2011LearnDataVault.com104.5.HR Says:Thank-you! Weâre Happy!First Star!Customer_IDCustomer_NameCustomer_AddrCustomer_Addr1Customer_CityCustomer_StateCustomer_ZipCustomer_PhoneCustomer_TagCustomer_ScoreCustomer_RegionCustomer_StatsCustomer_PhoneCustomer_TypeCustomer_IDCustomer_NameCustomer_AddrCustomer_Addr1Customer_CityCustomer_StateCustomer_ZipCustomer_PhoneCustomer_TagCustomer_ScoreCustomer_RegionCustomer_StatsCustomer_PhoneCustomer_TypeCustomer_IDCustomer_NameCustomer_AddrCustomer_Addr1Customer_CityCustomer_StateCustomer_ZipCustomer_PhoneFact_ABCFact_DEFFact_PDQFact_MYFACTCustomer_IDCustomer_NameCustomer_AddrCustomer_Addr1Customer_CityCustomer_StateCustomer_ZipCustomer_PhoneCustomer_TagCustomer_ScoreCustomer_RegionCustomer_StatsCustomer_PhoneCustomer_TypeCustomer_IDCustomer_NameCustomer_AddrCustomer_Addr1Customer_CityCustomer_StateCustomer_ZipCustomer_PhoneCustomer_TagCustomer_ScoreCustomer_RegionCustomer_StatsCustomer_PhoneCustomer_Type
- 18. So Whereâs the PAIN?10/6/2011LearnDataVault.com11
- 19. The PAIN is RIGHT HERE!!Contracts Sees Success, wants the same for their systems.10/6/2011LearnDataVault.com121.2.IT Says⌠Ok, but⌠It wonât be $125k and 90 daysâŚBecause we have to âmerge itâ with HRâ it will be $250 and 180 days.3.Contracts Says:Ouch! Thatâs not reasonable, but we need it, so go aheadâŚ
- 20. And HEREâŚ.10/6/2011LearnDataVault.com13Finance, Sales, and Marketing want inâŚ.IT Says⌠Ok, but⌠It wonât be $250k and 90 days⌠Because we have to âmerge itâ with HR and Contracts it will be $350k and 250 days.And this continuesâŚ.Business Says...âCanât you just make-a-copy of the Star Schema, and give me my own for cheaper & less time?
- 21. Silo Building / IT Non-Agility10/6/2011LearnDataVault.com14First StarSALESWe built our own because IT costs too muchFINANCEWe built our own because IT took too longMARKETINGWe built our own because we need customized dimension dataWhy is this happening? Whatâs Causing this Problem?
- 22. Root Cause of Pain: Re-Engineering!10/6/2011LearnDataVault.com15IT is forced to Re-EngineerETL loading code + SQL BI Queries WHENEVER:WHENEVER table structures change
- 23. New systems are introduced1. Adding fields to DimensionsBusiness Rules Change
- 24. (causing ETL Loading to change, and forcing Engineers to RELOAD existing data)Customer_IDCustomer_NameCustomer_AddrCustomer_Addr1Customer_CityCustomer_StateCustomer_ZipCustomer_PhoneCustomer_TagCustomer_ScoreCustomer_RegionCustomer_StatsCustomer_PhoneCustomer_TypeCustomer_IDCustomer_NameCustomer_AddrCustomer_Addr1Customer_CityCustomer_StateCustomer_ZipCustomer_PhoneCustomer_TagCustomer_ScoreCustomer_RegionCustomer_StatsCustomer_PhoneCustomer_TypeCustomer_IDCustomer_NameCustomer_AddrCustomer_Addr1Customer_CityCustomer_StateCustomer_ZipCustomer_PhoneFact_ABCFact_DEFFact_PDQFact_MYFACTCustomer_IDCustomer_NameCustomer_AddrCustomer_Addr1Customer_CityCustomer_StateCustomer_ZipCustomer_PhoneCustomer_TagCustomer_ScoreCustomer_RegionCustomer_StatsCustomer_PhoneCustomer_TypeCustomer_IDCustomer_NameCustomer_AddrCustomer_Addr1Customer_CityCustomer_StateCustomer_ZipCustomer_PhoneCustomer_TagCustomer_ScoreCustomer_RegionCustomer_StatsCustomer_PhoneCustomer_TypeCustomer_IDCustomer_NameCustomer_AddrCustomer_Addr1Customer_CityCustomer_StateCustomer_ZipCustomer_PhoneCustomer_TagCustomer_ScoreCustomer_RegionCustomer_StatsCustomer_PhoneCustomer_Type3. Adding Dimensions to Facts2. Adding fields to Facts
- 25. Why Re-Engineering?10/6/2011LearnDataVault.com16Adding fields to a conformed dimensionâŚ.Adding fields to a shared factâŚ.Changing code to match new business rulesâŚRequire adding/changingFields in target tables!Require Re-Engineering!
- 26. Other Pains?10/6/2011LearnDataVault.com17Dimension-Itis?IT â Non-Agility?Deformed Dimensions?What about the âdataâ you donât see?What about the âBADâ data left in the source systems?
- 27. The SolutionGo the Data Vault Route!10/6/2011LearnDataVault.com18
- 28. EDW Architecture: Generation 210/6/2011LearnDataVault.com19SOAEnterprise BI SolutionStarSchemas(real-time)Sales(batch)DVEDW(batch)StagingErrorMartsFinanceContractsReportCollectionsBusiness Rules Downstream!(the Lens Filter)
- 29. Unstructured Data And Data Vault10/6/2011LearnDataVault.com20Unstructured Data SetsOntologies/TaxonomiesUnstructured Processing EngineEmail
- 30. Docs
- 31. Images
- 32. Movies
- 33. SoundOn-DemandCubesJoins through LINK StructuresData Vault EDW
- 34. IT Agility10/6/2011LearnDataVault.com21RAWâwhat-isâStarSchemasComplexBusiness RulesETL-TData Vault(EDW)SourceStagingBusinessDrivenStarSchemas2. Business Gap AnalysisUnknown TimeâŚ
- 36. Start new phase1. Fast Load & Fast Integration3. IT Implementation of Business Rules
- 37. What are the Facts Jack?10/6/2011LearnDataVault.com22Generation 1 EDWâs tried to provideâOne version of the truthâGeneration 2 (Data Vaults) provideâŚâOne version of the facts, for each point in time.â
- 38. Business Gap Analysis10/6/2011LearnDataVault.com23The Way Business Perceives itâs business to be runningGapAnalysisOperationalReportsGapAnalysisDynamicCubes(Data Marts)The way the source systems see the business running.
- 39. Secured/Protected Information Systems10/6/2011LearnDataVault.com24Non-Classified DVClassified Data VaultHubSatHubData CopyLinkLinkSatSatSatModel CopySatHubHubLinkHubSatSatSatSatSatSatSatSatYellow = New TablesModel changes are absorbed seamlessly into the classified system
- 40. Classified world can add all their own structures while maintaining congruence with standard unclassified Data VaultExtensibility Factor10/6/2011LearnDataVault.com25New AdditionsNew CodeBilledAmountsProduct ShippedDatesProductQuantitiesExisting EDWNo Impact!ProductSupplierLinkSuppliersProductsDescriptionsDescriptionsAddressAvailability DatesStock QuantitiesStock QuantitiesDefect ReasonsRating Score
- 41. Whereâs the Solution?10/6/2011LearnDataVault.com26Re-EngineeringHandle Changes Wherever⌠Whenever⌠with EASE!
- 42. The Three vehiclesâŚPros and Cons of the Modeling Methodologies10/6/2011LearnDataVault.com27
- 43. 3rd Normal Form Pros/Cons as an EDWPROS (as 3NF)Many to many linkagesHandle lots of informationTightly integrated informationHighly structuredConducive to near-real time loadsRelatively easy to extend10/6/2011LearnDataVault.com28CONS (as EDW)Time driven PK issuesParent-child complexitiesCascading change impactsDifficult to loadNot conducive to BI toolsNot conducive to drill-downDifficult to architect for an enterpriseNot conducive to spiral/scope controlled implementationPhysical design usually doesnât follow business processes
- 44. Star Schema Pros/Cons as an EDWPROS (as Data Mart)Good for multi-dimensional analysisSubject oriented answersExcellent for aggregation pointsRapid development / deploymentGreat for some historical storage10/6/2011LearnDataVault.com29CONS (as EDW)Not cross-business functionalUse of junk / helper tablesTrouble with VLDWUnable to provide integrated enterprise informationCanât handle ODS or exploration warehouse requirementsTrouble with data explosion in near-real-time environmentsTrouble with updates to type 2 dimension primary keysTrouble with late arriving data in dimensions to support real-time arriving transactionsNot granular enough information to support real-time data integration
- 45. Data Vault Pros/Cons as an EDWPROS (as EDW)Supports near-real time and batch feedsSupports functional business linkingExtensible / flexibleProvides rapid build / delivery of star schemaâsSupports VLDB / VLDWDesigned for EDWSupports data mining and AIProvides granular detailIncrementally built10/6/2011LearnDataVault.com30CONS (as EDW)Not conducive to OLAP processingRequires business analysis to be firmIntroduces many join operations
- 46. The Three VehiclesâŚWhich would you use to win a race?Which would you use to move a house?Would you adapt the truck and enter a race with Porches and expect to win?10/6/2011LearnDataVault.com31
- 47. #1 complaint about DV architectureSo you want to deal with Joins do you?10/6/2011LearnDataVault.com32
- 48. Joins, Everywhere!10/6/2011LearnDataVault.com33Yes, the DV is full of joins butâŚThese are highly normalized tables (thin & Narrow), reducing I/Oâs to read large numbers of rows, at high speed, in parallel. Joins occur in RAM instead of on disk. The Optimizer is given a chance to âdrop tablesâ from the join that arenât necessary.When Parallelism is too muchâŚNot enough CPU or RAM to handle the extra work-load
- 49. Not enough rows being queried, (the overhead of starting the threads takes longer than an original scan.End Result? The DV Scales to the Petabyte Levels when necessaryâŚ
- 50. Mathematics Behind the Data Vault Model*** The Data Vault is BACKED by Mathematical Principles***Parallel versus sequential execution modelsSet LogicI/O Bandwidth & ThroughputCompression (for query performance gains)Process Repeatability (tuning & predictability measurements)RAM versus electromagnetic disk (Solid-State Drives are not measured)http://osl.cs.uiuc.edu/docs/IPDPS-TR04/TCA_TR04.pdf10/6/2011LearnDataVault.com34
- 51. Know when to hold âem, know when to fold âemWhen to use DV, and when notâŚ10/6/2011LearnDataVault.com35
- 52. The ChallengerâŚ.10/6/2011LearnDataVault.com36The challenger says:My system works fine, why should I use the Data Vault?
- 53. I donât have volume problemsâŚ
- 54. I donât have compliance/auditability problemsâŚ
- 55. I donât have real-time problemsâŚ
- 56. My system produces matching results across lines of businessâŚ
- 57. Iâve never had to âre-stateâ the data in the warehouseâŚ
- 58. I can still build new marts, and conform dimensions in 30 days or lessâŚ
- 59. My business doesnât acquire new systems often (if ever)
- 60. My incoming data sets donât changeI SayâŚThatâs wonderful, donât fix whatâs broken. Have a nice day, oh- but call me when or if you ever run into these problemsâŚ
- 61. When to Apply the Data Vault10/6/2011LearnDataVault.com37Benefits:Scalability
- 62. Auditability
- 63. Flexibility
- 65. IT and Business Accountability
- 68. Money Savings
- 69. Risk Mitigation
- 70. Successful EDW ImplementationsHow to build a data vaultIn 10 easy stepsâŚ10/6/2011LearnDataVault.com38
- 71. Step 110/6/2011LearnDataVault.com39Identify your business processes, followed by your business keys (that are used to identify the data that flows through the business processes)** NOTE: Along the way, document your assumptions, document your reasons for choosing keys, and modeling designs, develop a list of questions to be answered by business usersâŚ
- 72. Step 210/6/2011LearnDataVault.com40Identify the issues/problems that might be carried with the identified business keys, annotate the risks, and mitigate each one.
- 73. Step 310/6/2011LearnDataVault.com41Identify the units of work, the associations â LINK tables, where keys combine to form a notion, a concept, and a relationship.
- 74. Step 410/6/2011LearnDataVault.com42Identify the descriptive data that belongs to SINGLE Hub Keys, ensure that the data doesnât represent or rely on a relationship.
- 75. Step 510/6/2011LearnDataVault.com43Identify the Satellite data that depends on relationships â move it to the appropriate LINK table.HINT: If you âwantâ to put a Foreign Key in a Satellite, you have a clear sign that the Satellite is in the WRONG place, and needs to be assigned to a LINK table rather than a HUB.
- 76. Step 610/6/2011LearnDataVault.com44Scope the Model Down to a managable chunk. Implement the first two Hubs, Hub Satellites, and first Link. BUILD IN INCREMENTS!
- 77. Step 710/6/2011LearnDataVault.com45Setup the key generation load routines, setup the staging area, and begin loading data.
- 78. Step 810/6/2011LearnDataVault.com46Review any âtruncationâ errors, or any data-type conversion problems, fix the staging area, and remove duplicates.
- 79. Step 910/6/2011LearnDataVault.com47Begin Loading the Data Vault. Load all Hubs, then all Hub Satellites, Then all Links, and finish with All Link Satellites.
- 80. Step 1010/6/2011LearnDataVault.com48Reconcile the Data Vault to the source system, then build a first data mart from the results. Bring business value FAST!
- 82. 10 minutes to find the HubsâŚ.10/6/2011LearnDataVault.com50
- 83. Possible Hubs From Northwind10/6/2011LearnDataVault.com51
- 84. 10 Minutes to find the LinksâŚ10/6/2011LearnDataVault.com52
- 85. Possible Links From Northwind10/6/2011LearnDataVault.com53
- 86. 10 minutes to find the SatellitesâŚ10/6/2011LearnDataVault.com54
- 87. Possible Satellites From Northwind10/6/2011LearnDataVault.com55
- 88. What did we learn?We often deal with more than 1 system at a time⌠this was a lab with only one model.We didnât have any business requirements that we might need to answer questions, but doesnât that reflect real-life?The data set is extremely dirty (you never have that in your systems right?)Time Zone based data can be a problemLack of metadata causes integration issues and modeling decisions10/6/2011LearnDataVault.com56
- 89. The Experts SayâŚâThe Data Vault is the optimal choice for modeling the EDW in the DW 2.0 framework.â Bill InmonâThe Data Vault is foundationally strong and exceptionally scalable architecture.âStephen BrobstâThe Data Vault is a technique which some industry experts have predicted may spark a revolution as the next big thing in data modeling for enterprise warehousing....â Doug Laney57
- 90. More NotablesâŚâThis enables organizations to take control of their data warehousing destiny, supporting better and more relevant data warehouses in less time than before.â Howard Dresnerâ[The Data Vault] captures a practical body of knowledge for data warehouse development which both agile and traditional practitioners will benefit from..âScott Ambler58
- 91. Where To Learn MoreThe Technical Modeling Book: http://LearnDataVault.comThe Discussion Forums: & eventshttp://LinkedIn.com â Data Vault DiscussionsContact me:http://DanLinstedt.com - web [email protected] - emailWorld wide User Group (Free)http://dvusergroup.com59