Your favorite data modeling tool, your partner in crime for Data Warehouse Automation?
- 1. Your favorite data modeling tool - your partner in crime for DWHA? Frederik Naessens Freelance BI Architect
- 2. Why this presentation? Data Vault training & certification
- 3. Data vault automation conference – Utrecht – 6 October 2011
- 4. Model driven development ¤ Use a dwh automation software package ¤ If not possible: ¤ Don’t just do nothing! ¤ Go solo ¤ Start small ¤ Create awareness within the company
- 5. Custom generators have a bad reputation ¤ Doesn’t scale well ¤ More people ¤ How to share/version/merge your input? ¤ More data ¤ Lots of data redundancy ¤ Can you ever migrate to DWHA tool?
- 6. My goal ¤ Show how to benefit from the (underused) potential of your data modeling tool: ¤ Extend your model (with missing 10%) ¤ Support your multi-user & incremental approach ¤ Standardize ¤ Share information with others ¤ Automate ¤ This by: ¤ Only using standard functionality of the tool API – custom plugins ¤ Based on common available information
- 7. My preparation ¤ Installed CA ERwin Data Modeler Standard Edition on april 3rd ¤ Read this very good book during easter holidays
- 8. Scope ¤ “Left to right” Vs. “Right to Left” approach
- 9. Scope ¤ “Left to right” Vs. “Right to Left” approach
- 10. Scope today ¤ Logical to physical Data Vault model conversion ¤ Logical model ¤ 3NF extended model ¤ More “DV-unaware” people friendly ¤ Less tables!! ¤ Physical tables are automatically built based on naming conventions
- 11. Model extensions ¤ Define user defined properties in model on following levels ¤ Model level ¤ Version number ¤ Table level Hub ¤ Driving Data Vault Type Link Satellite ¤ Column level ¤ Business Key ¤ Attribute Cluster ¤ Relationship level ¤ Is Reference Link ¤ Unit Of Work
- 12. Demo screenshots ¤ Step 1 – v.01 ¤ Business 3NF Model ¤ Generated Data Vault Model ¤ Step 2 – Modifications ¤ Assign business key to Order_Details table ¤ Split attributes of Orders in 2 distinct Attribute Clusters ¤ Assign different unit of work to Orders foreign keys ¤ Step 3 – v.02 ¤ Changed Business 3NF model ¤ Slightly different generated Data Vault Model
- 13. Region RegionID RegionDescription BM v.01 Employees EmployeeID LastName Suppliers Customers FirstName SupplierID Title CustomerID TitleOfCourtesy CompanyName Categories CompanyName Territories BirthDate ContactName Shippers ContactName HireDate CategoryID ContactTitle CustomerDemographics TerritoryID ShipperID ContactTitle Address Address CustomerTypeID Address City CategoryName City TerritoryDescription CompanyName City Region Description Region CustomerDesc RegionID Phone Region PostalCode Picture PostalCode PostalCode Country Country Country HomePhone Phone Phone Extension Fax Fax Photo HomePage Notes ReportsTo CustomerCustomerDemo PhotoPath CustomerID CustomerTypeID Orders OrderID Products ProductID CustomerID (FK) EmployeeID (FK) ProductName OrderDate EmployeeTerritories SupplierID (FK) RequiredDate EmployeeID CategoryID (FK) ShippedDate TerritoryID QuantityPerUnit ShipVia (FK) UnitPrice Freight UnitsInStock ShipName UnitsOnOrder ShipAddress ReorderLevel ShipCity Discontinued ShipRegion ShipPostalCode ShipCountry Blue: Business entity Order_Details Red: Association between business entities OrderDetailsID OrderID (FK) Underlined: Business Key ProductID (FK) UnitPrice Quantity Discount
- 14. Hub_Employees Hub_Employees_SQN Hub_Employees_LDTS Hub_Employees_RSRC EmployeeID ReportsTo DV v.01 Sat_Employees_Default Sat_Employees_Default_LDTS Hub_Employees_SQN Sat_Employees_Default_RSRC Sat_Employees_Default_LEDTS LastName Lnk_Orders_Default FirstName Lnk_EmployeeTerritories Lnk_Employees_Default Title Lnk_Orders_Default_SQN TitleOfCourtesy Lnk_EmployeeTerritories_SQN Lnk_Employees_Default_SQN Lnk_Orders_Default_LDTS BirthDate Lnk_EmployeeTerritories_LDTS Lnk_Orders_Default_RSRC Lnk_Employees_Default_LDTS HireDate Lnk_EmployeeTerritories_RSRC Hub_Orders_SQN Lnk_Employees_Default_RSRC Address Hub_Employees_SQ N Hub_Customers_SQ N Hub_Employees_SQN City Hub_Territories_SQN Hub_Employees_SQ N Hub_Employees_ReportsTo_SQ N Region Hub_Shippers_ShipVia_SQ N PostalCode Country HomePhone Extension Photo Notes PhotoPath Hub_Territories Hub_Orders Hub_Customers Hub_Shippers Hub_Territories_SQN Hub_Orders_SQN Hub_Customers_SQN Hub_Shippers_SQN Hub_Territories_LDTS Hub_Orders_LDTS Hub_Customers_LDTS Hub_Shippers_LDTS Hub_Territories_RSRC Hub_Orders_RSRC Hub_Customers_RSRC Hub_Shippers_RSRC TerritoryID OrderID CustomerID ShipperID Sat_Orders_Default Sat_Customers_Default Sat_Orders_Default_LDTS Sat_Customers_Default_LDTS Hub_Orders_SQN Hub_Customers_SQN Sat_Orders_Default_RSRC Sat_Customers_Default_RSRC Sat_Shippers_Default Sat_Territories_Default Lnk_Territories_Default Lnk_Order_Details Lnk_CustomerCustomerDemo Sat_Orders_Default_LEDTS Sat_Customers_Default_LEDTS Sat_Shippers_Default_LDTS Sat_Territories_Default_LDTS Lnk_Territories_Default_SQN OrderDate Lnk_Order_Details_SQN CompanyName Lnk_CustomerCustomerDemo_SQN Hub_Shippers_SQN Hub_Territories_SQN RequiredDate ContactName Lnk_Territories_Default_LDTS Lnk_Order_Details_LDTS Lnk_CustomerCustomerDemo_LDTS ShippedDate ContactTitle Sat_Shippers_Default_RSRC Sat_Territories_Default_RSRC Lnk_Territories_Default_RSRC Lnk_Order_Details_RSRC Lnk_CustomerCustomerDemo_RSRC Freight Address Sat_Shippers_Default_LEDTS Sat_Territories_Default_LEDTS Hub_Territories_SQN Hub_Orders_SQ N Hub_Customers_SQN TerritoryDescription Hub_Region_SQ N ShipName Hub_Products_SQ N City Hub_CustomerDemographics_SQN CompanyName ShipAddress Region Phone ShipCity PostalCode ShipRegion Country ShipPostalCode Phone ShipCountry Fax Sat_Order_Details_Default Hub_Products Sat_Order_Details_Default_LDTS Hub_Region Hub_CustomerDemographics Hub_Products_SQN Lnk_Order_Details_SQN Hub_Region_SQN Hub_CustomerDemographics_SQN Hub_Products_LDTS Sat_Order_Details_Default_RSRC Hub_Region_LDTS Hub_Products_RSRC Sat_Order_Details_Default_LEDTS Hub_CustomerDemographics_LDTS Hub_Region_RSRC ProductID OrderDetailsID Hub_CustomerDemographics_RSRC RegionID SupplierID UnitPrice CustomerTypeID CategoryID Quantity Discount Sat_Products_Default Sat_Products_Default_LDTS Hub_Products_SQN Lnk_Products_Default Sat_Region_Default Sat_CustomerDemographics_Default Sat_Products_Default_RSRC Lnk_Products_Default_SQN Sat_Region_Default_LDTS Sat_CustomerDemographics_Default_LDTS Hub_Region_SQN Sat_Products_Default_LEDTS Hub_CustomerDemographics_SQN Lnk_Products_Default_LDTS ProductName Lnk_Products_Default_RSRC Sat_Region_Default_RSRC QuantityPerUnit Sat_CustomerDemographics_Default_RSRC Hub_Products_SQ N Sat_Region_Default_LEDTS UnitPrice Sat_CustomerDemographics_Default_LEDTS Hub_Suppliers_SQN RegionDescription UnitsInStock Hub_Categories_SQ N CustomerDesc UnitsOnO rder ReorderLevel Discontinued Hub_Suppliers Hub_Categories Hub_Suppliers_SQN Hub_Categories_SQN Hub_Suppliers_LDTS Hub_Categories_LDTS Hub_Suppliers_RSRC Hub_Categories_RSRC SupplierID CategoryID Sat_Suppliers_Default Sat_Suppliers_Default_LDTS Hub_Suppliers_SQN Sat_Suppliers_Default_RSRC Sat_Categories_Default Sat_Suppliers_Default_LEDTS Sat_Categories_Default_LDTS CompanyName Hub_Categories_SQN ContactName ContactTitle Sat_Categories_Default_RSRC Address Sat_Categories_Default_LEDTS City CategoryName Region Description PostalCode Picture Country Phone Fax HomePage
- 15. Region RegionID RegionDescription BM v.02 Employees EmployeeID LastName Suppliers Customers FirstName SupplierID Title CustomerID TitleOfCourtesy CompanyName CompanyName BirthDate Categories ContactName T erritories Shippers ContactName HireDate ContactTitle CustomerDemographics CategoryID ContactTitle TerritoryID Address Address ShipperID Address CustomerTypeID City CategoryName City TerritoryDescription CompanyName City Region Description Region CustomerDesc RegionID Phone Region PostalCode Picture PostalCode PostalCode Country Country Country HomePhone Phone Phone Extension Fax Fax Photo HomePage Notes ReportsTo PhotoPath Orders OrderID P roducts ProductID CustomerID (FK) EmployeeID (FK) ProductName OrderDate EmployeeTerritories Cus tomerCustomerDemo SupplierID (FK) RequiredDate EmployeeID CategoryID (FK) ShippedDate CustomerID TerritoryID QuantityPerUnit ShipVia (FK) CustomerTypeID UnitPrice Freight UnitsInStock ShipName UnitsOnOrder ShipAddress ReorderLevel ShipCity Discontinued ShipRegion ShipPostalCode ShipCountry Order_Details OrderDetailsID OrderID (FK) ProductID (FK) UnitPrice Quantity Discount
- 16. 2 satellites for order H ub_Orders Hub_Orders_SQN Hub_Orders_LDTS Hub_Orders_RSRC OrderID S a t_Orders_ShipInfo S at_Orders_Default DV v.02 Sat_Orders_ShipInfo_LDTS Hub_Orders_SQN Sat_Orders_Default_LDTS L n k _Order_Details_Default L n k_Orders_ShipInfo L n k_Orders_Default Hub_Orders_SQN Lnk_Order_Details_Default_SQN Lnk_Orders_Default_SQN Sat_Orders_ShipInfo_RSRC Lnk_Orders_ShipInfo_SQN Sat_Orders_ShipInfo_LEDTS Sat_Orders_Default_RSRC Lnk_Order_Details_Default_LDTS Lnk_Orders_ShipInfo_LDTS Lnk_Orders_Default_LDTS ShipName Sat_Orders_Default_LEDTS Lnk_Order_Details_Default_RSRC Lnk_Orders_Default_RSRC ShipAddress OrderDate Hub_Order_Details_SQN Lnk_Orders_ShipInfo_RSRC Hub_Orders_SQN Hub_Orders_SQN ShipCit y RequiredDate Hub_Orders_SQN Hub_Shippers_ShipVia_SQ N Hub_Customers_SQN ShipRegion ShippedDate Hub_Products_SQN Hub_Employees_SQN ShipPostalCode Freight ShipCountry Hub_Products Hub_Employees H ub_Order_Details Hub_Shippers H ub_Customers Hub_Products_SQN Hub_Employees_SQN Hub_Order_Details_SQN Hub_Shippers_SQN Hub_Customers_SQN Hub_Products_LDTS Hub_Employees_LDTS Hub_Order_Details_LDTS Hub_Products_RSRC Hub_Shippers_LDTS Hub_Customers_LDTS Hub_Order_Details_RSRC ProductID Hub_Shippers_RSRC Hub_Customers_RSRC Hub_Employees_RSRC EmployeeID OrderDetailsID SupplierID ShipperID CustomerID ReportsTo CategoryID S at_Employees_Default Sat_Employees_Default_LDTS Hub_Employees_SQN S a t_Customers_Default Sat_Employees_Default_RSRC S at_Products_Default Sat_Customers_Default_LDTS Hub_Customers_SQN Sat_Employees_Default_LEDTS Sat_Products_Default_LDTS LastName S a t_Order_Details_Default Hub_Products_SQN L n k_Products_Default S at_Shippers_Default Sat_Customers_Default_RSRC L n k _CustomerCustomerDemo FirstName L n k _EmployeeTerritories L nk_Employees_Default Sat_Order_Details_Default_LDTS Sat_Customers_Default_LEDTS Title Hub_Order_Details_SQN Sat_Products_Default_RSRC Lnk_Products_Default_SQN Sat_Shippers_Default_LDTS Lnk_CustomerCustomerDemo_SQN Lnk_EmployeeTerritories_SQN Lnk_Employees_Default_SQN Hub_Shippers_SQN CompanyName TitleOfCourtesy Sat_Products_Default_LEDTS Lnk_Products_Default_LDTS ContactName BirthDate Sat_Order_Details_Default_RSRC ProductName Lnk_Products_Default_RSRC Sat_Shippers_Default_RSRC ContactTitle Lnk_CustomerCustomerDemo_LDTS HireDate Lnk_EmployeeTerritories_LDTS Lnk_Employees_Default_LDTS Sat_Order_Details_Default_LEDTS QuantityPerUnit Lnk_CustomerCustomerDemo_RSRC Lnk_EmployeeTerritories_RSRC Lnk_Employees_Default_RSRC UnitPrice UnitPrice Hub_Products_SQN Sat_Shippers_Default_LEDTS Address Hub_Customers_SQN Address Hub_Employees_SQN Hub_Employees_SQN Hub_Suppliers_SQN CompanyName City City Quantity UnitsInStock Hub_Categories_SQN Phone Region Hub_CustomerDemographics_SQN Region Hub_Territories_SQN Hub_Employees_ReportsTo_SQN Discount UnitsOnOrder PostalCode PostalCode ReorderLevel Country Country Discontinued Phone HomePhone Fax Extension Photo Notes PhotoPath Hub_Suppliers H ub_Categories H u b _CustomerDemographics H u b_Territories Order_details hub Hub_Suppliers_SQN Hub_Suppliers_LDTS Hub_Suppliers_RSR SupplierID C Hub_Categories_SQN Hub_Categories_LDTS Hub_Categories_RSRC CategoryID Hub_CustomerDemographics_SQN Hub_CustomerDemographics_LDTS Hub_CustomerDemographics_RSRC CustomerTypeID Hub_Territories_SQN Hub_Territories_LDTS Hub_Territories_RSRC TerritoryID S at_Suppliers_Default Sat_Suppliers_Default_LDTS Hub_Suppliers_SQN Sat_Suppliers_Default_RSRC S a t_Categories_Default Sat_Suppliers_Default_LEDTS Sat_Categories_Default_LDTS S a t _CustomerDemographics_Default S a t _Territories_Default L n k _Territories_Default CompanyName Hub_Categories_SQN Sat_CustomerDemographics_Default_LDTS Sat_Territories_Default_LDTS Lnk_Territories_Default_SQN ContactName Hub_CustomerDemographics_SQN Hub_Territories_SQN ContactTitle Sat_Categories_Default_RSRC Lnk_Territories_Default_LDTS Address Sat_Categories_Default_LEDTS Sat_CustomerDemographics_Default_RSRC Sat_Territories_Default_RSRC Lnk_Territories_Default_RSRC City CategoryName Sat_CustomerDemographics_Default_LEDTS Sat_Territories_Default_LEDTS Hub_Territories_SQN Region Description CustomerDesc TerritoryDescription Hub_Region_SQN PostalCode Picture Country Phone Fax HomePage Hub_Region Hub_Region_SQN Hub_Region_LDTS Hub_Region_RSRC RegionID 2 link tables for order foreign keys S at_Region_Default Sat_Region_Default_LDTS Hub_Region_SQN Sat_Region_Default_RSRC Sat_Region_Default_LEDTS RegionDescription
- 17. Extract & Share information ¤ Forward Engineering ¤ Based on FE templates ¤ Based on FE option sets ¤ Xml exports ¤ Crystal reports ¤ Standard reports ¤ Customized reports ¤ ODBC Query tool
- 18. FE result
- 19. Query tool
- 20. Multi user ¤ Merge & compare ¤ Part of Complete compare wizard ¤ User defined properties are also taken into account! ¤ Versioning ¤ Part of workgroup edition ¤ Not tested
- 21. Bulk editing & Automation ¤ Bulk editor ¤ Phased approach: ¤ Modify model ¤ Forward engineer ¤ Convert with external utility ¤ Reverse engineer to DV model ¤ Not experimented with add-ins yet
- 22. My goal (repeated) ¤ Show how to benefit from the (underused) potential of your modeling tool: ¤ Extend your model (with missing 10%) ¤ Supports your multi-user & incremental approach ¤ Standardize ¤ Share information with others ¤ Automate ¤ This by: ¤ Only using standard functionality of the tool API – custom plugins ¤ Based on common available information
- 23. Thanks!! [email protected] +32 473 82 60 80 A special thanks to: Donna Burbank, Hans Hultgren Volvo IT colleagues: Stijn Roelens & Kristof Vanduren Frédéric Barbier, Erik-Jan Koning