(Lecture 4)Slowly Changing Dimensions.pdf
- 2. SCD overview In the real world, the business does not remain static over time; products change size and weight, customers relocate, stores change layouts, and sales staff are assigned to different locations. Most systems of record keep only the current values for business subjects (e.g., the current customer address). An operational data store keeps only a short history of changes to indicate that changes have occurred and to support business processes handling the immediate changes. In a data warehouse or data mart, we need to know the history of values to match the history of facts with the correct dimensional descriptions at the time the facts happened.
- 3. For example, We need to associate a sales fact with the description of the associated customer during the time period of the sales fact, which may not be the description of that customer today. Of course, business subjects change slowly compared with most transactional data (e.g., inventory level).
- 4. Three ways for handling SCD attributes 1. Overwrite the current value with the new value. (Type1) 2. For each dimension attribute that changes, create a current value field and as many old value fields as we wish (i.e., a multivalued attribute with a fixed number of occurrences for a limited historical view). (Type 3) 3. Create a new dimension table row (with a new surrogate key) each time the dimension object changes; this new row contains all the dimension characteristics at the time of the change; the new surrogate key is the original surrogate key plus the start date for the period when these dimension values are in effect. (Type 2)
- 5. Example of Type 2 SCD Customer dimension table
- 6. Example of Type 2 SCD Customer dimension table (cont…) Finding the dimension row for a fact row is a little more complex; the SQL WHERE clause would include the following: WHERE Fact.CustomerKey = Customer.CustomerKey AND Fact.DateKey BETWEEN Customer.StartDate and Customer.EndDate
- 7. Note: Type 2 SCD can cause an excessive number of dimension table rows when dimension objects frequently change or when dimension rows are large “monster dimensions.” Also, if only a small portion of the dimension row has changing values, there are excessive redundant data created. The solution for that is dimension segmentation.
- 8. Dimension Segmentation A dimension is segmented into two dimension tables; one segment may hold nearly constant or very slowly changing dimensions and other segments hold clusters of attributes that change more rapidly and, for attributes in the same cluster, often change at the same time. These more rapidly changing attributes are often called “hot” attributes by data warehouse designers.
- 9. Example of Dimension Segmentation
- 10. Determining Dimensions and Facts 1. What were the dollar sales of health and beauty products in North America to customers over the age of 50 in each of the past three years? 2. What is the name of the salesperson who had the highest dollar sales of each product in the first quarter of this year? 3. How many European customer complaints did we receive on pet food products during the past year? How has it changed from month to month this year? 4. What is the name of the stores that had the highest average monthly quantity sales of casual clothing during the summer?
- 11. Fact-qualifier matrix for sales and customer service tracking
- 12. Star schema for sales and customer service tracking
- 13. Ten Essential Rules of Dimensional Modeling 1. Use atomic facts: Eventually, users want detailed data, even if their initial requests are for summarized facts. 2. Create single-process fact tables: Each fact table should address the important measurements for one business process, such as taking a customer order or placing a material purchase order. 3. Include a date dimension for every fact table: A fact should be described by the characteristics of the associated day (or finer) date/time to which that fact is related. 4. Enforce consistent grain: Each measurement in a fact table must be atomic for the same combination of keys (the same grain). 5. Disallow null keys in fact tables: Facts apply to the combination of key values, and helper tables may be needed to represent some M:N relationships.
- 14. Ten Essential Rules of Dimensional Modeling(cont…) 6. Honor hierarchies: Understand the hierarchies of dimensions and carefully choose to snowflake the hierarchy or denormalize into one dimension. 7. Decode dimension tables: Store descriptions of surrogate keys and codes used in fact tables in associated dimension tables, which can then be used to report labels and query filters. 8. Use surrogate keys: All dimension table rows should be identified by a surrogate key, with descriptive columns showing the associated production and source system keys. 9. Conform dimensions: Conformed dimensions should be used across multiple fact tables. 10. Balance requirements with actual data: Unfortunately, source data may not precisely support all business requirements, so you must balance what is technically possible with what users want and need.
- 15. Role of MetaData The metadata associated with data marts are often referred to as a “data catalog,” “data directory,” or some similar term. Metadata serve as kind of a “yellow pages” directory to the data in the data marts. The metadata should allow users to easily answer questions such as the following:
- 16. Role of MetaData (cont…) 1. What subjects are described in the data mart? (Typical subjects are customers, patients, students, products, courses, and so on.) 2. What dimensions and facts are included in the data mart? What is the grain of the fact table? 3. How are the data in the data mart derived from the enterprise data warehouse data? What rules are used in the derivation? 4. How are the data in the enterprise data warehouse derived from operational data? What rules are used in this derivation? 5. What reports and predefined queries are available to view the data? 6. What drill-down and other data analysis techniques are available? 7. Who is responsible for the quality of data in the data marts, and to whom are requests for changes made?
- 17. SQL OLAP Querying Question: Which customer has bought the most of each product we sell? Show the product ID and description, customer ID and name, and the total quantity sold of that product to that customer; show the results in sequence by product ID.
- 18. Query: SELECT P1.ProductId, ProductDescription, C1.CustomerId, CustomerName,SUM(OL1.OrderedQuantity) AS TotOrdered FROM Customer_T AS C1, Product_T AS P1, OrderLine_T AS OL1, Order_T AS O1 WHERE C1.CustomerId = O1.CustomerId AND O1.OrderId = OL1.OrderId AND OL1.ProductId = P1.ProductId GROUP BY P1.ProductId, ProductDescription, C1.CustomerId, CustomerName HAVING TotOrdered >= ALL (SELECT SUM(OL2.OrderedQuantity) FROM OrderLine_T AS OL2, Order_T AS O2 WHERE OL2.ProductId = P1.ProductId AND OL2.OrderId = O2.OrderId AND O2.CustomerId <> C1.CustomerId GROUP BY O2.CustomerId) ORDER BY P1.ProductId;