Data Warehousing about data ware house.pptx
- 1. Introduction to Data Warehousing Data Warehousing March 2024 1 S. Hassan Adelyar, Ph.D Instructor of Computer Science Faculty Kabul University March 2024 Introduction to Data Warehousing 07:50:40 AM
- 2. Introduction to Data Warehousing Data Warehousing March 2024 2 A subject-oriented, integrated, time-variant, & non-volatile collection of data in support of management’s decision-making process. An enterprise system used for the analysis & reporting of structured & semi-structured data. Receives data periodically & on a regular basis from multiple sources such as: Point-of-sale transactions Marketing automation Data warehouse
- 3. Introduction to Data Warehousing Data Warehousing March 2024 3 Relational databases Customer relationship management Operational sources External data sources Websites Store both current & historical data in one place & is designed to give a long-range view of data over time, supports business intelligence (BI) activities, specifically analysis. Data warehouse
- 4. Introduction to Data Warehousing Data Warehousing March 2024 4 This data is then made available for decision- makers to access & analyze. A data warehouse is not a single software or hardware product you purchase to provide strategic information. It is a computing environment where users can find strategic information, & users are put directly in touch with the data they need to make better decisions. It is a user-centric environment.
- 5. Introduction to Data Warehousing Data Warehousing March 2024 5 Answer questions users have about the business, the performance of the various operations, the business trends, & about what can be done to improve the business.
- 6. Introduction to Data Warehousing Data Warehousing March 2024 6 The environment for data warehouses & marts includes the following: Data integration technology & processes that are needed to prepare the data for use; Different tools & applications for a variety of users; The basic concept of data warehousing is: Take all the data from the operational systems. A Blend of Many Technologies
- 7. Introduction to Data Warehousing Data Warehousing March 2024 7 Integrate all the data from the various sources. Remove inconsistencies & transform the data. Store the data in formats suitable for easy access for decision making. Figure 1-9 shows how a data warehouse is a blend of the many technologies.
- 8. Introduction to Data Warehousing Data Warehousing March 2024 8 Figure 1-9 The data warehouse: a blend of technologies
- 9. Introduction to Data Warehousing Data Warehousing March 2024 9 Every data warehouse has three fundamental components: Load Manager Warehouse Manager Data Access Manager Data warehouse architecture
- 10. Introduction to Data Warehousing Data Warehousing March 2024 10 Load manager Responsible for Data collection from operational systems. Performs data conversion into some usable form to be further utilized by the user. Includes all the programs & application interfaces which are required for extracting data from the operational systems.
- 11. Introduction to Data Warehousing Data Warehousing March 2024 11 It should perform the following tasks: Data Identification Data Validation for its accuracy Data Extraction from the original source Data Cleansing Data formatting Consolidates data from multiple sources to one place
- 12. Introduction to Data Warehousing Data Warehousing March 2024 12 Warehouse manager The main part of Data Warehousing system. Holds the massive amount of information from many sources. Organizes data in a way so it becomes easy for anyone to analyze or find the required information.
- 13. Introduction to Data Warehousing Data Warehousing March 2024 13 Architecture of a data warehouse
- 14. Introduction to Data Warehousing Data Warehousing March 2024 14 Database: The main difference is that in a database, data is collected for multiple transactional purposes. Databases provide real-time data. Data Warehouse: In a data warehouse, data is collected on an extensive scale to perform analytics. Data warehouses store data to be accessed for big analytical queries. Database vs. Data Warehouse
- 15. Introduction to Data Warehousing Data Warehousing March 2024 15 Most common data warehouse usages are: Making real-time decisions: Analyze data in real time to proactively address challenges, identify opportunities, gain efficiency, reduce costs, & proactively respond to business events. Data warehouse usages
- 16. Introduction to Data Warehousing Data Warehousing March 2024 16 Consolidating siloed data: Quickly pull data from multiple structured sources across your organization, such as point-of-sale systems, websites, & email lists, & bring it together into one location so that you can perform analysis & get insights.
- 17. Introduction to Data Warehousing Data Warehousing March 2024 17 Enabling business reporting & ad hoc analysis: Keep historical data on a separate server from operational data so that end users can access it & run their own queries & reports without impacting the performance of operational systems or waiting to get help from IT.
- 18. Introduction to Data Warehousing Data Warehousing March 2024 18 Implementing machine learning & AI: Collect historical & real-time data to develop algorithms that can provide predictive insights, such as anticipating traffic points or suggesting relevant products to a customer browsing a website.
- 19. Introduction to Data Warehousing Data Warehousing March 2024 19 If your organization has or does any of the following, you’re probably a good candidate for a data warehouse: Multiple sources of disparate data Big-data analysis & visualization Machine learning models & other AI- driven processes Custom report generation & ad hoc analysis
- 20. Introduction to Data Warehousing Data Warehousing March 2024 20 Enterprise Data Warehouse (EDW) This type of warehouse serves as a key or central database that facilitates decision- support services throughout the enterprise. The advantage to this type of warehouse is that it provides access to cross- organizational information, offers a unified approach to data representation, & allows running complex queries. Types of Data Warehouse
- 21. Introduction to Data Warehousing Data Warehousing March 2024 21 Operational Data Store (ODS) This type of data warehouse refreshes in real-time. It is often preferred for routine activities like storing employee records. It is required when data warehouse systems do not support reporting needs of the business. Data Mart A data mart is a subset of a data warehouse built to maintain a particular department, region, or business unit.
- 22. Introduction to Data Warehousing Data Warehousing March 2024 22 Every department of a business has a central repository or data mart to store data. The data from the data mart is stored in the ODS periodically. The ODS then sends the data to the EDW, where it is stored & used.
- 23. Introduction to Data Warehousing Data Warehousing March 2024 23 Business intelligence for an organization requires two environments : Transformation of data to information; Derivation of knowledge from information. Business intelligence (BI), therefore, is a broad group of applications & technologies. First, the term refers to the systems & technologies for gathering, cleaning, consolidating, & storing corporate data. Evolution of Business Intelligence (BI)
- 24. Introduction to Data Warehousing Data Warehousing March 2024 24 Next, business intelligence (BI) relates to the tools, techniques, & applications for analyzing the stored data. BI is an umbrella term to include concepts & methods to improve business decision making by fact-based support systems.
- 25. Introduction to Data Warehousing Data Warehousing March 2024 25 When you consider all that BI encompasses, you may view BI for an enterprise as composed of two environments: Data to Information In this environment data from multiple operational systems are extracted, integrated, cleansed, transformed & stored as information in specially designed repositories. BI: Two Environments
- 26. Introduction to Data Warehousing Data Warehousing March 2024 26 Information to Knowledge In this environment analytical tools are made available to users to access & analyze the information content in the specially designed repositories & turn information into knowledge.
- 27. Introduction to Data Warehousing Data Warehousing March 2024 27 Figure 1-10 shows the two complementary environments, the data warehousing environment, which transforms data into information, & the analytical environment, which produces knowledge from information.
- 28. Introduction to Data Warehousing Data Warehousing March 2024 28 Figure 1-10 BI: data warehousing & analytical environments
- 29. Introduction to Data Warehousing Data Warehousing March 2024 29 Common functions of business intelligence technologies include: Reporting Online analytical processing Data mining Process mining Complex event processing Business performance management
- 30. Introduction to Data Warehousing Data Warehousing March 2024 30 Text mining Predictive analytics Prescriptive analytics
- 31. Introduction to Data Warehousing Data Warehousing March 2024 31 Traditional data warehouses: Hosted on-premises, with data flowing in from relational databases, transactional systems, business applications, & other source systems. Typically designed to capture a subset of data in batches & store it, making them unsuitable for unstructured queries or real- time analysis. Traditional vs. cloud-based data warehouse
- 32. Introduction to Data Warehousing Data Warehousing March 2024 32 Companies also must purchase their own hardware & software with an on-premises data warehouse, making it expensive to scale & maintain. Storage is typically limited compared to compute, so data is transformed quickly & then discarded to keep storage space free.
- 33. Introduction to Data Warehousing Data Warehousing March 2024 33 Cloud-based data warehouse: Today’s data analytics activities have transformed to the center of all core business activities, including revenue generation, cost containment, improving operations, & enhancing customer experiences.
- 34. Introduction to Data Warehousing Data Warehousing March 2024 34 As data evolves & diversifies, organizations need more robust data warehouse solutions & advanced analytic tools for storing, managing, & analyzing large quantities of data across their organizations. These systems must be scalable, reliable, secure enough for regulated industries, & flexible enough to support a wide variety of data types & big data use cases.
- 35. Introduction to Data Warehousing Data Warehousing March 2024 35 The data stored in the warehouse is uploaded from the operational systems. There are two main approaches used to build a data warehouse system: Extract, transform, load (ETL) Extract, load, transform (ELT) Architecture of a Data Warehouse
- 36. Introduction to Data Warehousing Data Warehousing March 2024 36 Subject-Oriented A data warehouse is subject-oriented since it provides topic-wise information rather than the overall processes of a business. Such subjects may be sales, promotion, inventory, etc. For example, if you want to analyze your company’s sales data, you need to build a data warehouse that concentrates on sales. Key Characteristics of Data Warehouse
- 37. Introduction to Data Warehousing Data Warehousing March 2024 37 Such a warehouse would provide valuable information like ‘who was your best customer last year?’ or ‘who is likely to be your best customer in the coming year?’
- 38. Introduction to Data Warehousing Data Warehousing March 2024 38 Integrated A data warehouse is developed by integrating data from varied sources into a consistent format. The data must be stored in the warehouse in a consistent & universally acceptable manner in terms of naming, format, & coding. This facilitates effective data analysis.
- 39. Introduction to Data Warehousing Data Warehousing March 2024 39 Non-Volatile Data once entered into a data warehouse must remain unchanged. All data is read-only. Previous data is not erased when current data is entered. This helps you to analyze what has happened & when.
- 40. Introduction to Data Warehousing Data Warehousing March 2024 40 Time-Variant The data stored in a data warehouse is documented with an element of time, either explicitly or implicitly. An example of time variance in Data Warehouse is exhibited in the Primary Key, which must have an element of time like the day, week, or month.
- 41. Introduction to Data Warehousing Data Warehousing March 2024 41 Data warehouse tools are software components used to perform several operations on an extensive data set. These tools help to collect, read, write & transfer data from various sources. Data warehouses support are designed to support operations like data sorting, filtering, merging, etc. Data warehouse applications can be categorized as: Data Warehousing Tools
- 42. Introduction to Data Warehousing Data Warehousing March 2024 42 Query & reporting tools Application Development tools Data mining tools OLAP tools Some popular data warehouse tools are Xplenty, Amazon Redshift, Teradata, Oracle 12c, Informatica, IBM Infosphere, Cloudera, & Panoply.
- 43. End of Chapter 3 Question / Discussion?