Data warehousing and machine learning primer

Data warehousing and machine learning primer

• 1.
  Data Warehousing and MachineLearning Tom Donoghue
• 2.
  Data Warehouse • Businessuser friendly stories about past events (including near time) • Designed to support decision making • Serves a digest of answers in grouped and aggregated ways • More meaningful and therefore more important to the business • Ingests data from disparate sources which need to be merged to enable business friendly queries
• 3.
  Data Warehouse Definition •A consolidating bolt-on to existing operational systems • Structured data associated with a specific user base and a specific set of predefined business queries • The data schema is predefined and structured to facilitate regular and ad- hoc queries • Populating the data warehouse requires multiple ETL processes designed in advance • Halts the proliferation of reports O'Leary (2014)
• 4.
  Data Warehouse BasicArchitecture ETL Staging Area Source Data Data Warehouse Business Users Source Data Source Data Operational Data Soures Data Preparation Business Queries
• 5.
  Data Warehouse Requirements •Organisational Data is easy to access • Information is presented consistently • Adaptive and resilient to change • Secure • Serves as a base for improved decision making • Accepted by the business community (Kimball, 2002)
• 6.
  Machine Learning • AData warehouse provides historic information for decision making • Machine Learning uses algorithms to process features in the data to learns patterns, make predictions and solution outcomes • Image recognition, Classification, Forecasting, Anomaly detection • Learning is Supervised (labelled with the desired outcome) or Unsupervised (unlabelled, the model learns unaided)
• 7.
  Machine Learning -Supervised • A predictive model is trained using a labelled training data set and the outcome evaluated on its performance • The model is tweaked to improve performance • The model is then run against a test data set which is unlabelled and evaluated on its performance in identifying the correct label • Examples: • k-Nearest Neighbours • Linear and Logistical Regression • Decision Trees • Support Vector Machines (Lantz, 2015)
• 8.
  Machine Learning -Unsupervised • The training data set is unlabelled • The descriptive model is trained and evaluated on its performance • Examples: • Clustering - k-Means • Association Rules • Natural Language Processing (Lantz, 2015)
• 9.
  Machine Learning anExtension to Data Warehousing • Much of the hard work to cleanse and transform data has been accomplished • Ask the Business Question – what is the objective? Is it descriptive or predictive? • Does the data contain the desired features? • Is further data transformation required • Which ML algorithm is optimal for answering the question? • Iterative approach assessing and evaluating model(s) performance • Present the Solution
• 10.
  References • Kimball, R.,Ross, M., Thornthwaite, W., Mundy. J and Becker, B. (2008) The data warehouse lifecycle toolkit. 2nd ed. Indianapolis: Wiley Publishing, Inc. • Lantz, B. (2015). Machine Learning with R, 2nd edn, Birmingham: Packt. • O'Leary, D. E. (2014), ‘Embedding AI and Crowdsourcing in the Big Data Lake’, IEEE Intelligent Systems, Volume 29, Issue 5, pp. 70-73.