Platforming the Major Analytic Use Cases for Modern Engineering
2 likes432 views
The document by William McKnight outlines various analytic use cases for modern engineering, focusing on major technologies such as AWS, Azure, and Snowflake. It details specific use cases like customer churn management, predictive maintenance, and fraud detection, emphasizing the need for businesses to leverage data for insights and operational efficiency. Additionally, it touches upon challenges in interoperability and cost within heterogeneous system architectures.
Platforming the Major Analytic Use Cases for Modern Engineering
- 1. Platforming the Major Analytic Use Cases for Modern Engineering Presented by: William McKnight President, McKnight Consulting Group williammcknight www.mcknightcg.com (214) 514-1444
- 2. William McKnight President, McKnight Consulting Group • Consulted to Pfizer, Scotiabank, Fidelity, TD Ameritrade, Teva Pharmaceuticals, Verizon, and many other Global 1000 companies • Frequent keynote speaker and trainer internationally • Hundreds of articles, blogs and white papers in publication • Focused on delivering business value and solving business problems utilizing proven, streamlined approaches to information management • Former Database Engineer, Fortune 50 Information Technology executive and Ernst&Young Entrepreneur of Year Finalist • Owner/consultant: Data strategy and implementation consulting firm 2
- 3. McKnight Consulting Group Client Portfolio
- 5. AWS • AWS's implementation of popular big data frameworks is Amazon Elastic Map Reduce, or EMR, a fully-managed system designed to get an implementation off the ground with little administrative overhead. • EMR packages many of the usual Apache suspects—Hadoop, Hive, HBase, Spark, ZooKeeper, et cetera. • Redshift, Amazon’s data warehouse option, has undergone some major improvements in recent releases. – At the end of 2019, Amazon released RA3 nodes which are built on the AWS Nitro System and feature high bandwidth networking and performance, reportedly indistinguishable from bare metal.
- 6. AWS • For data science and machine learning, Amazon offers SageMaker as another fully managed service which leverages Jupyter notebooks for building models. • Amazon offers a data catalog as part of their ETL tool, AWS Glue. • You only have workload management component- by-component. • You use the AWS Portal to manage and deploy resources. • Security in AWS is driven nearly completely by Identity Access Management (IAM).
- 7. Azure • Beginning with data engineering, Azure HDInsight is Microsoft’s managed Hadoop, Spark, and Kafka offering based on the original Hortonworks Data Platform. – With HDInsight, you can choose between Azure Blob Storage, Azure Data Lake Storage (ADLS) Gen1, and ADLS Gen 2. • Azure Synapse Analytics (formerly known as Azure SQL Data Warehouse) is Microsoft’s data warehouse offering. • Azure Machine Learning is heavily slanted towards MLOps 7
- 8. Azure • Azure Data Catalog does a good job crawling external databases and applications, like Oracle or Salesforce. • Microsoft seems fully invested in Azure Data Factory (ADF) as the data integration method of the future (not SSIS). • Azure lacks an overarching workload management offering. Each individual component, like Synapse and HDInsight, offer their own, but the all-in-one view is missing. • The Azure Portal serves as a single console to deploy resources and has improved considerably with managed resource deployments. 8
- 9. Snowflake/Heterogenous • Piecemeal ecosystem with a variety of tools from a number of vendors. • The overwhelming challenges are interoperability, cost, and complexity. • It is not all bad, as there are some interoperability beacons of hope. • With multiple vendors, different licensing arrangements from perpetual-use to subscription-based, maintenance contracts, and surprise bills from cloud vendors for burst, on-demand compute usage, the net OpEx lines make the true total cost of ownership (TCO) a more difficult web to disentangle. 9
- 10. Snowflake/Heterogenous • The overall complexity of the architecture can be overwhelming, including sourcing the right talent, engaging multiple professional service providers, managing organizational change, and driving user adoption. • Still organizations have plenty of vendors from which to choose. 10
- 13. Project ROI & TCO 13 ROI = Benefit TCO Infrastructure Software + FTE + Consulting +
- 14. Use Cases
- 15. Customer Churn • Customer Churn use cases are about generating high levels of profitable customer satisfaction through the use of knowledge generated from corporate and external data to help drive a more positive Customer Experience (CX). • Churn Management must have segmented customers based on years of detailed data in order to understand them based on specificity of purchase at a product, day, method, and order details levels. • This complete view comprises a majority of customer touchpoints purchases, promotions, payments, call center activity, GPS, incoming calls, and Web site hits and activity. • True analytics will be enabled as the organization embraces the fact that data is predictive and not just descriptive of the past. • Customer Churn is about understanding the customer experience. With it, organizations can gain insights about tastes and habits, interactions can be in real-time with personalized, relevant data and companies can understand the complete customer journey to churn.
- 16. Customer Churn Example • An online mobile phone retailer has been looking at how it can reduce churn across its customer base. • First, this requires setting up the model and data environment; and second, pipelines are needed for training and inference. • A next step is to prepare the ground for both data and modeling, by putting in place an environment that can manage both within an iterative process. • Next, you need to identify and configure the data sources to be used by ML models. • Customer loyalty factors may be fed back into the model as variables. • During training and other steps, scripts can read from or write to datastores, and records of execution are saved as runs in the workspace, grouped under experiments. • By reviewing these experiments, the retailer can monitor results for applicability and effectiveness of insights. • Data scientists need to ensure that the models deliver the right prediction. 16
- 17. Machine Learning Data Warehouse Categorical Model (e.g. Decision Tree) Categorical Data Quantitative Data Split Quantitative Model (e.g. Regression) Train Train Score Score Evaluate Data Engineering Customer Data Customer Churn Architecture
- 18. Predictive Maintenance • Predictive models use known results to develop (or train) a model that can be used to predict outcomes. • Predictive maintenance solutions store sensor data from equipment in near real time and predict equipment failure ahead of time to provide continuity of service. • There are savings from not just preventing breakdowns but also from being able to optimize the supply chain and labor supporting the maintenance process. 18
- 19. Predictive Maintenance Example • A manufacturing company engineers, produces, and sells equipment to its customers. This equipment consists of complex mechanical, electrical, and hydraulic components. – All of these components have integrated sensors which monitor the activity and health of each part and send those signals in real-time over the customer’s network to a central server that processes the data. • The manufacturer would like to ingest all of this data and predict an imminent part failure at a customer site with a high degree of confidence and proactively respond by alerting their inside sales department to contact the customer and persuade them to order a replacement part in advance to prevent the part failure. • Streaming data and ingest will be required as a software application script that executes on the monitored equipment delivering the production and broadcasting of streaming. • A dataset must be used to create and “learn” the machine learning statistical model by analyzing past part failures and to preload into the solution’s database as history data. • Using machine learning algorithms, a statistical model (like logistic regression) that predicts the probability (on a scale from 0 to 1) of an imminent part failure (a binary outcome) based on the measurement data broadcasted by the part’s sensor. 19
- 20. Machine Learning Data Engineering Training Dataset Predictive Model (e.g. Logistic Regression) Train Evaluate Deploy Sensor Data Up Machines Failing/ed Machines n Machines Scores Actions Predictive Maintenance Architecture
- 21. Fraud Detection • Fraudulent Activity Detection is exponentially more effective when risk actions are taken immediately (i.e., stop the fraudulent transaction), instead of after the fact. • The solution, with high-quality, up-to-date, and reliable corporate information, can transform the reactive nature of risk management to proactive. • It enables a full lifecycle look at data, the ability to see antecedents well in advance, and pattern matching to current activity. 21
- 22. Fraud Detection Example • An IT security analyst wants to analyze network traffic deeper and discover the existence of unknown anomalies, threats, or more sophisticated fraudulent or malicious intent. • The analyst wants to use machine learning statistical analysis techniques to explore and discover outlying network traffic behavior and build an unsupervised model to detect and alert when traffic is outside the bounds of typical network behavior. • The solution needs to be self-contained and easy to deploy by a business-level user with average technical aptitude using step-by-step instructions. The solution will clearly show the ingestion of data and automated analytics that make the user aware of a business action that needs to be taken. • The use case for this Fraudulent and Malicious Activity Detection system will have the ability to ingest high volumes of log data, its analytical capability to query the data using SQL statements or with self-discovery AI techniques, and how this data can be applied in a real-world scenario. 22
- 23. Data Engineering Data Warehouse Data Engineering Machine Learning Categorical Model (e.g. Decision Tree) Categorical Data Quantitative Data Split Quantitative Model (e.g. Regression) Train Train Score Score Evaluate Historical Transaction Data Deploy Scores Real Time Transaction s Actions Fraud Detection Architecture
- 24. Supply Chain Management • Supply chain leaders are under constant pressure to reduce overall Supply Chain Management (SCM) costs while maintaining a flexible and diverse supplier ecosystem. • Optimizing cost and expectations to manage multi-channel demand, real-time forecasts and inventory tracking is critical to prevent gaps or breaks in the supply chain that would disrupt downstream processes and operations. • Layered and complex dependencies are common in supply chain optimization — from just-in-time deliveries of goods, to supply planning, to distribution network planning. • A company must represent these interconnected factors and find the most appropriate suppliers or supply channels, remodel distribution, and identify alternative parts compared for evolving engineering specifications. 24
- 25. Supply Chain Management Example • A company designs solutions for product analytics and manufacturing operations in the semiconductor industry. • They need to leverage advanced product analytics that improve quality, yield, and productivity for semiconductor and electronics brand owners and their suppliers. • The company needs to conduct deep analysis that leverages both ‘hot’ data, originating in real-time from multiple plant floor assets, as well as suppliers, inventory and ‘cold’, historical data from the supply chain. • The solution will accelerate the speed of analytical insights, while also combining data from different production processes to gain visibility across the whole supply chain. 25
- 26. Data Engineering Machine Learning Data Warehouse Data Engineering Historical Order Data Train Train Score Score Evaluate Deploy Scores Real Time Orders Actions Sensor Data Status OK Problems n Sensors Supply Chain Management Architecture
- 27. Summary • After several years of stale development, competition caught up to Redshift, but at the end of 2019, Amazon released RA3 nodes which are built on the AWS Nitro System and feature high bandwidth networking and performance, reportedly indistinguishable from bare metal. • Azure Machine Learning is heavily slanted towards what they call “MLOps” or making the operational tasks of machine learning and data science more streamlined and efficient. • Heterogenous approaches are not all bad as there are some interoperability beacons of hope. • Predictive models use known results to develop (or train) a model that can be used to predict outcomes. • Major investments in advanced analytics, warehouse re/location and automation both in distribution centers and stores will be essential for survival. 27
- 28. Platforming the Major Analytic Use Cases for Modern Engineering Presented by: William McKnight President, McKnight Consulting Group williammcknight www.mcknightcg.com (214) 514-1444