Anomaly detection with machine learning at scale
9 likes5,481 views
The document outlines the significance of anomaly detection using machine learning in various sectors such as manufacturing, cybersecurity, and healthcare, detailing the challenges and lifecycle of the anomaly detection process. It highlights the different techniques involved, including supervised, semi-supervised, and unsupervised learning, along with the specific steps in building an anomaly detection system. Additionally, it discusses the obstacles enterprises face in fully adopting machine learning and introduces StreamAnalytix as a solution to simplify and accelerate application development.
Anomaly detection with machine learning at scale
- 1. Copyright © 2019 Impetus Technologies, Inc. You are prohibited from making a copy or modification of, or from redistributing, rebroadcasting, or re-encoding of this content without the prior written consent of Impetus. This presentation may include images from other products and services. These images are used for illustrative purposes only. Unless explicitly stated there is no implied endorsement or sponsorship of these products by Impetus. All copyrights and trademarks are property of their respective owners.
- 2. Anomaly Detection with Machine Learning at Scale
- 3. Our mission Enabling a unified, clear, and present view of your business
- 4. Agenda Background Steps and techniques Enterprise challenges and solutions Anomaly detection lifecycle Implementation on StreamAnalytix Q&A
- 5. Poll
- 6. Speakers Saurabh Dutta Technical Product Manager – StreamAnalytix Impetus Technologies Richa Pathak Lead Engineer – Data Science Impetus Technologies
- 7. What is an anomaly? Deviation in an event from its expected value within a group
- 8. Application of anomaly detection Manufacturing Cyber security Banking and finance Healthcare Travel
- 9. Identification of anomalies Normal Anomalous Temperature Temperature Visits Visits Normal ?
- 10. Why is it difficult? Normal behavior changes over time Anomalies differ with domains Noise tends to be similar to anomalies
- 11. Machine learning for anomaly detection Simple yet effective approach for detecting and classifying anomalies
- 12. Steps for anomaly detection with ML Categories Data characteristics Techniques Target
- 13. Step 1: Categories Point anomaly Contextual anomaly Collective anomaly O3 O2 O1 N1 N2
- 14. Step 2: Data characteristics Data Numerical Discrete Continuous Categorical Nominal Ordinal Binary Tid Source IP Duration Destination IP Bytes Internal 1 206.163.37.81 0.10 160.94.179.208 150 No 2 206.163.37.99 0.27 160.94.179.235 208 No 3 160.94.123.45 1.23 160.94.179.221 195 Yes 4 206.163.37.37 112.03 160.94.179.253 199 No 5 206.163.37.41 0.32 160.94.179.244 181 No
- 15. Step 3: Techniques Supervised (Classification) Semi-supervised (Novelty detection) Unsupervised (Clustering) Labeled data Subset of labelled data Unlabeled data
- 16. Supervised learning Supervised (Classification) Test data Training data Result Model Neural networks Decision tree SVM Bayesian network K-nearest neighbor Supervised (Classification)
- 17. Semi-supervised learning Semi-supervised (Novelty detection) Test data Training data Result Model Clustering Gaussian SVM Tree-based Neural Semi-supervised (Novelty detection)
- 18. Unsupervised learning K-means Bisecting k-means Autoencoder Unsupervised (Clustering) Unlabeled data Result Unsupervised (Clustering) Unsupervised algorithms
- 19. Step 4: Target Scores NormalAnomalous 0.71 AnomalyNormal Labels Target: Class Target: Score
- 20. Blueprint of an anomaly detection system OutputRaw data Insights Anomaly detection system Feature selection ML technique Detection and alert Data pre- processing Evaluation
- 21. Anomaly detection algorithms Credit card fraud detection Clustering Decision trees SVM Neural networks Health monitoring Nearest neighbor technique Naïve bayes Parametric statistical modeling Neural networks Fault detection in mechanical units Random forest Gradient boosted trees Spectral methods Neural networks
- 22. Recap
- 23. Poll results
- 24. Why have enterprises not been able to adopt machine learning completely?
- 25. Challenges Finding the right talent Building a solution, not a model Agility Consolidation of data Cross department expertise Maintenance
- 26. Solutions Finding the right talent—Extremely easy to use and create solutions Building a solution not a model—Ability to prototype and create solutions Agility—Rapidly develop and operationalize Consolidation of data—Connect to a variety of sources to enrich data Cross department expertise—Collaboration capabilities Maintenance—Retraining/versioning/upgrades
- 27. StreamAnalytix Self-service ETL + Analytics Simplify and accelerate application development
- 28. Raw data Structured data Pre- processing Feature selection Modeling technique Evaluation Tuning Anomaly detection lifecycle Training
- 29. Anomaly detection lifecycle Performance Metrics • RMSE • R2 error • Explained variance Confusion matrix Metrics • Precision • Recall • Specificity • Accuracy Actual vs. Predicted Regression Feature coefficients ROC curve Feature importance Classification
- 30. Anomaly detection lifecycle Scoring ETL pipeline Model scoring BI & downstream apps Kafka Register as Table SQL TempStore CustomerHistory
- 31. Anomaly detection lifecycle Model management Model selection DEV PROD Model deploymentRetraining & versioning
- 32. Machine failure prediction use case Identify potential failure in machines
- 34. Demo
- 35. StreamAnalytix platform approach Reusability Monitoring and debugging Promoting from Dev to Test to Production Orchestration of multiple workflows Auditing user changes On-premise or cloud Scalability and elasticity
- 37. Thank you. Questions? Visit www.streamanalytix.com or get in touch with us at [email protected] Meet us at the Strata Data Conference NY, Booth #1321, September 24–26
Editor's Notes
- #4: It is important to identify unusual cases within data that are homogeneous and may have some significance in respective domain. Anomaly detection is used widely to perform various tasks such as fraud detection in the financial industry, network breach for cyber-security, and enemy surveillance for the military https://www.google.com/imgres?imgurl=https%3A%2F%2Fblue-sea-697d.quartiers047.workers.dev%3A443%2Fhttps%2Fimages.slideplayer.com%2F26%2F8632187%2Fslides%2Fslide_2.jpg&imgrefurl=https%3A%2F%2Fblue-sea-697d.quartiers047.workers.dev%3A443%2Fhttps%2Fslideplayer.com%2Fslide%2F8632187%2F&docid=fc_EKZQGxy3rzM&tbnid=XTUWmOTndm_YJM%3A&vet=10ahUKEwj2zYzj0fDjAhXNSH0KHQYfAT8QMwhBKAIwAg..i&w=960&h=720&bih=610&biw=1280&q=anomaly%20detection%20in%20cyber%20security&ved=0ahUKEwj2zYzj0fDjAhXNSH0KHQYfAT8QMwhBKAIwAg&iact=mrc&uact=8
- #5: It is important to identify unusual cases within data that are homogeneous and may have some significance in respective domain. Anomaly detection is used widely to perform various tasks such as fraud detection in the financial industry, network breach for cyber-security, and enemy surveillance for the military https://www.google.com/imgres?imgurl=https%3A%2F%2Fblue-sea-697d.quartiers047.workers.dev%3A443%2Fhttps%2Fimages.slideplayer.com%2F26%2F8632187%2Fslides%2Fslide_2.jpg&imgrefurl=https%3A%2F%2Fblue-sea-697d.quartiers047.workers.dev%3A443%2Fhttps%2Fslideplayer.com%2Fslide%2F8632187%2F&docid=fc_EKZQGxy3rzM&tbnid=XTUWmOTndm_YJM%3A&vet=10ahUKEwj2zYzj0fDjAhXNSH0KHQYfAT8QMwhBKAIwAg..i&w=960&h=720&bih=610&biw=1280&q=anomaly%20detection%20in%20cyber%20security&ved=0ahUKEwj2zYzj0fDjAhXNSH0KHQYfAT8QMwhBKAIwAg&iact=mrc&uact=8
- #8: 1) Anomalies might be induced in the data for a variety of reasons, such as malicious activity, e.g., credit card fraud, cyber-intrusion, terrorist activity or breakdown of a system. From Marketer’s point of view the examples of anomalies would be : Decrease in number of sales of movie ticket. Rise in the number of downloads of app. Increase in number of visits to a website. The anomalies can be positive and negative. It is applicable in domains such as fraud detection, intrusion detection, fault detection, system health monitoring and event detection systems in sensor networks.
- #9: It is important to identify unusual cases within data that are homogeneous and may have some significance in respective domain. Anomaly detection is used widely to perform various tasks such as fraud detection in the financial industry, network breach for cyber-security, and enemy surveillance for the military https://www.google.com/imgres?imgurl=https%3A%2F%2Fblue-sea-697d.quartiers047.workers.dev%3A443%2Fhttps%2Fimages.slideplayer.com%2F26%2F8632187%2Fslides%2Fslide_2.jpg&imgrefurl=https%3A%2F%2Fblue-sea-697d.quartiers047.workers.dev%3A443%2Fhttps%2Fslideplayer.com%2Fslide%2F8632187%2F&docid=fc_EKZQGxy3rzM&tbnid=XTUWmOTndm_YJM%3A&vet=10ahUKEwj2zYzj0fDjAhXNSH0KHQYfAT8QMwhBKAIwAg..i&w=960&h=720&bih=610&biw=1280&q=anomaly%20detection%20in%20cyber%20security&ved=0ahUKEwj2zYzj0fDjAhXNSH0KHQYfAT8QMwhBKAIwAg&iact=mrc&uact=8
- #10: which is time series data collected from temperature sensors in a factory
- #11: https://towardsdatascience.com/a-note-about-finding-anomalies-f9cedee38f0b
- #12: Now we understand the challenge in Anomaly detection problem, so first we need to identify what is Normal. We can define the definition of Normal depending upon the application but we not only define what is Normal but we also build the model to identify normal pattern is and then take a decision about the abnormal behaviour/pattern. This is where Machine learning comes into picture.
- #13: Model development for anomaly detection depends on:
- #14: Anomaly detection is related to, but distinct from noise removal which deal with unwanted removal of noise in the data. Point Anomaly: If an object significantly deviates from the rest of the data. A single instance of data is anomalous if it's too far off from the rest. Business use case: Detecting credit card fraud based on "amount spent.“ Contextual Anomaly: If an individual data instance is anomalous in a specific context(but not otherwise). The abnormality is context specific. This type of anomaly is common in time-series data. Business use case: Spending $100 on food every day during the holiday season is normal, but may be odd otherwise. Collective Anomaly: If a collection of related data instances is anomalous with respect to entire data set. A set of data instances collectively helps in detecting anomalies. Business use case: Someone is trying to copy data form a remote machine to a local host unexpectedly, an anomaly that would be flagged as a potential cyber attack. https://www.datascience.com/blog/python-anomaly-detection
- #15: Need to work on this.
- #20: An important aspect for any anomaly detection technique is the manner in which the anomalies are reported. Label: Each observation is assigned with a label. True for classification/supervised based approaches. Scores: Each observation is assigned with an anomalous score. Output is an ranked list of anomalies Use a cut-off threshold to select the anomalies
- #26: Finding the right talent hinders the effectiveness of the solution. Performance metrics in SAX can help https://www.analyticsindiamag.com/10-challenges-that-data-science-industry-still-faces/
- #27: https://www.analyticsindiamag.com/10-challenges-that-data-science-industry-still-faces/ Maintenance https://www.forbes.com/sites/laurencebradford/2018/09/06/8-real-challenges-data-scientists-face/#360466b6d999
- #30: Model Development Aspects/Lifecycle
- #31: Model Development Aspects/Lifecycle
- #33: https://archive.ics.uci.edu/ml/machine-learning-databases/heart-disease/ This database contains 76 attributes, but all published experiments refer to using a subset of 14 of them
- #36: Scalability due to distributed computing and elasticity is built in scale in scale out Reusability of logic