Machine Learning Trends of 2018 combined with the Apache Kafka Ecosystem
15 likes17,183 views
The document discusses the integration of machine learning with the Apache Kafka ecosystem, highlighting its applications in real-world scenarios such as spam detection and predictive maintenance. It covers the building and deployment of neural networks using frameworks like TensorFlow and H2O, as well as the use of Kafka Streams for real-time data processing. Key takeaways emphasize the importance of collaboration between data scientists and developers for successful machine learning projects.
Machine Learning Trends of 2018 combined with the Apache Kafka Ecosystem
- 1. 1Confidential Highly scalable machine learning and deep learning in real time with open source frameworks Kai Waehner Technology Evangelist [email protected] LinkedIn @KaiWaehner www.confluent.io www.kai-waehner.de
- 2. 2Apache Kafka and Machine Learning – Kai Waehner Agenda 1) Machine Learning and Real World Applications 2) Machine Learning and the Apache Kafka Ecosystem 3) Building Neural Networks with TensorFlow and H2O 4) Deployment of Neural Networks with Kafka Streams
- 3. 3Apache Kafka and Machine Learning – Kai Waehner Agenda 1) Machine Learning and Real World Applications 2) Machine Learning and the Apache Kafka Ecosystem 3) Building Neural Networks with TensorFlow and H2O 4) Deployment of Neural Networks with Kafka Streams
- 4. 4Apache Kafka and Machine Learning – Kai Waehner Machine Learning ... allows computers to find hidden insights without being explicitly programmed where to look. Machine Learning • Decision Trees • Naïve Bayes • Clustering • Neural Networks • etc. Deep Learning • CNN • RNN • Autoencoder • etc.
- 5. 5Apache Kafka and Machine Learning – Kai Waehner Neural Network compared to other algorithms
- 6. 6Apache Kafka and Machine Learning – Kai Waehner Real World Examples of Machine Learning Spam Detection Search Results + Product Recommendation Picture Detection (Friends, Locations, Products) Your Company The Next Disruption: Google Beats Go Champion
- 7. 7Apache Kafka and Machine Learning – Kai Waehner Leverage Machine Learning to Analyze and Act on Critical Business Moments Seconds Minutes Hours Price Optimization Predictive Maintenance Fraud Detection Cross Selling Transportation Rerouting Customer Service Inventory Management Windows of Opportunity
- 8. 8Apache Kafka and Machine Learning – Kai Waehner Live Demo – Building an Analytic Model Neural Networks in Action http://playground.tensorflow.org/
- 9. 9Apache Kafka and Machine Learning – Kai Waehner Agenda 1) Machine Learning and Real World Applications 2) Machine Learning and the Apache Kafka Ecosystem 3) Building Neural Networks with TensorFlow and H2O 4) Deployment of Neural Networks with Kafka Streams
- 10. 10Apache Kafka and Machine Learning – Kai Waehner Hidden Technical Debt in Machine Learning Systems https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf
- 11. 11Apache Kafka and Machine Learning – Kai Waehner The Log ConnectorsConnectors Producer Consumer Streaming Engine Apache Kafka – A Streaming Platform
- 12. 12Apache Kafka and Machine Learning – Kai Waehner Apache Kafka’s Open Source Ecosystem and Machine Learning Kafka Streams Kafka Connect Rest Proxy Schema Registry Go / .NET / Python Kafka Producer KSQL Kafka Streams
- 13. 13Apache Kafka and Machine Learning – Kai Waehner Uber’s internal ML-as-a-Service Platform https://eng.uber.com/michelangelo/ • Cover the end-to-end ML workflow: manage data, train, evaluate, and deploy models, make predictions, and monitor predictions • Supports various AI technologies: Traditional ML models, time series forecasting, and deep learning
- 14. 14Apache Kafka and Machine Learning – Kai Waehner Netflix’ Meson: Automation Engine for ML Pipelines https://www.infoq.com/presentations/netflix-ml-meson
- 15. 15Apache Kafka and Machine Learning – Kai Waehner Agenda 1) Machine Learning and Real World Applications 2) Machine Learning and the Apache Kafka Ecosystem 3) Building Neural Networks with TensorFlow and H2O 4) Deployment of Neural Networks with Kafka Streams
- 16. 16Apache Kafka and Machine Learning – Kai Waehner Languages, Frameworks and Tools for Machine Learning There is no Allrounder! Portable Format for Analytics (PFA)
- 17. 17Confidential Machine Learning with H2O.ai H2O Engine R / Python / Scala / Flow UI Java Code
- 18. 18Apache Kafka and Machine Learning – Kai Waehner Live Demo Use Case: Airline Flight Delay Prediction Machine Learning Algorithm: Deep Learning using Neural Networks Technology: H2O.ai, TensorFlow
- 19. 19Apache Kafka and Machine Learning – Kai Waehner H2O Deep Water (TensorFlow, MXNet, …) https://h2o-release.s3.amazonaws.com/h2o/rel-vapnik/1/ docs-website/h2o-docs/booklets/DeepWaterBooklet.pdf Deep Water (H2O + TensorFlow) Pre-Defined Networks + User-Defined Networks
- 20. 20Apache Kafka and Machine Learning – Kai Waehner Agenda 1) Machine Learning and Real World Applications 2) Machine Learning and the Apache Kafka Ecosystem 3) Building Neural Networks with TensorFlow and H2O 4) Deployment of Neural Networks with Kafka Streams
- 21. 21Apache Kafka and Machine Learning – Kai Waehner Stream Processing Data at Rest Data in Motion
- 22. 22Apache Kafka and Machine Learning – Kai Waehner Stream Processing Pipeline APIs Adapters / Channels Integration Messaging Stream Ingest Transformation Aggregation Enrichment Filtering Stream Preprocessing Process Management Analytics (Real Time) Applications & APIs Analytics / DW Reporting Stream Outcomes • Contextual Rules • Windowing • Patterns • Analytics • Machine Learning • … Stream Analytics Index / SearchNormalization Applying an Analytic Model is just a piece of the puzzle!
- 23. 23Apache Kafka and Machine Learning – Kai Waehner Kafka Streams (shipped with Apache Kafka) Map, filter, aggregate, apply analytic model, „any business logic“ Input Stream (Kafka Topic) Kafka Cluster Output Stream (Kafka Topic) Kafka Cluster Stream Processing Microservice (Kafka Streams) Deployed Anywhere Java App, Docker, Kubernetes, Mesos, “you-name-it”
- 24. 24Apache Kafka and Machine Learning – Kai Waehner When to use Kafka Streams for Stream Processing?
- 25. 25Apache Kafka and Machine Learning – Kai Waehner A complete streaming microservices, ready for production at large-scale Word Count App configuration Define processing (here: WordCount) Start processing
- 26. 26Apache Kafka and Machine Learning – Kai Waehner Use Case: Airline Flight Delay Prediction Machine Learning Algorithm: Neural Network built with H2O and TensorFlow Streaming Platform: Apache Kafka and Kafka Streams Live Demo
- 27. 27Apache Kafka and Machine Learning – Kai Waehner H2O.ai Model + Kafka Streams Filter Map 1) Create H2O DL model 2) Configure Kafka Streams Application 3) Apply H2O DL model to Streaming Data 4) Start Kafka Streams App
- 28. 28Apache Kafka and Machine Learning – Kai Waehner Github Examples: Kafka + Machine Learning https://github.com/kaiwaehner/kafka-streams-machine-learning-examples 1) git clone … 2) mvn clean package … 3) look at implementations and unit tests
- 29. 29Apache Kafka and Machine Learning – Kai Waehner Online Model Training with Apache Kafka and Kafka Streams How to improve models? 1. Manual Update 2. Automated Batch 3. Real Time Your choice… All possible with Kafka!
- 30. 30Apache Kafka and Machine Learning – Kai Waehner Caveats for Online Model Training • Processes and infrastructure not ready • Validation needed before production • Slows down the system • Only a few ML implementations à Build your own! • Only possible for unsupervised ML (e.g. clustering) • Many use cases do not need it à Do it only when feasible!
- 31. 31Apache Kafka and Machine Learning – Kai Waehner Key Take-Aways à Data Scientist and Developers have to work together continuously (org + tech!) à Mission critical, scalable production deployment is key for success of Machine Learning projects à Apache Kafka Ecosystem for Batch and Real Time Machine Learning (Training, Inference, Monitoring)
- 32. 32Apache Kafka and Machine Learning – Kai Waehner Kai Waehner Technology Evangelist [email protected] @KaiWaehner www.kai-waehner.de www.confluent.io LinkedIn Questions? Feedback? Please contact me!