Automated Machine Learning Using Spark Mllib to Improve Customer Experience-(Sourabh Chaki, [24]7
8 likes3,494 views
This document summarizes an automated machine learning framework for customer journey analysis. The framework uses GraphX to process event streams and connect discrete interactions into user journeys. It then applies feature engineering techniques like binning and frequent item counting to transform journey data into a format suitable for model building. Multiple models can be configured and built in Spark MLlib, with the best model selected using performance metrics. The selected model is stored for serving predictions to applications through a prediction server. The goal is to automate the full machine learning cycle from raw event streams to predictive models.
![Automated Machine
Learning @[24]7
Sourabh Chaki](https://image.slidesharecdn.com/07sourabhchaki-150624022615-lva1-app6892/85/Automated-Machine-Learning-Using-Spark-Mllib-to-Improve-Customer-Experience-Sourabh-Chaki-24-7-1-320.jpg)
val qtree = journeyViewRDD
.map(row=>QTree(row(productPriceIndex)))
.reduce(aggregator.plus(_,_) )
val median = qtree.quantileBounds(0.5)._1](https://image.slidesharecdn.com/07sourabhchaki-150624022615-lva1-app6892/85/Automated-Machine-Learning-Using-Spark-Mllib-to-Improve-Customer-Experience-Sourabh-Chaki-24-7-18-320.jpg)
Automated Machine Learning Using Spark Mllib to Improve Customer Experience-(Sourabh Chaki, [24]7
- 2. Anticipate, Simplify, Learn A Proactive Framework for Intuitive Customer Care
- 3. 3 Visitor Visitor Caller Homepage Product Page Cart Purchased Chat Offered Chat Accepted Chat Started Chat Ended Call Connected Call Ended Call Stats Event Discrete action Interaction Group of discrete events Product Page Web Chat Voice
- 4. Graph Problem Web Voice Chat Scattered Interactions Finding Connected Components Visitor 1 Journey Visitor 2 Journey
- 5. Graph processing in Hadoop • HashToMin for Connected Component – http://arxiv.org/pdf/1203.5387.pdf • Log(n) mapreduce iterations for connecting n diameter graph • Challenges: – Thinking graph problem in map and reduce – Not fit for iterative algorithms
- 6. GraphX • Spark fits for iterative programming • In-memory data • Think as edges and vertices • High level APIs for graph processing
- 7. Connected Component in GraphX • Vertices >> Edges – Graph from only edges – Interactions as external KVP RDD • val cc = graph.connectedComponents() • Join cc.vertices with interactions • Group interactions with leaders
- 8. Join • Broadcast join • Hash Partition Join • Adaptive – Small data: Broadcast – Large data: HashPartition
- 9. Memory Share • Default shuffle memory 20% – Low for shuffle heavy application • Shuffle memory: 40%, storage memory 40%
- 10. Performance Gain 0 10 20 30 40 50 60 2 4 8 Mins Graph Diameter Hadoop Vs Spark (Mins), 350M events Hadoop GraphX
- 11. Interactions to Journey web Voice Chat Events Connecting User Journeys User Journey Journey Views
- 12. User Journey to Model Journey View Pre-processing and model building Model
- 13. Feature Engineering • Training Set/Test Set • Balanced Sampling • Frequent Items • Quantile Bins • Transformation Function
- 15. Column wise data analysis Quantile TopK Frequent Sum (set1 , set2) = Sum(set1) + Sum(set2) Quantile(Set1 + Set2) = Quantile(Set1) + Quantile(Set2) TopKFrequent(set1, set2) = TopKFrequent(set1) + TopKFrequent(set2) Quantile TopK Frequent Quantile
- 16. Problems • Non distributed • Need data shuffle • Shuffle on different columns • High disk and network IO
- 17. Algebird with Spark F(set1 + set2) ≈ F(set1) + F(set2) • QTree • CountMinSketch https://github.com/twitter/algebird
- 18. Binning Product Price Product price => low / high value product val aggregator= new QTreeSemigroup[Double](8) val qtree = journeyViewRDD .map(row=>QTree(row(productPriceIndex))) .reduce(aggregator.plus(_,_) ) val median = qtree.quantileBounds(0.5)._1
- 19. Binning Product Price Product price => low / high value product class TransformationFunc(val median:Double) extends Serializable { def transform (value:Double): String = { if(value<median) "low" else "high" } } val transFunc = new TransformationFunc(median) val priceCat = transFunc.transform(productPrice)
- 20. Automated Feature Engineering Journey View Data Set and Transformation Function Feature Engineering Configs
- 21. Model Building • Spark MLlib • Random Forest for classification • Model Testing • Performance Metrics
- 23. Automated Model Building Multiple Models Model Configs Data Set and Transformation Function Model Testing Best Model Model Building
- 24. Prediction Entity • Transformation Function Object • Random Forest Model Object class PredictionEntity(model:RandomForestModel, tansformationFunc:TransformationFunc) extends Serializable { def predict(vector:Vector):Double = { //transform the vector using transformationFunc //predict using model } }
- 25. Prediction outside Spark • Need export – No PMML support • Model store • Synchronous prediction call
- 26. Prediction outside Spark Model Store Prediction Web Voice Chat Wrapper over MySql Application Server Serialized Byte Array Prediction Entity
- 27. Machine Learning Cycle Web Voice Chat Event Stream GraphX Mllib Journey View Model Model Store Prediction Server Configurations
- 28. Thank You