Building a Large Scale Recommendation Engine with Spark and Redis-ML with Shay Nativ
3 likes•2,218 views
The document outlines the use of Redis and Redis-ML for real-time machine learning with Apache Spark, detailing an agenda that covers Redis introduction, model serving, and building recommendation systems. It emphasizes Redis's ease of use, performance, and scalability for machine learning applications, specifically through dynamic modules like Redis-ML that enhance data handling. The document concludes with a case study of a movie recommendation system, illustrating the integration of Spark training and Redis-ML for efficient model serving and recommendations.
![A Quick Recap of Redis
Key
"I'm a Plain Text String!"
{ A: “foo”, B: “bar”, C: “baz” }
Strings / Bitmaps / BitFields
Hash Tables (objects!)
Linked Lists
Sets
Sorted Sets
Geo Sets
HyperLogLog
{ A , B , C , D , E }
[ A → B → C → D → E ]
{ A: 0.1, B: 0.3, C: 100, D: 1337 }
{ A: (51.5, 0.12), B: (32.1, 34.7) }
00110101 11001110 10101010](https://image.slidesharecdn.com/039shaynativ-170614012005/85/Building-a-Large-Scale-Recommendation-Engine-with-Spark-and-Redis-ML-with-Shay-Nativ-7-320.jpg)
![Using Redis-ML With Spark
scala> import com.redislabs.client.redisml.MLClient
scala> import com.redislabs.provider.redis.ml.Forest
scala> val jedis = new Jedis("localhost")
scala> val rfModel = pipelineModel.stages.last.asInstanceOf[RandomForest]
// Create a new forest instance
scala> val f = new Forest(rfModel.trees)
// Load the model to redis
scala> f.loadToRedis("forest-test", "localhost")
// Classify a feature vector
scala> jedis.getClient.sendCommand(MLClient.ModuleCommand.FOREST_RUN,
"forest-test", makeInputString (0))
scala> jedis.getClient.getStatusCodeReply
res53: String = 1](https://image.slidesharecdn.com/039shaynativ-170614012005/85/Building-a-Large-Scale-Recommendation-Engine-with-Spark-and-Redis-ML-with-Shay-Nativ-23-320.jpg)
![Step3: Train and Load to Redis
// Create a new forest instance
val rf = new
RandomForestClassifier().setFeatureSubsetStrategy("auto").setLabelCol("indexedLabel").setFeat
uresCol("indexedFeatures").setNumTrees(500)
…..
// Train model
val model = pipeline.fit(trainingData)
…..
val rfModel = model.stages(2).asInstanceOf[RandomForestClassificationModel]
// Load the model to redis
val f = new Forest(rfModel.trees)
f.loadToRedis(”movie-10", "127.0.0.1")
33](https://image.slidesharecdn.com/039shaynativ-170614012005/85/Building-a-Large-Scale-Recommendation-Engine-with-Spark-and-Redis-ML-with-Shay-Nativ-33-320.jpg)
![Getting Actual Recommendations - Python Script
#!/usr/bin/python
import operator
import redis
config = {"host":"localhost", "port":6379}
r = redis.StrictRedis(**config)
user_profile = r.get("user-1-profile")
results = {}
for i in range(1, 11):
results[i] = r.execute_command("ML.FOREST.RUN", "movie-{}".format(i), user_profile)
print "Movies sorted by scores:"
sorted_results = sorted(results.items(), key=operator.itemgetter(1), reverse=True)
for k,v in sorted_results:
print "movie-{}:{}".format(k,v)
print ""
print "Recommended movie: movie-{}".format(sorted_results[0][0])](https://image.slidesharecdn.com/039shaynativ-170614012005/85/Building-a-Large-Scale-Recommendation-Engine-with-Spark-and-Redis-ML-with-Shay-Nativ-38-320.jpg)
Building a Large Scale Recommendation Engine with Spark and Redis-ML with Shay Nativ
- 1. Shay Nativ - Redis Labs Real-time Machine Learning with Redis-ML and Apache Spark
- 2. Agenda ● Intro to Redis and Redis Labs - 5 min ● Using Redis-ML for Model Serving - why and how - 10 min ● Building a recommendation system using Spark-ML and Redis-ML - 10 min ● QA 2
- 3. Redis Labs – Home of Redis Founded in 2011 HQ in Mountain View CA, R&D center in Tel-Aviv IL The commercial company behind Open Source Redis Provider of the Redis Enterprise (Redise ) technology, platform and products 3
- 4. Redise Cloud Private Redis Labs Products Redise Cloud Redise Pack ManagedRedise Pack SERVICES SOFTWARE Fully managed Redise service in VPCs within AWS, MS Azure, GCP & IBM Softlayer Fully managed Redise service on hosted servers within AWS, MS Azure, GCP, IBM Softlayer, Heroku, CF & OpenShift Downloadable Redise software for any enterprise datacenter or cloud environment Fully managed Redise Pack in private data centers && &
- 5. A Brief Overview of Redis ● Started in 2009 by Salvatore Sanfilippo ● Most popular KV store ● In memory - disk backed ● Notable Users: ○ Twitter, Netflix, Uber, Groupon, Twitch ○ Many, many more...
- 6. Redis Main Differentiations Simplicity (through Data Structures) Extensibility (through Redis Modules) Performance ListsSorted Sets Hashes Hyperlog-logs Geospatial Indexes Bitmaps SetsStrings Bit field 6
- 7. A Quick Recap of Redis Key "I'm a Plain Text String!" { A: “foo”, B: “bar”, C: “baz” } Strings / Bitmaps / BitFields Hash Tables (objects!) Linked Lists Sets Sorted Sets Geo Sets HyperLogLog { A , B , C , D , E } [ A → B → C → D → E ] { A: 0.1, B: 0.3, C: 100, D: 1337 } { A: (51.5, 0.12), B: (32.1, 34.7) } 00110101 11001110 10101010
- 8. Simple Redis Example (string, hash) 127.0.0.1:6379> SET spark summit OK 127.0.0.1:6379> GET spark "summit" 127.0.0.1:6379> HMSET spark_hash org apache version 2.1.1 OK 127.0.0.1:6379> HGET spark_hash version "2.1.1" 127.0.0.1:6379> HGETALL spark_hash 1) "org" 2) "apache" 3) "version" 4) "2.1.1"
- 9. Another Simple Redis Example (sorted set) 127.0.0.1:6379> zadd my_sorted_set 1 foo (integer) 1 127.0.0.1:6379> zadd my_sorted_set 5 bar (integer) 1 127.0.0.1:6379> zadd my_sorted_set 3 baz (integer) 1 127.0.0.1:6379> ZRANGE my_sorted_set 0 2 1) "foo" 2) "baz" 3) "bar" 127.0.0.1:6379>
- 10. New Capabilities What Modules Actually Are • Dynamic libraries loaded to redis • Written in C/C++ • Use a C ABI/API isolating redis internals • Use existing or add new data-structures • Near Zero latency access to data New Commands New Data Types
- 11. Modules : A Revolutionary Approach Adapt your database to your data, not the other way around Secure way to store data in Redis via encrypt/decrypt with various Themis primitives Time Series Graph Time series values aggregation in Redis Crypto Engine Wrapper Graph database on Redis based on Cypher language Based on Generic Cell Rate Algorithm (GCRA) Rate Limiter ReJSON Secondary Index/RQL JSON Engine on Redis. Pre-released Indexing + SQL -like syntax for querying indexes. Pre-released Neural Redis Redis-ML RediSearch Full Text Search Engine in RedisMachine Learning Model Serving Simple Neural Network Native to Redis
- 12. Redis ML Machine Learning Model Server
- 13. Spark-ML End-to-End Flow Spark Training Custom Server Model saved to Parquet file Data Loaded to Spark Pre-computed results Batch Evaluation ? ClientApp
- 14. ML Models Serving Challenges • Models are becoming bigger and more complex • Can be challenging to deploy & serve • Do not scale well, speed and size • Can be very expensive 14
- 15. A Simpler Machine Learning Lifecycle Any Training Platform Data loaded into Spark Model is saved in Redis-ML Redis-ML Serving Client Client App Client App Client App + Spark Training 15
- 16. Redis-ML – ML Serving Engine • Store training output as “hot model” • Perform evaluation directly in Redis • Enjoy the performance, scalability and HA of Redis 16
- 17. ML Models Tree Ensembles Linear Regression Logistic Regression Matrix + Vector Operations More to come... Redis-ML 17
- 18. Random Forest Model • A collection of decision trees • Supports classification & regression • Splitter Node can be: ◦ Categorical (e.g. day == “Sunday”) ◦ Numerical (e.g. age < 43) • Decision is taken by the majority of decision trees 18
- 19. Titanic Survival Predictor on a Decision Tree YES Sex = Male ? Age < 9.5? *Sibps > 2.5? Survived Died SurvivedDied *Sibps = siblings + spouses NO 19
- 20. Titanic Survival Predictor on a Random Forest YES Sex = Male ? Age < 9.5? *Sibps > 2.5? Survived Died SurvivedDied NO YES Country= US? State = CA? Height> 1.60m? Survived Died SurvivedDied NO YES Weight< 80kg? I.Q<100? Eye color =blue? Survived Died SurvivedDied NO Tree #1 Tree #2 Tree #3
- 21. Would John Survive The Titanic • John’s features: {male, 34, married + 2, US, CA, 1.78m, 78kg, 110iq, blue eyes} • Tree#1 – Survived • Tree#2 – Failed • Tree#3 – Survived • Random forest decision - Survived 21
- 22. Forest Data Type Example > MODULE LOAD "./redis-ml.so" OK > ML.FOREST.ADD myforest 0 . CATEGORIC sex “male” .L LEAF 1 .R LEAF 0 OK > ML.FOREST.RUN myforest sex:male "1" > ML.FOREST.RUN myforest sex:no_thanx "0"
- 23. Using Redis-ML With Spark scala> import com.redislabs.client.redisml.MLClient scala> import com.redislabs.provider.redis.ml.Forest scala> val jedis = new Jedis("localhost") scala> val rfModel = pipelineModel.stages.last.asInstanceOf[RandomForest] // Create a new forest instance scala> val f = new Forest(rfModel.trees) // Load the model to redis scala> f.loadToRedis("forest-test", "localhost") // Classify a feature vector scala> jedis.getClient.sendCommand(MLClient.ModuleCommand.FOREST_RUN, "forest-test", makeInputString (0)) scala> jedis.getClient.getStatusCodeReply res53: String = 1
- 24. Real World Challenge • Ad serving company • Need to serve 20,000 ads/sec @ 50msec data-center latency • Runs 1k campaigns → 1K random forest • Each forest has 15K trees • On average each tree has 7 levels (depth) • Would require < 1000 x c4.8xlarge 24
- 25. Redis ML with Spark ML Classification Time Over Spark 40x Faster 25
- 26. Real World Example: Movie Recommendation System
- 28. Concept: One Forest For Each Movie 28 User Features: (Age, Gender, Movie Ratings) Movie_1 Forest 3 Movie_2 Forest 2 Movie_n Forest 5 . . .
- 30. Using the Dockers $ docker pull shaynativ/redis-ml $ docker run --net=host shaynativ/redis-ml & $ $ docker pull shaynativ/spark-redis-ml $ docker run --net=host shaynativ/spark-redis-ml
- 31. Step 1: Get The Data 31 • Download and extract the MovieLens 100K Dataset • The data is organized in separate files: • Ratings: user id | item id | rating (1-5) | timestamp • Item (movie) info: movie id | genre info fields (1/0) • User info: user id | age | gender | occupation • Our classifier should return the expected rating (from 1 to 5) a user would give the movie in question
- 32. Step 2: Transform 32 • The training data for each movie should contain 1 line per user: • class (rating from 1 to 5 the user gave to this movie) • user info (age, gender, occupation) • user ratings of other movies (movie_id:rating ...) • user genre rating averages (genre:avg_score ...) • Run gen_data.py to transform the files to the desired format
- 33. Step3: Train and Load to Redis // Create a new forest instance val rf = new RandomForestClassifier().setFeatureSubsetStrategy("auto").setLabelCol("indexedLabel").setFeat uresCol("indexedFeatures").setNumTrees(500) ….. // Train model val model = pipeline.fit(trainingData) ….. val rfModel = model.stages(2).asInstanceOf[RandomForestClassificationModel] // Load the model to redis val f = new Forest(rfModel.trees) f.loadToRedis(”movie-10", "127.0.0.1") 33
- 34. Step 3: Execute in Redis Redis-ML + Spark Training Client App
- 35. Python Client Example >> import redis >> config = {"host":"localhost", "port":6379} >> r = redis.StrictRedis(**config) >> user_profile = r.get("user_shay_profile") >> print(user_profile) 12:1.0,13:1.0,14:3.0,15:1.0,17:1.0,18:1.0,19:1.0,20:1.0,23:1.0,24:5.0,1.0,115:1.0,116:2. 0,117:2.0,119:1.0,120:4.0,121:2.0,122:2.0, ........ 1360:1.0,1361:1.0,1362:1.0, 1701:6.0,1799:435.0,1801:0.2,1802:0.11,1803:0.04,1812:0.04,1813:0.07,1814:0.24,1815:0.09 ,1816:0.32,1817:0.06 >> r.execute_command("ML.FOREST.RUN", "movie-10", user_profile) '3'
- 36. Redis CLI Example >keys * 127.0.0.1:6379> KEYS * 1) "movie-5" 2) "movie-1" ........ 8) "movie-6" 9) "movie-4" 10) "movie-10" 11) "user_1_profile” >ML.FOREST.RUN movie-10 12:1.0,13:1.0,,332:3.0,333:1.0,334:1.0,335:2.0,336:1.0,357:2.0,358:1.0,359:1.0,362:1.0,367:1. ........ ,410:3.0,411:2.0,412:2.0,423:1.0,454:1.0,455:1.0,456:1.0,457:3.0,458:1.0,459:1.0,470:1” "3” >
- 37. Performance Redis time: 0.635129ms, res=3 Spark time: 46.657662ms, res=3.0 --------------------------------------- Redis time: 0.644444ms, res=3 Spark time: 49.028983ms, res=3.0 --------------------------------------- Classification averages: redis: 0.9401250000000001 ms spark: 58.01970206666667 ms ratio: 61.71488053893542 diffs: 0.0
- 38. Getting Actual Recommendations - Python Script #!/usr/bin/python import operator import redis config = {"host":"localhost", "port":6379} r = redis.StrictRedis(**config) user_profile = r.get("user-1-profile") results = {} for i in range(1, 11): results[i] = r.execute_command("ML.FOREST.RUN", "movie-{}".format(i), user_profile) print "Movies sorted by scores:" sorted_results = sorted(results.items(), key=operator.itemgetter(1), reverse=True) for k,v in sorted_results: print "movie-{}:{}".format(k,v) print "" print "Recommended movie: movie-{}".format(sorted_results[0][0])
- 39. Getting Actual Recommendations - Results $ ./classify_user.py 1 Movies sorted by scores: movie-4:3 movie-3:2 movie-6:2 movie-7:2 movie-8:2 movie-9:2 movie-1:1 movie-2:1 movie-5:1 movie-10:0 Recommended movie for user 1: movie-4
- 40. Summary • Train with Spark, Serve with Redis • 97% resource cost serving • Simplify ML lifecycle • Redise (Cloud or Pack): ‒Scaling, HA, Performance ‒PAYG – cost optimized ‒Ease of use ‒Supported by the teams who created Spark and Redis Spark Training Data loaded into Spark Model is saved in Redis-ML Redis-ML Serving Client Client App Client App Client App +
- 41. Resources ● Redis-ML: https://github.com/RedisLabsModules/redis-ml ● Spark-Redis-ML: https://github.com/RedisLabs/spark-redis-ml ● Databricks Notebook: http://bit.ly/sparkredisml ● Dockers: https://hub.docker.com/r/shaynativ/redis-ml/ https://hub.docker.com/r/shaynativ/spark-redis-ml/
- 42. Q&A