Taking the Pain out of Data Science - RecSys Machine Learning Framework Over Spark
0 likes121 views
1. The document discusses Outbrain's machine learning framework for personalized recommendations using Spark. It describes challenges in collecting and processing large datasets, building predictive models, and deploying models to production through A/B testing. 2. It outlines Outbrain's distributed machine learning framework for data collection, feature engineering, model training, evaluation, and deployment. Standardized model interfaces allow easy implementation of various algorithms and ensemble modeling. 3. The framework aims to streamline the research cycle and connect research to production through automated data preparation, simple model evaluation and simulation, and fast A/B testing and model updates in production.
Taking the Pain out of Data Science - RecSys Machine Learning Framework Over Spark
- 1. | CONTENT-BASED PERSONALIZATION Taking the Pain out of Data Science RecSys Machine Learning Framework Over Spark Sonya Liberman Personalization Team Lead Outbrain Recommendations Group
- 2. ? What Is
- 3. The Lighthouse Help people discover content they can trust to be interesting, relevant and timely for them
- 13. Agenda • Defining our machine learning challenge • Distributed Spark framework for machine learning • Deploying models to production
- 14. Know Your Reader
- 15. Oubtrain’s NLP Engine 3 MILLION NEW ARTICLES OVER
- 16. What is a Document About?
- 19. The Machine Learning Challenge Features Vector Supervision User Profile Click / No Click Past Interactions Current Context Supervised Machine Learning Algorithms
- 21. 1. Content Based Models Recommends content based on semantic similarity with user interests Predictive Models Music Tech Travel
- 22. 1. Content Based Models 2. Behavioural Models Finding behavioural patterns beyond a semantic connection Predictive Models Retirement Investing Health
- 23. 1. Content Based Models 2. Behavioural Models 3. Collaborative Models Recommend content that readers with your reading patterns like Predictive Models
- 24. 3. Collaborative Models Matrix Factorization Factorization Machines Feature Embedding with Deep Learning Predictive Models
- 25. Data Processing and Distributed Machine Learning Framework
- 26. Data Processing 3 Data Centers 300 Machines in each cluster 7 petabytes of data 5 terabytes of compressed new data daily Distributed Machine Learning Framework
- 27. Distributed Machine Learning Framework Data Collection Feature Engineering Model Training Offline Evaluation & Simulation Model Deployment 1 2 3 4 5
- 28. Distributed Machine Learning Framework Feature Engineering Model Training 2 3 Used for Research Used for Production Agile Development of new Models
- 29. Preparing datasets – hard work!
- 30. Data Collection challenges • Many output tables Requests, served recommendations, clicks, user profiles, document profiles.
- 31. Data Collection challenges • Many output tables Requests, served recommendations, clicks, user profiles, document profiles. • Multiple data stores Hive, MySQL, Cassandra
- 32. Data Collection challenges • Many output tables Requests, served recommendations, clicks, user profiles, document profiles. • Multiple data stores Hive, MySQL, Cassandra • Scale Huge tables, queries take long
- 33. Data Collection challenges • Many output tables Requests, served recommendations, clicks, user profiles, document profiles. • Multiple data stores Hive, MySQL, Cassandra • Scale Huge tables, queries take long • Irrelevant data We only need recommendations served on specific traffic variants
- 34. Silos and partitioning • Ability to train models on different silos Enabling fast retrieval of data for specific silos • Partitioning Adding a list of partitions for each listing for fast access • Partitions Platform, country, language, user activity level
- 35. Building Models Recs from specific variants Clicks + sample of no clicks Simulation – Business metrics Clicked widgets only All served recs Data Collection - Required datasets
- 36. Solution – automatic data collection • Hourly triggered job Generate an hourly dataset with only the relevant data • Join tables and partition Join the input tables and add partitions to enable silos • Split between model and simulation Create different datasets for modeling and simulation
- 37. Monitoring
- 38. Modeling boilerplate – time consuming
- 39. Modeling challenges and boilerplate • Handling input Reading, train-test split
- 40. Modeling challenges and boilerplate • Handling input Reading, train-test split • Evaluating test metrics Run model on test data, calculate test metrics
- 41. Modeling challenges and boilerplate • Handling input Reading, train-test split • Getting test metrics Run model on test data, calculate test metrics • Checking business metrics Simulate production use cases Estimate business metrics
- 43. Simple model interface Data scientists and algorithm engineers only need to implement their model’s logic. Spark.ML packages, open source implementations and “home made” algorithms can be used. Everything else - out-of-the-box
- 45. Ensemble modeling • Implementing multiple models Models that use different features or algorithms • Combine the models Run the models one after the other, or combine their output scores
- 46. 46 Bridging research and production
- 47. 47 Control Treatment Evaluation Criteria engagement & monetization metrics Statistical tests Traffic Split Models productization Fast and easy A/B testing
- 48. Models productization Fast and easy A/B testing Music Tech A B
- 49. Models productization Fast and easy A/B testing Promising offline metrics – New A/B test variant using the model. The Spark job ends with updating the A/B test configuration on the DB
- 50. Models productization Automatic coefficients tuning After a model has proven value, we want to update it regularly and keep learning based on new data. • Periodic model creation • Business metrics validation • Production configuration update
- 53. High scale modeling – prepare your data well! Takeaways
- 54. High scale modeling – prepare your data well! Effective research cycle – requires good Big Data and ML infra Takeaways
- 55. High scale modeling – prepare your data well! Effective research cycle – requires good Big Data and ML infra Connect research and production - easy A/B testing and model updates Takeaways
- 56. Outbrain Tech Blog - Taking the pain out of Data Science Outbrain tech blog
- 57. Thank You