Meson: Heterogeneous Workflows with Spark at Netflix
Download as PPTX, PDF•3 likes•1,530 views
Netflix uses Meson, an in-house workflow system, to orchestrate and schedule machine learning pipelines using Spark. Meson allows for heterogeneous environments, multi-tenancy, and optimized workflows for machine learning tasks like parameter sweeping with over 30,000 docker containers. It delegates execution to resource managers like Mesos and supports standard and custom steps, parameter passing, structured constructs, and two-way communication between steps.
Meson: Heterogeneous Workflows with Spark at Netflix
- 1. Heterogeneous Workflows with Spark at Netflix 0 Antony Arokiasamy | Kedar Sadekar | Personalization Infrastructure
- 2. 1 Help members find content to watch and enjoy to maximize member satisfaction and retention
- 3. Everything is a Recommendation 2 Recommendations are driven by Machine Learning Ranking Rows
- 4. Machine Learning Pipeline 3 User Selection Feature Generation Model Validation Publish Model Model Training
- 5. Machine Learning Pipeline Challenges 4 • Innovation • Heterogeneous Environments • Spark • Native Support • Separate Orchestration and Execution • Multi Tenancy • Machine Learning Constructs • Parameter Sweep – 30k Dockers
- 6. Meson Workflow System 5 • General Purpose Workflow Orchestration and Scheduling framework • Delegates execution to resource managers like Mesos • Optimized for Machine Learning Pipelines and Visualization • Checkout the Blog • http://bit.ly/mesonws or techblog.netflix.com • Plan to Open Sourced soon
- 8. Standard and Custom Step Types 7
- 9. Parameter Passing 8 Hive Query User DataSet Regional DataSet Global DataSet Get Users Regional Model Global Model User DataSet Wrangle Data
- 11. Top Down or Bottom Up 10
- 13. Spark Step 12
- 14. Artifacts 13 • Step outputs tracked as Artifacts • Visualization • Memoization
- 15. Multi Tenancy 14 • Resource Attributes • spark.cores.max • spark.executor.memory • spark.mesos.constraints • Dynamic Resource Allocation
- 16. Cluster Management 15 • Red-Black software updates • Scale up/Scale down
- 17. Meson/Spark Cluster 16 • 100s of Concurrent Jobs • 700 Nodes • 5000 Cores • 25 TB Memory • Apps: Meson Workflow System, Spark and Dockers • Few smaller clusters
- 18. 17 Antony Arokiasamy Kedar Sadekar @aasamy /aasamy [email protected] @kedar_sadekar /kedar-sadekar [email protected]
Editor's Notes
- #7: We build Meson…