Supporting Highly Multitenant Spark Notebook Workloads with Craig Ingram and Brad Kaiser
0 likes1,106 views
The document discusses the challenges and best practices for hosting multitenant Spark notebook workloads, particularly focusing on improvements made at IBM. Key strategies include utilizing kernel remoting, efficient resource scheduling, and establishing smaller, isolated clusters to enhance performance and security. Additionally, the document emphasizes the importance of proactive resource management to prevent issues like starvation and improve overall analytics capabilities.
Supporting Highly Multitenant Spark Notebook Workloads with Craig Ingram and Brad Kaiser
- 1. Brad Kaiser, IBM/TWC Craig Ingram, IBM/TWC Supporting Highly Multitenant Spark Notebook Workloads #EUdev8
- 2. Hosting Multitenant Spark Notebooks Is Hard But It Doesn't Have To Be • Our Journey • Best Practices • New Work 2#EUdev8
- 5. IBM's Commitment to Open Source 5#EUdev8 • Contribute intellectual and technical capital to the Apache Spark community. • Make the core technology enterprise- and cloud-ready. • Build data science skills to drive intelligence into business applications — https://cognitiveclass.ai/ • http://spark.tc
- 6. Mission • Provide: – a secure, performant, stable cluster. – interactive analytics, visualizations, and reports. – collaboration and sharing with other data scientists, engineers, and consumers. – job scheduling capabilities. – a quick and easy way to get started with Spark. 6#EUdev8
- 7. Goals • Support hundreds of analysts/data scientists using Spark – Quick kernel creation (<10s from notebook creation to available sc). – Utilize cluster resources efficiently. – Elastically scale based on load. 7#EUdev8
- 8. Lessons Learned at TWC 8#EUdev8
- 9. One Big Cluster 9#EUdev8 • Spark collocated with Cassandra • Fast • Stable
- 10. One Big Cluster - Cons 10#EUdev8 • Outside analysts start using our cluster • Provided notebook services • Interrupted our perfectly scheduled jobs • Used a lot of resources causing Cassandra to crash
- 11. Add Smaller Clusters 11#EUdev8 • We built some smaller clusters • Still platform agnostic • Other teams couldn't affect our production cluster
- 12. Add Smaller Clusters - Cons 12#EUdev8 • Hassle to set up • Required a lot of maintenance • Sat idle
- 13. EMR • Analysts make ad hoc clusters for their needs • No maintenance from us • Learning curve for analysts • They tend to leave them running 13#EUdev8
- 14. Lessons Learned - IBM 14#EUdev8
- 15. Data Science Experience • Collaborative environment on the front end – Collaboration Tools – Shared Data Sets – Flows – GitHub Integration • Multiple compute environments on the back end – DSX on the cloud: compute runs on IBM cloud – DSX Local: compute runs on private cloud or Z 15#EUdev8
- 16. Lessons Learned - IBM • You need kernel remoting – Allows advanced collaborative tools in the application tier – Allows resource consolidation in the analytics tier • Resource consolidation puts stress on the analytics tier – Starvation – Management of cached data – Performance bottlenecks (example: Spark web UI) 16#EUdev8
- 18. Best Practices • Use kernel remoting • Use fewer, bigger clusters • Know your workloads • Isolate users • Schedule resources efficiently 18#EUdev8
- 19. Use Kernel Remoting • Running all of your notebook kernels on the same server is a bottle neck • Run your kernels distributed on the cluster • You can run a lot more notebooks 19#EUdev8
- 20. Jupyter Enterprise Gateway • New Open Source project from IBM • Goals: – Allow hundreds of notebook users to share a single Spark cluster… – …with enterprise-level security and performance. • Used in IBM Analytics Engine (GA) • developer.ibm.com/code/openprojects/jupyter- enterprise-gateway-2 20#EUdev8
- 21. Jupyter Enterprise Gateway: How it Works 21#EUdev8 Spark Cluster Security Layer Jupyter Enterprise Gateway • Multitenant • Remote Kernel Lifecycle Management YARN Workers Impersonation: Alice’s kernel runs under Alice’s user ID. Spark ExecutorsSpark Executors Spark Executors Yarn Container Jupyter Kernel Spark Driver Spark ExecutorsSpark Executors Spark Executors Yarn Container Jupyter Kernel Spark Driver Secure Secure Secure Secure
- 22. Use Fewer, Bigger Clusters • Better resource utilization through statistical multiplexing • Improved security and auditing due to centralization – Hive, Ranger, and Atlas are common in the ecosystem. – Many new, platform specific solutions to address this problem. • Easier collaboration between users – Shared notebooks with interactive visualizations and markdown support. – GitHub integration for versioning and external sharing. – Catalog based data discovery and sharing. – Governance and auditing support. 22#EUdev8
- 23. Know your workloads • What is the main resource they use? • Overprovision the hardware • When to scale up and down? – CPU load – YARN/RM Queue Stats (depth, waiting jobs, available CPU/mem, preemptions) • If containers are getting preempted, it’s due to queues filling up. 23#EUdev8
- 24. Isolate Users • Don't have a generic user account • Catalog and Governance – Hive – Atlas – Ranger • You don't want users embedding keys and passwords in their notebooks. 24#EUdev8
- 26. YARN Queues • Take advantage of YARN’s hierarchical queue system to manage and organize resources. – Over-allocate queues for better resource utilization and sharing. – Take advantage of node labels for users that have priority jobs that require an SLA. – Intra-queue preemption and asynchronous container allocation should be available in YARN 3.0. 26#EUdev8
- 28. Dynamic Allocation • Dynamic allocation lets you take advantage of varying activity – Proactively scales the number of executors based on the scheduler's backlog. – Removes idle executors after a timeout. – Be sure to set a sensible number of initial executors and minimum executor floor and let them ramp up on demand. • Static allocation best for known workloads 28#EUdev8
- 30. Improvements to Spark • Alleviate the tradeoffs inherent in current best practices. – Recover cached data when shutting down idle executors – Proactively shut down executors to prevent starvation 30#EUdev8
- 31. Recovering Cached Data • Replicates cached data to remaining executors before shutting them down. • Ameliorates cache issues with dynamic allocation • Useful in shared spark notebook environments 31#EUdev8
- 34. Check out my PR – SPARK-21097 – github.com/apache/spark/pull/19041 34#EUdev8
- 35. Preventing Starvation • Eliminate issues where users are unable to run anything due to other users taking up all of the cluster’s resources. • Especially useful in shared spark notebook environments where idle resources can be reclaimed easily. • Preemption can solve this. 35#EUdev8
- 36. Enter Preemption • Requests containers associated with over- allocated queues to shut down. • Handle YARN's PreemptionMessage in a way that best suits the workload. • Pick the right executors to terminate. 36#EUdev8
- 37. Keep up with the JIRA • SPARK-21122 • PR coming soon 37#EUdev8
- 38. Call to action • Look at our JIRAs • Try out our PRs 38#EUdev8
- 39. Shout-outs!!! • Our notebook workload simulator, benchmark, and tracing tools. – spark-bench - github.com/SparkTC/spark-bench • Check out Emily Curtin’s talk tomorrow about spark-bench. – spark-tracing - github.com/SparkTC/spark-tracing • Matthew Schauer's baby awaiting open-source approval. • Special thanks to Vijay Bommireddipalli and Fred Reiss for their guidance and support! 39#EUdev8
- 40. Contact Info Brad Kaiser [email protected] Craig Ingram [email protected] 40#EUdev8
- 42. YARN Asynchronous Scheduling • Enable asynchronous scheduling of containers in YARN. – yarn.scheduler.capacity.schedule-asynchronously.enable – YARN-7327 and YARN-5139 42#EUdev8
- 43. References • Some Icons provided by Icons8 43#EUdev8
- 44. Quick Settings • Use spark.yarn.jars or spark.yarn.archive. • Running Spark from tmpfs did not improve performance. • Support for multiple/new versions of Spark. 44#EUdev8 Move to backup
- 45. Disable unused credential providers • spark.yarn.security.credentials.hive.enabled • spark.yarn.security.credentials.hbase.enabled 45#EUdev8 conf avg stdev default 8.745 0.051 nohive 8.7 0.05 nohbase 7.87 0.05 Move to backup
- 46. Problem Domain • Security – UI and service protection – Data governance and auditing • Stability • Performance 46#EUdev8
- 47. What’s next… • spark-on-k8s • Scheduler improvements • Executor startup time reduction 47#EUdev8
- 48. • Hundreds of notebook users leads to a highly multitenant and interactive workload. • Challenge: Give each user the illusion of having a large cluster all to herself 48#EUdev8 Many users at once Bursty offered load Latency is important
- 49. What platforms are out there… Hosted Solutions • Data Science Experience (DSX)/Watson Data Platform (WDP) • databricks • Google Cloud Platform • Microsoft Azure HDInsight • and others!!! 49#EUdev8
- 50. What platforms are out there… Self-Hosted Solutions • HDP – Hadoop Data Platform • CDH – Cloudera Distribution Including Hadoop • MapR • Mesosphere 50#EUdev8
- 51. Reasons to Self Host at TWC • Cloud Agnostic • Flexibility • Sensitive Data • Fewer Options in 2014 • Cassandra Colocation • Cost…maybe? 51#EUdev8
- 52. Potential Downfalls • In a self-hosted environment, everything is up to you. – Security – Stability and Performance – Scalability • Compute • Storage – Monitoring – Alerting – Logging 52#EUdev8