Spark Summit EU talk by Luc Bourlier
3 likes•1,748 views
This document discusses dynamic resource allocation in Spark clusters. It explains how Spark can add or remove executors from a cluster based on workload to optimize resource usage for jobs with variable loads. It also describes how the external shuffle service moves shuffle data management out of executors to improve performance and fault tolerance. The document provides details on configuring dynamic allocation and the external shuffle service and demonstrates dynamic allocation in action. It also discusses applying these techniques to Spark Streaming workloads.
Spark Summit EU talk by Luc Bourlier
- 1. DYNAMIC RESOURCE ALLOCATION, DO MORE WITH YOUR CLUSTER Luc Bourlier Lightbend
- 2. ● Dynamic Resource Allocation ● ^^ in Spark ● External Shuffle Service ● Configuration ● Demo ● Spark Streaming
- 4. Dynamic Resource Allocation Cluster I’d like some resources for a job
- 5. Dynamic Resource Allocation Cluster Oki. Thanks
- 9. Dynamic Resource Allocation Cluster Hmm, actually, I don’t need all this power anymore.
- 11. Why? ● Shared cluster ● Optimization of resource usage When? ● variable load job Dynamic Resource Allocation Cluster
- 14. Spark Dynamic Allocation Cluster Manager Worker Node Executor Worker Node Worker Node Executor Driver Scheduler(s) need 2 executors
- 15. tasks are waiting too long Spark Dynamic Allocation Cluster Manager Worker Node Executor Worker Node Worker Node Executor Driver Scheduler(s) need 1 more executor Executor
- 16. executor has been idle for a while Spark Dynamic Allocation Cluster Manager Worker Node Executor Worker Node Worker Node Executor Driver Scheduler(s) Executor terminate the executor
- 18. External Shuffle Service ● Did we lose any data?
- 19. External Shuffle Service Shuffle write Shuffle fetch Map Task Reduce Task Aggregator bucket bucket bucket Aggregator Map Task Reduce Task Aggregator bucket bucket bucket Aggregator Map Task Reduce Task Aggregator bucket bucket bucket Aggregator
- 20. External Shuffle Service Shuffle write Shuffle fetch Map Task Reduce Task Aggregator bucket bucket bucket Aggregator Map Task Reduce Task Aggregator bucket bucket bucket Aggregator Map Task Reduce Task Aggregator bucket bucket bucket Aggregator
- 21. External Shuffle Service Shuffle write Shuffle fetch Map Task Reduce Task Aggregator bucket bucket bucket Aggregator Map Task Reduce Task Aggregator bucket bucket bucket Aggregator Map Task Reduce Task Aggregator bucket bucket bucket Aggregator
- 22. External Shuffle Service ● Extracted from executor ● Manage the local aggregated data for the shuffle operations ● Maintain the data until the application is done.
- 23. Configuration
- 24. Configuration ● Dynamic Allocation ○ spark.dynamicAllocation.enabled ○ spark.dynamicAllocation.initialExecutors ○ spark.dynamicAllocation.maxExecutors ○ spark.dynamicAllocation.minExecutors
- 25. Configuration ● Dynamic Allocation ○ spark.dynamicAllocation.schedulerBacklogTimeout ○ spark.dynamicAllocation.executorIdleTimeout ○ spark.dynamicAllocation.sustainedSchedulerBacklogTimeout
- 26. Configuration ● External Shuffle Service ○ spark.shuffle.service.enabled ○ spark.shuffle.service.port
- 27. Demo Dynamic Allocation in Action
- 29. Configuration Values? It depends …. No, seriously
- 30. Configuration Values ● spark.dynamicAllocation.initialExecutors ● spark.dynamicAllocation.maxExecutors ● spark.dynamicAllocation.minExecutors Depends on workload and how many resources are potentially available to you.
- 31. Configuration Values ● spark.dynamicAllocation.schedulerBacklogTimeout Too short, might trigger for short burst of tasks. Too long, might be less effective. ● spark.dynamicAllocation.sustainedSchedulerBacklogTimeout Executor start duration. Default set to schedulerBacklogTimeout.
- 32. Configuration Values ● spark.dynamicAllocation.executorIdleTimeout Relative to the duration of the longer task. No big drawback on being too long, except cost.
- 33. Spark Streaming
- 35. • In most case, schedulerBacklogTimeout longer than batch interval. • executorIdleTimeout a portion of batch interval. • Should allow to manage processing delay. • Not compatible with the dynamic rate estimator. Spark Streaming
- 36. More Dynamic?
- 37. More Dynamic? https://github.com/twosigma/Cook ‘Fair’ job scheduler for Spark on top of Mesos ● Not a recommendation, just a suggestion. ● Some assembly required.
- 39. External Shuffle Service Cluster Manager Worker Node Executor Worker Node Worker Node Driver Scheduler(s) Executor External Shuffle Service External Shuffle Service External Shuffle Service