Updates from Project Hydrogen: Unifying State-of-the-Art AI and Big Data in Apache Spark
2 likes492 views
The document discusses Project Hydrogen, a major Spark initiative to unify state-of-the-art AI and big data workloads in Apache Spark. It describes key features of Project Hydrogen including a barrier execution mode for distributed training, accelerator-aware scheduling, and optimized data exchange through improvements to Pandas UDF. These features aim to allow unified training, inference, and analytics on the same Spark cluster.
Updates from Project Hydrogen: Unifying State-of-the-Art AI and Big Data in Apache Spark
- 1. WIFI SSID:Spark+AISummit | Password: UnifiedDataAnalytics
- 2. Xingbo Jiang, Databricks Updates from Project Hydrogen: Unifying State-of-the-Art AI and Big Data in Apache Spark #UnifiedDataAnalytics #SparkAISummit
- 3. About Me • Software Engineer at • Committer of Apache Spark Xingbo Jiang (Github: jiangxb1987)
- 4. 4 Announced last June, Project Hydrogen is a major Spark initiative to unify state-of-the-art AI and big data workloads. About Project Hydrogen Barrier Execution Mode Optimized Data Exchange Accelerator Aware Scheduling
- 5. 5 Why Spark + AI?
- 6. Runtime Delta Spark Core Engine Big Data Processing ETL + SQL +Streaming Machine Learning MLlib + SparkR Apache Spark: The First Unified Analytics Engine 6
- 7. and many more... Internet of ThingsDigital Personalization Huge disruptive innovations are affecting most enterprises on the planet Healthcare and Genomics Fraud Prevention AI is re-shaping the world 7
- 8. Better AI needs more data 8
- 9. The cross... 99 Map/Reduce CaffeOnSpark TensorFlowOnSpark DataFrame-based APIs 50+ Data Sources Python/Java/R interfaces Structured Streaming ML Pipelines API Continuous Processing RDD Project Tungsten Pandas UDF TensorFrames scikit-learn pandas/numpy/scipy LIBLINEAR R glmnet xgboost GraphLab Caffe/PyTorch/MXNet TensorFlow Keras Distributed TensorFlow Horovod tf.data tf.transform AI/ML ?? TF XLA
- 11. Two simple stories 11 data warehouse load fit model data stream load predict model
- 12. Distributed training data warehouse load fit model Required: Be able to read from Delta Lake, Parquet, MySQL, Hive, etc. Answer: Apache Spark Required: distributed GPU cluster for fast training Answer: Horovod, Distributed Tensorflow, etc 12
- 13. Two separate data and AI clusters? load using a Spark cluster fit on a GPU cluster model save data required: glue code 13
- 14. Streaming model inference Kafka load predict model required: ● save to stream sink ● GPU for fast inference 14
- 15. A hybrid Spark and AI cluster? load using a Spark cluster w/ GPUs fit a model distributedly on the same cluster model load using a Spark cluster w/ GPUs predict w/ GPUs as a Spark task model 15
- 16. Unfortunately, it doesn’t work out of the box. See a previous demo.
- 17. 17 Project Hydrogen to fill the major gaps Barrier Execution Mode Optimized Data Exchange Accelerator Aware Scheduling
- 18. 18 Updates from Project Hydrogen ● Available features ● Future Improvement ● How to utilize
- 19. 19 Story #1: Distributed training load using a Spark cluster w/ GPUs fit a model distributedly on the same cluster model
- 20. 20 Project Hydrogen: barrier execution mode Barrier Execution Mode Optimized Data Exchange Accelerator Aware Scheduling
- 21. 21 Different execution models Task 1 Task 2 Task 3 Spark (MapReduce) Tasks are independent of each other Embarrassingly parallel & massively scalable Distributed training Complete coordination among tasks Optimized for communication Task 1 Task 2 Task 3
- 22. 22 Barrier execution mode • All tasks start together • Sufficient info to run a hybrid distributed job • Cancel and restart all tasks on failure JIRA: SPARK-24374 (Spark 2.4)
- 23. 23 API: RDD.barrier() RDD.barrier() tells Spark to launch the tasks together. rdd.barrier().mapPartitions { iter => val context = BarrierTaskContext.get() ... }
- 24. 24 API: context.barrier() context.barrier() places a global barrier and waits until all tasks in this stage hit this barrier. val context = BarrierTaskContext.get() … // preparation context.barrier()
- 25. 25 API: context.getTaskInfos() context.getTaskInfos() returns info about all tasks in this stage. if (context.partitionId == 0) { val addrs = context.getTaskInfos().map(_.address) ... // start a hybrid training job, e.g., via MPI } context.barrier() // wait until training finishes
- 27. 27 Horovod (an LF AI hosted project) ● Little modification to single-node code ● High-performance I/O via MPI and NCCL ● Same convergence theory ● Limitations
- 28. 28 Hydrogen integration with Horovod ● HorovodRunner with Databricks Runtime 5.0 ML has released ● Runs Horovod under barrier execution mode ● Hides details from users def train_hvd(): hvd.init() … # train using Horovod HorovodRunner(np=2).run(train_hvd)
- 29. 29 Implementation of HorovodRunner Integrating Horovod with barrier mode is straightforward: ● Pickle and broadcast the train function. ○ Inspect code and warn users about potential issues. ● Launch a Spark job in barrier execution mode. ● In the first executor, use worker addresses to launch the Horovod MPI job. ● Terminate Horovod if the Spark job got cancelled. ○ Hint: PR_SET_PDEATHSIG Limitation: ● Tailored for Databricks Runtime ML ○ Horovod built with TensorFlow/PyTorch, SSH, OpenMPI, NCCL, etc. ○ Spark 2.4, GPU cluster configuration, etc.
- 30. 30 Project Hydrogen: Accelerator-aware scheduling Barrier Execution Mode Optimized Data Exchange Accelerator Aware Scheduling
- 31. 31 Accelerator-aware scheduling JIRA: SPARK-24615 (ETA: Spark 3.0) Executor 0 GPU:0 GPU:1 Task 0 Task 1 Executor 1 GPU:0 GPU:1 Task 2 Task 3 Task 4 Driver Cluster Manager
- 32. 32 ● Some cluster managers already support accelerators (GPU/FPGA/etc...) ● Spark still need to be aware of accelerators. Example: Why Spark needs accelerator awareness? Executor 0 GPU:0 GPU:1 Task 0 Task 1 Executor 1 GPU:0 GPU:1 Task 2 Task 3 Task 4 ?
- 33. 33 Workarounds (a.k.a hacks) ● Only allow one Spark task on each node ○ Pros: avoid accelerator resources contention ○ Cons: waste resources, poor performance ● Running tasks choose resources collaboratively (e.g. shared locks)
- 34. 34 User Spark Cluster Manager 0. Auto-discover resources. 1. Submit an application with resource requests. 2. Pass resource requests to cluster manager. 4. Register executors. 3. Allocate executors with resource isolation. 5. Submit a Spark job. 6. Schedule tasks on available executors. 7. Dynamic allocation. 8. Retrieve assigned resources and use them in tasks. 9. Monitor and recover failed executors. Proposed workflow
- 35. 35 Discover and request accelerators Admin can specify a script to auto-discover accelerators (SPARK-27024) ● spark.driver.resource.${resourceName}.discoveryScript ● spark.executor.resource.${resourceName}.discoveryScript ● e.g., `nvidia-smi --query-gpu=index ...` User can request accelerators at application level (SPARK-27366) ● spark.executor.resource.${resourceName}.amount ● spark.driver.resource.${resourceName}.amount ● spark.task.resource.${resourceName}.amount
- 36. 36 Retrieve assigned accelerators User can retrieve assigned accelerators from task context (SPARK-27366) context = TaskContext.get() assigned_gpu = context.resources().get(“gpu”).get.addresses.head with tf.device(assigned_gpu): # training code ...
- 37. 37 Cluster manager support YARN SPARK-27361 Kubernetes SPARK-27362 Mesos (not started) SPARK-27363 Standalone SPARK-27360
- 38. 38 Web UI for accelerators
- 39. 39 Support general accelerator types We keep the interfaces general to support other types of accelerators other than GPU in the future, e.g. FPGA ● “GPU” is not a hard-coded resource type. ● spark.executor.resource.{resourceName}.discoveryScript ● context.resources() returns a map from resourceName to ResourceInformation (resource name and addresses).
- 40. 40 Features beyond Project Hydrogen ● Resource request at task level. ● Fine-grained scheduling within one GPU. ● Affinity and anti-affinity. ● ...
- 41. 41 Story #2: Streaming model inference load using a Spark cluster w/ GPUs predict w/ GPUs as a Spark task model
- 42. 42 Project Hydrogen: Optimized data exchange Barrier Execution Mode Optimized Data Exchange Accelerator Aware Scheduling
- 43. 43 Optimized data exchange None of the integrations are possible without exchanging data between Spark and AI frameworks. And performance matters. JIRA: SPARK-24579
- 44. 44 Pandas UDF Pandas UDF was introduced in Spark 2.3, which uses Arrow for data exchange and utilizes Pandas for vectorized computation.
- 45. 45 Pandas UDF for distributed inference Pandas UDF makes it simple to apply a model to a data stream. @pandas_udf(...) def predict(features): ... spark.readStream(...) .withColumn(‘prediction’, predict(col(‘features’)))
- 46. 46 Return StructType from Pandas UDF We improved scalar Pandas UDF to complex return types. So users can return predicted labels and raw scores together. JIRA: SPARK-23836 (Spark 3.0) @pandas_udf(...) def predict(features): # ... return pd.DataFrame({'labels': labels, 'scores': scores})
- 47. 47 Data pipelining CPU GPU t1 fetch batch #1 t2 fetch batch #2 process batch #1 t3 fetch batch #3 process batch #2 t4 process batch #3 CPU GPU t1 fetch batch #1 t2 process batch #1 t3 fetch batch #2 t4 process batch #2 t5 fetch batch #3 t6 process batch #3 (pipelining)
- 48. 48 Pandas UDF prefetch To improve the throughput, we prefetch Arrow record batches in the queue while executing Pandas UDF on the current batch. ● Enabled by default since Databricks Runtime 5.2. ● Up to 2x for I/O and compute balanced workload. ● Observed 1.5x in real workload. JIRA: SPARK-27569 (ETA: Spark 3.0)
- 49. 49 Per-batch initialization overhead A new Pandas UDF interface that load the model only once and reuse it on an iterator of batches. JIRA: SPARK-26412 (Spark 3.0) @pandas_udf(...) def predict(batches): model = … # load model once for batch in batches: yield model.predict(batch)
- 50. 50 Acknowledgement ● Many ideas in Project Hydrogen are based on previous community work: TensorFrames, BigDL, Apache Arrow, Pandas UDF, Spark GPU support, MPI, etc. ● We would like to thank many Spark committers and contributors who helped the project proposal, design, and implementation.
- 51. 51 Acknowledgement ● Alex Sergeev ● Andy Feng ● Bryan Cutler ● Felix Cheung ● Hyukjin Kwon ● Imran Rashid ● Jason Lowe ● Jerry Shao ● Li Jin ● Madhukar Korupolu ● Mark Hamstra ● Robert Evans ● Sean Owen ● Shane Knapp ● Takuya Ueshin ● Thomas Graves ● Wenchen Fan ● Xiangrui Meng ● Xiao Li ● Yi Wu ● Yinan Li ● Yu Jiang ● … and many more!
- 52. DON’T FORGET TO RATE AND REVIEW THE SESSIONS SEARCH SPARK + AI SUMMIT