Bringing HPC Algorithms to Big Data Platforms: Spark Summit East talk by Nikolay Malitsky
3 likes1,797 views
This document discusses bringing high-performance computing (HPC) algorithms to big data platforms. It describes using Spark, an integrated big data platform, for experimental facilities like synchrotrons. A key application discussed is ptychography image reconstruction, which can involve large datasets. The document proposes a Spark-MPI approach to leverage both Spark and MPI for distributed computing. It provides examples of benchmarking a ptychography algorithm on Spark versus MPI and discusses a path towards Spark-MPI applications.
Bringing HPC Algorithms to Big Data Platforms: Spark Summit East talk by Nikolay Malitsky
- 1. Bringing HPC Algorithms to Big Data Platforms Nikolay Malitsky Brookhaven National Laboratory
- 2. Outline Spark as an integrated platform for experimental facilities Ptychographic application Spark-MPI approach Summary
- 3. National Synchrotron Light Source II highly optimized 3rd generation synchrotron facility started operations in 2014 at Brookhaven National Laboratory, New York State suite of six experimental programs: John Hill et al. NSLS-II Strategic Plan, 2015 Hard X-Ray Spectroscopy Imaging & Microscopy Structural Biology Soft X-Ray Scattering & Spectroscopy Complex Scattering Diffraction & In Situ Scattering
- 4. DOE Science Drivers Many years ago … Basic Energy Sciences Fusion Energy Sciences High Energy Physics Nuclear Physics Biological and Environmental Research NSLS-II is here Data-Intensive Science Now
- 5. Closing a gap between Big Data and HPC computing Ecosystems*: Big Data HPC Computing *Geoffrey Fox et al. HPC-ABDC High Performance Computing Enhanced Apache Big Data Stack, CCGrid, 2015 Leaders: Spark MPI New Frontiers
- 6. Three directions Spark + MPI-oriented extension MPI + Spark-oriented extension New model topic of this talk
- 7. Spark an integrated platform for experimental facilities Beamline Applications Data-Information-Knowledge Discovery Path MPI-oriented extension Receiver (s) X-Ray Detector Composite Connector (s) Heterogeneous Data Layer Resilient Distributed Dataset API Big Volume + Variety Big Velocity • Ptychography • Tomography • GISAXS • Deep Learning • …
- 9. Ptychography Ptychography is one of the essential image reconstruction techniques used in light source facilities. This method consists of measuring multiple diffraction patterns by scanning a finite illumination (also called the probe) on an extended specimen (the object). Stefano Marchesini et al. SHARP: a distributed, GPU-based ptychographic solver, J. Appl. Cryst., 49, 2016
- 10. SHARP-NSLS2 application Probe ObjectStefano Marchesini et al. SHARP GPU-based Framework Center for Advanced Mathematics for Energy Research, LBNL Xiaojing Huang, HXN Approach NSLS-II, BNL 512 384x384 detector frames
- 11. Next: near-real-time ptychographic pipeline Tomographic experiment based on 100 ptychographic projections SHARP Multi-GPU engine based on the MPI Allreduce communication model 10 K – 40 K detector frames ptychographic projection Relaxed Averaged Alternating Reflections (RAAR) iteration loop
- 13. Deep Learning Parallel Approaches* Model Parallelism Data Parallelism *Jeffrey Dean et al. Large Scale Distributed Deep Networks, NIPS, 2012 Stochastic Gradient Descent (SGD) iteration loop
- 14. (Some) Spark-Based Distributed Deep Learning Models* SparkNet. AMPLab, UC Berkeley TensorSpark, Arimo CaffeOnSpark, Yahoo Inter-Worker Interface (C++): Ethernet/TCP InfiniBand/RDMA GPU or CPU *Yu Cao and Zhe Dong. Which is Deeper – Comparison of Deep Learning Frameworks, Spark Summit, June 6-8, 2016
- 15. SHARP-SPARK Benchmark Application* *Nikolay Malitsky, Bringing the HPC Reconstruction Algorithms to Big Data Platforms, NYSDS, August 14-17, 2016 SHARP Multi-GPU Framework CAMERA, LBNL CaffeOnSpark RDMA Inter-Worker Interface Big ML Team, Yahoo RDMA Address Exchange Server Approach Time, s MPI Allreduce based on MVAPICH2 0.013 SHARP-SPARK based on the CaffeOnSpark library 0.016 Benchmark results on a cluster with 8 GPU nodes equally divided among the MPI and Spark applications Sum of 2M arrays of floats across the Spark workers SHARP Inter-Worker API
- 16. Message Passing Interface (MPI) Framework MPICH, 1992 - present: Argonne National Laboratory MVAPICH, 2001 - present: Ohio State University OpenMPI, 2003 – present: multiple members Major open-source implementations: MVAPICH2 architecture*: *MVAPICH Team. MVAPICH2 2.2 User Guide, 2016
- 17. From SHARP-SPARK to the MPI Framework SHARP-SPARK MPI Framework Application Programming Interface Communicator interface MPI-3 Standard: point-to-point, collective, etc. Inter-Process Initialization Mechanism RDMA address exchange server Process Manager Interface (PMI-1 and PMI-2) with the support of several internal and external process managers Inter-Process Communication CaffeOnSpark RDMA library Abstract Device Interface (ADI-3) with multiple communication adapters
- 18. Spark-MPI Conceptual Demo MPICH and MVAPICH Common Process Managers Driver Worker Worker Worker Worker PMI Server pm gforker hydra reshell … util … src https://github.com/SciDriver/spark-mpi/tree/master/examples/spark/ Spark driver-worker PMI server-worker MPI inter-worker Interfaces PMI Server variables MPI interface
- 19. Summary: Path towards the Spark-MPI Applications CaffeOnSpark: Spark + RDMA inter-worker interface + complex initialization procedure based on the Spark RDD mechanism SHARP-SPARK: Spark + CaffeOnSpark inter-worker interface + RDMA address exchange server Spark-MPI: Spark + MPI inter-worker interface + PMI Server Kitware and BNL. An in situ, streaming, data- and compute-intensive platform for experimental data. DOE ASCR SBIR Phase I grant. Feb 21, 2017
- 20. Acknowledgement SHARP Team, CAMERA, LBNL: H. Krishnan, S. Marchesini, T. Perciano, J. Sethian, D. Shapiro CaffeOnSpark Team, Yahoo: A. Feng, J. Shi, M. Jain NSLS-II, BNL: M. Cowan, L. Flaks, A. Heroux, X. Huang, L. Li, R. Petkus, T. Smith Computational Science Initiative, BNL: N. D’ Imperio, K. Kleese van Dam, R. D. Zhihua, Funding: National Synchrotron Light Source II, a U.S. Department of Energy (DOE) Office of Science User Facility operated for the DOE Office of Science by Brookhaven National Laboratory under Contract No. DE-SC0012704 Information Technology Division, BNL: R. Perez Scientific Computing, Kitware: A. Chaudhary, P. O’Leary