Designing Scalable HPC, Deep Learning and Cloud Middleware for Exascale Systems
1 like1,325 views
The document discusses the design and implementation of scalable high-performance computing (HPC), deep learning, and cloud middleware for exascale systems, focusing on programming models, challenges, and the MVAPICH2 project. It highlights the convergence of HPC, big data, and deep learning applications, driving the need for effective middleware and communication protocols. Key features, performance benchmarks, and enhancements of MVAPICH2, an open-source MPI library, are also detailed, emphasizing its widespread use in top supercomputing systems.
Designing Scalable HPC, Deep Learning and Cloud Middleware for Exascale Systems
- 1. Designing Scalable HPC, Deep Learning, and Cloud Middleware for Exascale Systems Dhabaleswar K. (DK) Panda The Ohio State University E-mail: [email protected] http://www.cse.ohio-state.edu/~panda Talk at Swiss Conference (April ‘18) by
- 2. HPCAC-Switzerland (April ’18) 2Network Based Computing Laboratory High-End Computing (HEC): Towards Exascale Expected to have an ExaFlop system in 2019-2021! 100 PFlops in 2016 1 EFlops in 2019- 2021?
- 3. HPCAC-Switzerland (April ’18) 3Network Based Computing Laboratory Big Data (Hadoop, Spark, HBase, Memcached, etc.) Deep Learning (Caffe, TensorFlow, BigDL, etc.) HPC (MPI, RDMA, Lustre, etc.) Increasing Usage of HPC, Big Data and Deep Learning Convergence of HPC, Big Data, and Deep Learning! Increasing Need to Run these applications on the Cloud!!
- 4. HPCAC-Switzerland (April ’18) 4Network Based Computing Laboratory • Traditional HPC – Message Passing Interface (MPI), including MPI + OpenMP – Support for PGAS and MPI + PGAS (OpenSHMEM, UPC) – Exploiting Accelerators • Deep Learning – Acceleration of Caffe, CNTK, TensorFlow, and many more – Out-of-core Processing • Cloud for HPC and BigData – Virtualization with SR-IOV and Containers HPC, Deep Learning, and Cloud
- 5. HPCAC-Switzerland (April ’18) 5Network Based Computing Laboratory Parallel Programming Models Overview P1 P2 P3 Shared Memory P1 P2 P3 Memory Memory Memory P1 P2 P3 Memory Memory Memory Logical shared memory Shared Memory Model SHMEM, DSM Distributed Memory Model MPI (Message Passing Interface) Partitioned Global Address Space (PGAS) Global Arrays, UPC, Chapel, X10, CAF, … • Programming models provide abstract machine models • Models can be mapped on different types of systems – e.g. Distributed Shared Memory (DSM), MPI within a node, etc. • PGAS models and Hybrid MPI+PGAS models are gradually receiving importance
- 6. HPCAC-Switzerland (April ’18) 6Network Based Computing Laboratory Partitioned Global Address Space (PGAS) Models • Key features - Simple shared memory abstractions - Light weight one-sided communication - Easier to express irregular communication • Different approaches to PGAS - Languages • Unified Parallel C (UPC) • Co-Array Fortran (CAF) • X10 • Chapel - Libraries • OpenSHMEM • UPC++ • Global Arrays
- 7. HPCAC-Switzerland (April ’18) 7Network Based Computing Laboratory Hybrid (MPI+PGAS) Programming • Application sub-kernels can be re-written in MPI/PGAS based on communication characteristics • Benefits: – Best of Distributed Computing Model – Best of Shared Memory Computing Model Kernel 1 MPI Kernel 2 MPI Kernel 3 MPI Kernel N MPI HPC Application Kernel 2 PGAS Kernel N PGAS
- 8. HPCAC-Switzerland (April ’18) 8Network Based Computing Laboratory Supporting Programming Models for Multi-Petaflop and Exaflop Systems: Challenges Programming Models MPI, PGAS (UPC, Global Arrays, OpenSHMEM), CUDA, OpenMP, OpenACC, Cilk, Hadoop (MapReduce), Spark (RDD, DAG), etc. Application Kernels/Applications Networking Technologies (InfiniBand, 40/100GigE, Aries, and Omni-Path) Multi-/Many-core Architectures Accelerators (GPU and FPGA) Middleware Co-Design Opportunities and Challenges across Various Layers Performance Scalability Resilience Communication Library or Runtime for Programming Models Point-to-point Communication Collective Communication Energy- Awareness Synchronization and Locks I/O and File Systems Fault Tolerance
- 9. HPCAC-Switzerland (April ’18) 9Network Based Computing Laboratory • Scalability for million to billion processors – Support for highly-efficient inter-node and intra-node communication (both two-sided and one-sided) – Scalable job start-up • Scalable Collective communication – Offload – Non-blocking – Topology-aware • Balancing intra-node and inter-node communication for next generation nodes (128-1024 cores) – Multiple end-points per node • Support for efficient multi-threading • Integrated Support for GPGPUs and Accelerators • Fault-tolerance/resiliency • QoS support for communication and I/O • Support for Hybrid MPI+PGAS programming (MPI + OpenMP, MPI + UPC, MPI + OpenSHMEM, MPI+UPC++, CAF, …) • Virtualization • Energy-Awareness Broad Challenges in Designing Communication Middleware for (MPI+X) at Exascale
- 10. HPCAC-Switzerland (April ’18) 10Network Based Computing Laboratory • Extreme Low Memory Footprint – Memory per core continues to decrease • D-L-A Framework – Discover • Overall network topology (fat-tree, 3D, …), Network topology for processes for a given job • Node architecture, Health of network and node – Learn • Impact on performance and scalability • Potential for failure – Adapt • Internal protocols and algorithms • Process mapping • Fault-tolerance solutions – Low overhead techniques while delivering performance, scalability and fault-tolerance Additional Challenges for Designing Exascale Software Libraries
- 11. HPCAC-Switzerland (April ’18) 11Network Based Computing Laboratory Overview of the MVAPICH2 Project • High Performance open-source MPI Library for InfiniBand, Omni-Path, Ethernet/iWARP, and RDMA over Converged Ethernet (RoCE) – MVAPICH (MPI-1), MVAPICH2 (MPI-2.2 and MPI-3.1), Started in 2001, First version available in 2002 – MVAPICH2-X (MPI + PGAS), Available since 2011 – Support for GPGPUs (MVAPICH2-GDR) and MIC (MVAPICH2-MIC), Available since 2014 – Support for Virtualization (MVAPICH2-Virt), Available since 2015 – Support for Energy-Awareness (MVAPICH2-EA), Available since 2015 – Support for InfiniBand Network Analysis and Monitoring (OSU INAM) since 2015 – Used by more than 2,875 organizations in 86 countries – More than 462,000 (> 0.46 million) downloads from the OSU site directly – Empowering many TOP500 clusters (Nov ‘17 ranking) • 1st, 10,649,600-core (Sunway TaihuLight) at National Supercomputing Center in Wuxi, China • 9th, 556,104 cores (Oakforest-PACS) in Japan • 12th, 368,928-core (Stampede2) at TACC • 17th, 241,108-core (Pleiades) at NASA • 48th, 76,032-core (Tsubame 2.5) at Tokyo Institute of Technology – Available with software stacks of many vendors and Linux Distros (RedHat and SuSE) – http://mvapich.cse.ohio-state.edu • Empowering Top500 systems for over a decade
- 12. HPCAC-Switzerland (April ’18) 12Network Based Computing Laboratory 0 50000 100000 150000 200000 250000 300000 350000 400000 450000 500000 Sep-04 Feb-05 Jul-05 Dec-05 May-06 Oct-06 Mar-07 Aug-07 Jan-08 Jun-08 Nov-08 Apr-09 Sep-09 Feb-10 Jul-10 Dec-10 May-11 Oct-11 Mar-12 Aug-12 Jan-13 Jun-13 Nov-13 Apr-14 Sep-14 Feb-15 Jul-15 Dec-15 May-16 Oct-16 Mar-17 Aug-17 Jan-18 NumberofDownloads Timeline MV0.9.4 MV20.9.0 MV20.9.8 MV21.0 MV1.0 MV21.0.3 MV1.1 MV21.4 MV21.5 MV21.6 MV21.7 MV21.8 MV21.9 MV2-GDR2.0b MV2-MIC2.0 MV2-GDR2.3a MV2-X2.3b MV2Virt2.2 MV22.3rc1 OSUINAM0.9.3 MVAPICH2 Release Timeline and Downloads
- 13. HPCAC-Switzerland (April ’18) 13Network Based Computing Laboratory Architecture of MVAPICH2 Software Family High Performance Parallel Programming Models Message Passing Interface (MPI) PGAS (UPC, OpenSHMEM, CAF, UPC++) Hybrid --- MPI + X (MPI + PGAS + OpenMP/Cilk) High Performance and Scalable Communication Runtime Diverse APIs and Mechanisms Point-to- point Primitives Collectives Algorithms Energy- Awareness Remote Memory Access I/O and File Systems Fault Tolerance Virtualization Active Messages Job Startup Introspection & Analysis Support for Modern Networking Technology (InfiniBand, iWARP, RoCE, Omni-Path) Support for Modern Multi-/Many-core Architectures (Intel-Xeon, OpenPower, Xeon-Phi, ARM, NVIDIA GPGPU) Transport Protocols Modern Features RC XRC UD DC UMR ODP SR- IOV Multi Rail Transport Mechanisms Shared Memory CMA IVSHMEM Modern Features MCDRAM* NVLink* CAPI* * Upcoming XPMEM*
- 14. HPCAC-Switzerland (April ’18) 14Network Based Computing Laboratory • Scalability for million to billion processors – Support for highly-efficient inter-node and intra-node communication – Scalable Start-up – Optimized Collectives using SHArP and Multi-Leaders – Optimized CMA-based Collectives – Upcoming Optimized XPMEM-based Collectives • Integrated Support for GPGPUs • Optimized MVAPICH2 for OpenPower (with/ NVLink) and ARM • Application Scalability and Best Practices Overview of A Few Challenges being Addressed by the MVAPICH2 Project for Exascale
- 15. HPCAC-Switzerland (April ’18) 15Network Based Computing Laboratory One-way Latency: MPI over IB with MVAPICH2 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 Small Message Latency Message Size (bytes) Latency(us) 1.11 1.19 0.98 1.15 1.04 TrueScale-QDR - 3.1 GHz Deca-core (Haswell) Intel PCI Gen3 with IB switch ConnectX-3-FDR - 2.8 GHz Deca-core (IvyBridge) Intel PCI Gen3 with IB switch ConnectIB-Dual FDR - 3.1 GHz Deca-core (Haswell) Intel PCI Gen3 with IB switch ConnectX-5-EDR - 3.1 GHz Deca-core (Haswell) Intel PCI Gen3 with IB Switch Omni-Path - 3.1 GHz Deca-core (Haswell) Intel PCI Gen3 with Omni-Path switch 0 20 40 60 80 100 120 TrueScale-QDR ConnectX-3-FDR ConnectIB-DualFDR ConnectX-5-EDR Omni-Path Large Message Latency Message Size (bytes) Latency(us)
- 16. HPCAC-Switzerland (April ’18) 16Network Based Computing Laboratory TrueScale-QDR - 3.1 GHz Deca-core (Haswell) Intel PCI Gen3 with IB switch ConnectX-3-FDR - 2.8 GHz Deca-core (IvyBridge) Intel PCI Gen3 with IB switch ConnectIB-Dual FDR - 3.1 GHz Deca-core (Haswell) Intel PCI Gen3 with IB switch ConnectX-5-EDR - 3.1 GHz Deca-core (Haswell) Intel PCI Gen3 IB switch Omni-Path - 3.1 GHz Deca-core (Haswell) Intel PCI Gen3 with Omni-Path switch Bandwidth: MPI over IB with MVAPICH2 0 5000 10000 15000 20000 25000 TrueScale-QDR ConnectX-3-FDR ConnectIB-DualFDR ConnectX-5-EDR Omni-Path Bidirectional Bandwidth Bandwidth (MBytes/sec) Message Size (bytes) 22,564 12,161 21,983 6,228 24,136 0 2000 4000 6000 8000 10000 12000 14000 Unidirectional Bandwidth Bandwidth (MBytes/sec) Message Size (bytes) 12,590 3,373 6,356 12,358 12,366
- 17. HPCAC-Switzerland (April ’18) 17Network Based Computing Laboratory Startup Performance on KNL + Omni-Path 0 5 10 15 20 25 64 128 256 512 1K 2K 4K 8K 16K 32K 64K TimeTaken(Seconds) Number of Processes MPI_Init & Hello World - Oakforest-PACS Hello World (MVAPICH2-2.3a) MPI_Init (MVAPICH2-2.3a) • MPI_Init takes 22 seconds on 229,376 processes on 3,584 KNL nodes (Stampede2 – Full scale) • 8.8 times faster than Intel MPI at 128K processes (Courtesy: TACC) • At 64K processes, MPI_Init and Hello World takes 5.8s and 21s respectively (Oakforest-PACS) • All numbers reported with 64 processes per node 5.8s 21s 22s New designs available since MVAPICH2-2.3a and as patch for SLURM 15, 16, and 17
- 18. HPCAC-Switzerland (April ’18) 18Network Based Computing Laboratory 0 0.05 0.1 0.15 0.2 (4,28) (8,28) (16,28) Latency(seconds) (Number of Nodes, PPN) MVAPICH2 MVAPICH2-SHArP 13% Mesh Refinement Time of MiniAMR Advanced Allreduce Collective Designs Using SHArP 12% 0 0.1 0.2 0.3 0.4 (4,28) (8,28) (16,28) Latency(seconds) (Number of Nodes, PPN) MVAPICH2 MVAPICH2-SHArP Avg DDOT Allreduce time of HPCG SHArP Support is available since MVAPICH2 2.3a M. Bayatpour, S. Chakraborty, H. Subramoni, X. Lu, and D. K. Panda, Scalable Reduction Collectives with Data Partitioning-based Multi- Leader Design, SuperComputing '17.
- 19. HPCAC-Switzerland (April ’18) 19Network Based Computing Laboratory Performance of MPI_Allreduce On Stampede2 (10,240 Processes) 0 50 100 150 200 250 300 4 8 16 32 64 128 256 512 1024 2048 4096 Latency(us) Message Size MVAPICH2 MVAPICH2-OPT IMPI 0 200 400 600 800 1000 1200 1400 1600 1800 2000 8K 16K 32K 64K 128K 256K Message Size MVAPICH2 MVAPICH2-OPT IMPI OSU Micro Benchmark 64 PPN 2.4X • For MPI_Allreduce latency with 32K bytes, MVAPICH2-OPT can reduce the latency by 2.4X M. Bayatpour, S. Chakraborty, H. Subramoni, X. Lu, and D. K. Panda, Scalable Reduction Collectives with Data Partitioning-based Multi-Leader Design, SuperComputing '17. Available in MVAPICH2-X 2.3b
- 20. HPCAC-Switzerland (April ’18) 20Network Based Computing Laboratory Optimized CMA-based Collectives for Large Messages 1 10 100 1000 10000 100000 1000000 1K 2K 4K 8K 16K 32K 64K 128K 256K 512K 1M 2M 4M Message Size KNL (2 Nodes, 128 Procs) MVAPICH2-2.3a Intel MPI 2017 OpenMPI 2.1.0 Tuned CMA Latency(us) 1 10 100 1000 10000 100000 1000000 1K 2K 4K 8K 16K 32K 64K 128K 256K 512K 1M 2M Message Size KNL (4 Nodes, 256 Procs) MVAPICH2-2.3a Intel MPI 2017 OpenMPI 2.1.0 Tuned CMA 1 10 100 1000 10000 100000 1000000 1K 2K 4K 8K 16K 32K 64K 128K 256K 512K 1M Message Size KNL (8 Nodes, 512 Procs) MVAPICH2-2.3a Intel MPI 2017 OpenMPI 2.1.0 Tuned CMA • Significant improvement over existing implementation for Scatter/Gather with 1MB messages (up to 4x on KNL, 2x on Broadwell, 14x on OpenPower) • New two-level algorithms for better scalability • Improved performance for other collectives (Bcast, Allgather, and Alltoall) ~ 2.5x Better ~ 3.2x Better ~ 4x Better ~ 17x Better S. Chakraborty, H. Subramoni, and D. K. Panda, Contention Aware Kernel-Assisted MPI Collectives for Multi/Many-core Systems, IEEE Cluster ’17, BEST Paper Finalist Performance of MPI_Gather on KNL nodes (64PPN) Available in MVAPICH2-X 2.3b
- 21. HPCAC-Switzerland (April ’18) 21Network Based Computing Laboratory Shared Address Space (XPMEM)-based Collectives Design 1 10 100 1000 10000 100000 16K 32K 64K 128K 256K 512K 1M 2M 4M Latency(us) Message Size MVAPICH2-2.3b IMPI-2017v1.132 MVAPICH2-Opt OSU_Allreduce (Broadwell 256 procs) • “Shared Address Space”-based true zero-copy Reduction collective designs in MVAPICH2 • Offloaded computation/communication to peers ranks in reduction collective operation • Up to 4X improvement for 4MB Reduce and up to 1.8X improvement for 4M AllReduce 73.2 1.8X 1 10 100 1000 10000 100000 16K 32K 64K 128K 256K 512K 1M 2M 4M Message Size MVAPICH2-2.3b IMPI-2017v1.132 MVAPICH2-Opt OSU_Reduce (Broadwell 256 procs) 4X 36.1 37.9 16.8 J. Hashmi, S. Chakraborty, M. Bayatpour, H. Subramoni, and D. Panda, Designing Efficient Shared Address Space Reduction Collectives for Multi-/Many-cores, International Parallel & Distributed Processing Symposium (IPDPS '18), May 2018. Will be available in future
- 22. HPCAC-Switzerland (April ’18) 22Network Based Computing Laboratory • Scalability for million to billion processors • Integrated Support for GPGPUs – CUDA-aware MPI – GPUDirect RDMA (GDR) Support • Optimized MVAPICH2 for OpenPower (with/ NVLink) and ARM • Application Scalability and Best Practices Overview of A Few Challenges being Addressed by the MVAPICH2 Project for Exascale
- 23. HPCAC-Switzerland (April ’18) 23Network Based Computing Laboratory At Sender: At Receiver: MPI_Recv(r_devbuf, size, …); inside MVAPICH2 • Standard MPI interfaces used for unified data movement • Takes advantage of Unified Virtual Addressing (>= CUDA 4.0) • Overlaps data movement from GPU with RDMA transfers High Performance and High Productivity MPI_Send(s_devbuf, size, …); GPU-Aware (CUDA-Aware) MPI Library: MVAPICH2-GPU
- 24. HPCAC-Switzerland (April ’18) 24Network Based Computing Laboratory CUDA-Aware MPI: MVAPICH2-GDR 1.8-2.3 Releases • Support for MPI communication from NVIDIA GPU device memory • High performance RDMA-based inter-node point-to-point communication (GPU-GPU, GPU-Host and Host-GPU) • High performance intra-node point-to-point communication for multi-GPU adapters/node (GPU-GPU, GPU-Host and Host-GPU) • Taking advantage of CUDA IPC (available since CUDA 4.1) in intra-node communication for multiple GPU adapters/node • Optimized and tuned collectives for GPU device buffers • MPI datatype support for point-to-point and collective communication from GPU device buffers • Unified memory
- 25. HPCAC-Switzerland (April ’18) 25Network Based Computing Laboratory 0 2000 4000 6000 1 2 4 8 16 32 64 128 256 512 1K 2K 4K Bandwidth(MB/s) Message Size (Bytes) GPU-GPU Inter-node Bi-Bandwidth MV2-(NO-GDR) MV2-GDR-2.3a 0 1000 2000 3000 4000 1 2 4 8 16 32 64 128 256 512 1K 2K 4K Bandwidth(MB/s) Message Size (Bytes) GPU-GPU Inter-node Bandwidth MV2-(NO-GDR) MV2-GDR-2.3a 0 10 20 30 0 1 2 4 8 16 32 64 128 256 512 1K 2K 4K 8K Latency(us) Message Size (Bytes) GPU-GPU Inter-node Latency MV2-(NO-GDR) MV2-GDR-2.3a MVAPICH2-GDR-2.3a Intel Haswell (E5-2687W @ 3.10 GHz) node - 20 cores NVIDIA Volta V100 GPU Mellanox Connect-X4 EDR HCA CUDA 9.0 Mellanox OFED 4.0 with GPU-Direct-RDMA 10x 9x Optimized MVAPICH2-GDR Design 1.88us 11X
- 26. HPCAC-Switzerland (April ’18) 26Network Based Computing Laboratory • Platform: Wilkes (Intel Ivy Bridge + NVIDIA Tesla K20c + Mellanox Connect-IB) • HoomdBlue Version 1.0.5 • GDRCOPY enabled: MV2_USE_CUDA=1 MV2_IBA_HCA=mlx5_0 MV2_IBA_EAGER_THRESHOLD=32768 MV2_VBUF_TOTAL_SIZE=32768 MV2_USE_GPUDIRECT_LOOPBACK_LIMIT=32768 MV2_USE_GPUDIRECT_GDRCOPY=1 MV2_USE_GPUDIRECT_GDRCOPY_LIMIT=16384 Application-Level Evaluation (HOOMD-blue) 0 500 1000 1500 2000 2500 4 8 16 32 AverageTimeStepsper second(TPS) Number of Processes MV2 MV2+GDR 0 500 1000 1500 2000 2500 3000 3500 4 8 16 32 AverageTimeStepsper second(TPS) Number of Processes 64K Particles 256K Particles 2X 2X
- 27. HPCAC-Switzerland (April ’18) 27Network Based Computing Laboratory Application-Level Evaluation (Cosmo) and Weather Forecasting in Switzerland 0 0.2 0.4 0.6 0.8 1 1.2 16 32 64 96NormalizedExecutionTime Number of GPUs CSCS GPU cluster Default Callback-based Event-based 0 0.2 0.4 0.6 0.8 1 1.2 4 8 16 32 NormalizedExecutionTime Number of GPUs Wilkes GPU Cluster Default Callback-based Event-based • 2X improvement on 32 GPUs nodes • 30% improvement on 96 GPU nodes (8 GPUs/node) C. Chu, K. Hamidouche, A. Venkatesh, D. Banerjee , H. Subramoni, and D. K. Panda, Exploiting Maximal Overlap for Non-Contiguous Data Movement Processing on Modern GPU-enabled Systems, IPDPS’16 On-going collaboration with CSCS and MeteoSwiss (Switzerland) in co-designing MV2-GDR and Cosmo Application Cosmo model: http://www2.cosmo-model.org/content /tasks/operational/meteoSwiss/
- 28. HPCAC-Switzerland (April ’18) 28Network Based Computing Laboratory • Scalability for million to billion processors • Unified Runtime for Hybrid MPI+PGAS programming (MPI + OpenSHMEM, MPI + UPC, CAF, UPC++, …) • Integrated Support for GPGPUs • Optimized MVAPICH2 for OpenPower (with/ NVLink) and ARM • Application Scalability and Best Practices Overview of A Few Challenges being Addressed by the MVAPICH2 Project for Exascale
- 29. HPCAC-Switzerland (April ’18) 29Network Based Computing Laboratory 0 0.5 1 1.5 0 1 2 4 8 16 32 64 128 256 512 1K 2K Latency(us) MVAPICH2-2.3rc1 SpectrumMPI-10.1.0.2 OpenMPI-3.0.0 Intra-node Point-to-Point Performance on OpenPower Platform: Two nodes of OpenPOWER (Power8-ppc64le) CPU using Mellanox EDR (MT4115) HCA Intra-Socket Small Message Latency Intra-Socket Large Message Latency Intra-Socket Bi-directional BandwidthIntra-Socket Bandwidth 0.30us 0 20 40 60 80 4K 8K 16K 32K 64K 128K 256K 512K 1M 2M Latency(us) MVAPICH2-2.3rc1 SpectrumMPI-10.1.0.2 OpenMPI-3.0.0 0 10000 20000 30000 40000 1 8 64 512 4K 32K 256K 2M Bandwidth(MB/s) MVAPICH2-2.3rc1 SpectrumMPI-10.1.0.2 OpenMPI-3.0.0 0 20000 40000 60000 80000 1 8 64 512 4K 32K 256K 2M Bandwidth(MB/s) MVAPICH2-2.3rc1 SpectrumMPI-10.1.0.2 OpenMPI-3.0.0
- 30. HPCAC-Switzerland (April ’18) 30Network Based Computing Laboratory 0 1 2 3 4 0 1 2 4 8 16 32 64 128 256 512 1K 2K Latency(us) MVAPICH2-2.3rc1 SpectrumMPI-10.1.0.2 OpenMPI-3.0.0 Inter-node Point-to-Point Performance on OpenPower Platform: Two nodes of OpenPOWER (Power8-ppc64le) CPU using Mellanox EDR (MT4115) HCA Small Message Latency Large Message Latency Bi-directional BandwidthBandwidth 0 50 100 150 200 4K 8K 16K 32K 64K 128K 256K 512K 1M 2M Latency(us) MVAPICH2-2.3rc1 SpectrumMPI-10.1.0.2 OpenMPI-3.0.0 0 5000 10000 15000 1 8 64 512 4K 32K 256K 2M Bandwidth(MB/s) MVAPICH2-2.3rc1 SpectrumMPI-10.1.0.2 OpenMPI-3.0.0 0 10000 20000 30000 1 8 64 512 4K 32K 256K 2M Bandwidth(MB/s) MVAPICH2-2.3rc1 SpectrumMPI-10.1.0.2 OpenMPI-3.0.0
- 31. HPCAC-Switzerland (April ’18) 31Network Based Computing Laboratory 0 5 10 15 20 1 2 4 8 16 32 64 128 256 512 1K 2K 4K 8K Latency(us) Message Size (Bytes) INTRA-NODE LATENCY (SMALL) INTRA-SOCKET(NVLINK) INTER-SOCKET MVAPICH2-GDR: Performance on OpenPOWER (NVLink + Pascal) 0 10 20 30 40 1 4 16 64 256 1K 4K 16K 64K 256K 1M 4M Bandwidth(GB/sec) Message Size (Bytes) INTRA-NODE BANDWIDTH INTRA-SOCKET(NVLINK) INTER-SOCKET 0 2 4 6 8 1 4 16 64 256 1K 4K 16K 64K 256K 1M 4M Bandwidth(GB/sec) Message Size (Bytes) INTER-NODE BANDWIDTH Platform: OpenPOWER (ppc64le) nodes equipped with a dual-socket CPU, 4 Pascal P100-SXM GPUs, and 4X-FDR InfiniBand Inter-connect 0 200 400 16K 32K 64K 128K256K512K 1M 2M 4M Latency(us) Message Size (Bytes) INTRA-NODE LATENCY (LARGE) INTRA-SOCKET(NVLINK) INTER-SOCKET 0 10 20 30 1 2 4 8 16 32 64 128 256 512 1K 2K 4K 8K Latency(us) Message Size (Bytes) INTER-NODE LATENCY (SMALL) 0 200 400 600 800 1000 Latency(us) Message Size (Bytes) INTER-NODE LATENCY (LARGE) Intra-node Bandwidth: 33.2 GB/sec (NVLINK)Intra-node Latency: 13.8 us (without GPUDirectRDMA) Inter-node Latency: 23 us (without GPUDirectRDMA) Inter-node Bandwidth: 6 GB/sec (FDR) Available in MVAPICH2-GDR 2.3a
- 32. HPCAC-Switzerland (April ’18) 32Network Based Computing Laboratory 0 10 20 30 40 4 8 16 32 64 128 256 512 1K 2K 4K Latency(us) MVAPICH2-GDR-Next SpectrumMPI-10.1.0.2 OpenMPI-3.0.0 Scalable Host-based Collectives with CMA on OpenPOWER (Intra-node Reduce & AlltoAll) (Nodes=1,PPN=20) 0 50 100 150 200 4 8 16 32 64 128 256 512 1K 2K 4K Latency(us) MVAPICH2-GDR-Next SpectrumMPI-10.1.0.2 OpenMPI-3.0.0 (Nodes=1,PPN=20) Up to 5X and 3x performance improvement by MVAPICH2 for small and large messages respectively 3.6X Alltoall Reduce 5.2X 3.2X 3.3X 0 400 800 1200 1600 2000 8K 16K 32K 64K 128K 256K 512K 1M Latency(us) MVAPICH2-GDR-Next SpectrumMPI-10.1.0.2 OpenMPI-3.0.0 (Nodes=1,PPN=20) 0 2500 5000 7500 10000 8K 16K 32K 64K 128K 256K 512K 1M Latency(us) MVAPICH2-GDR-Next SpectrumMPI-10.1.0.2 OpenMPI-3.0.0 (Nodes=1,PPN=20) 3.2X 1.4X 1.3X 1.2X
- 33. HPCAC-Switzerland (April ’18) 33Network Based Computing Laboratory 0 1000 2000 16K 32K 64K 128K 256K 512K 1M 2M Latency(us) MVAPICH2-GDR-Next SpectrumMPI-10.1.0 OpenMPI-3.0.0 3X 0 2000 4000 16K 32K 64K 128K 256K 512K 1M 2M Latency(us) Message Size MVAPICH2-GDR-Next SpectrumMPI-10.1.0 OpenMPI-3.0.0 34% 0 1000 2000 3000 4000 16K 32K 64K 128K 256K 512K 1M 2M Latency(us) Message Size MVAPICH2-GDR-Next SpectrumMPI-10.1.0 OpenMPI-3.0.0 0 1000 2000 16K 32K 64K 128K 256K 512K 1M 2M Latency(us) MVAPICH2-GDR-Next SpectrumMPI-10.1.0 OpenMPI-3.0.0 Optimized All-Reduce with XPMEM on OpenPOWER(Nodes=1,PPN=20) Optimized Runtime Parameters: MV2_CPU_BINDING_POLICY=hybrid MV2_HYBRID_BINDING_POLICY=bunch • Optimized MPI All-Reduce Design in MVAPICH2 – Up to 2X performance improvement over Spectrum MPI and 4X over OpenMPI for intra-node 2X (Nodes=2,PPN=20) 4X 48% 3.3X 2X 2X
- 34. HPCAC-Switzerland (April ’18) 34Network Based Computing Laboratory 0 1 2 3 0 1 2 4 8 16 32 64 128 256 512 1K 2K 4K Latency(us) MVAPICH2 Intra-node Point-to-point Performance on ARMv8 0 1000 2000 3000 4000 5000 Bandwidth(MB/s) MVAPICH2 0 2000 4000 6000 8000 10000 BidirectionalBandwidth MVAPICH2 Platform: ARMv8 (aarch64) MIPS processor with 96 cores dual-socket CPU. Each socket contains 48 cores. 0 200 400 600 8K 16K 32K 64K 128K 256K 512K 1M 2M Latency(us) MVAPICH2 Small Message Latency Large Message Latency Bi-directional BandwidthBandwidth Available since MVAPICH2 2.3a 0.74 microsec (4 bytes)
- 35. HPCAC-Switzerland (April ’18) 35Network Based Computing Laboratory • Scalability for million to billion processors • Unified Runtime for Hybrid MPI+PGAS programming (MPI + OpenSHMEM, MPI + UPC, CAF, UPC++, …) • Integrated Support for GPGPUs • Optimized MVAPICH2 for OpenPower (with/ NVLink) and ARM • Application Scalability and Best Practices Overview of A Few Challenges being Addressed by the MVAPICH2 Project for Exascale
- 36. HPCAC-Switzerland (April ’18) 36Network Based Computing Laboratory 0 50 100 150 200 250 300 350 400 milc leslie3d pop2 lammps wrf2 tera_tf lu ExecutionTimein(S) Intel MPI 18.0.0 MVAPICH2 2.3 rc1 2% 6% 1% 6% Performance of SPEC MPI 2007 Benchmarks (KNL + Omni-Path) Mvapich2 outperforms Intel MPI by up to 10% 448 processes on 7 KNL nodes of TACC Stampede2 (64 ppn) 10% 2% 4%
- 37. HPCAC-Switzerland (April ’18) 37Network Based Computing Laboratory 0 20 40 60 80 100 120 milc leslie3d pop2 lammps wrf2 GaP tera_tf lu ExecutionTimein(S) Intel MPI 18.0.0 MVAPICH2 2.3 rc1 2% 4% Performance of SPEC MPI 2007 Benchmarks (Skylake + Omni-Path) MVAPICH2 outperforms Intel MPI by up to 38% 480 processes on 10 Skylake nodes of TACC Stampede2 (48 ppn) 0% 1% 0% -4% 38% -3%
- 38. HPCAC-Switzerland (April ’18) 38Network Based Computing Laboratory Application Scalability on Skylake and KNL MiniFE (1300x1300x1300 ~ 910 GB) Runtime parameters: MV2_SMPI_LENGTH_QUEUE=524288 PSM2_MQ_RNDV_SHM_THRESH=128K PSM2_MQ_RNDV_HFI_THRESH=128K 0 50 100 150 2048 4096 8192 ExecutionTime(s) No. of Processes (KNL: 64ppn) MVAPICH2 0 20 40 60 2048 4096 8192 ExecutionTime(s) No. of Processes (Skylake: 48ppn) MVAPICH2 0 500 1000 1500 48 96 192 384 768 No. of Processes (Skylake: 48ppn) MVAPICH2 NEURON (YuEtAl2012) Courtesy: Mahidhar Tatineni @SDSC, Dong Ju (DJ) Choi@SDSC, and Samuel Khuvis@OSC ---- Testbed: TACC Stampede2 using MVAPICH2-2.3b 0 1000 2000 3000 4000 64 128 256 512 1024 2048 4096 No. of Processes (KNL: 64ppn) MVAPICH2 0 500 1000 1500 68 136 272 544 1088 2176 4352 No. of Processes (KNL: 68ppn) MVAPICH2 0 1000 2000 48 96 192 384 768 1536 3072 No. of Processes (Skylake: 48ppn) MVAPICH2 Cloverleaf (bm64) MPI+OpenMP, NUM_OMP_THREADS = 2
- 39. HPCAC-Switzerland (April ’18) 39Network Based Computing Laboratory • MPI runtime has many parameters • Tuning a set of parameters can help you to extract higher performance • Compiled a list of such contributions through the MVAPICH Website – http://mvapich.cse.ohio-state.edu/best_practices/ • Initial list of applications – Amber – HoomDBlue – HPCG – Lulesh – MILC – Neuron – SMG2000 – Cloverleaf – SPEC (LAMMPS, POP2, TERA_TF, WRF2) • Soliciting additional contributions, send your results to mvapich-help at cse.ohio-state.edu. • We will link these results with credits to you. Applications-Level Tuning: Compilation of Best Practices
- 40. HPCAC-Switzerland (April ’18) 40Network Based Computing Laboratory • Traditional HPC – Message Passing Interface (MPI), including MPI + OpenMP – Support for PGAS and MPI + PGAS (OpenSHMEM, UPC) – Exploiting Accelerators • Deep Learning – Acceleration of Caffe, CNTK, TensorFlow, and many more – Out-of-core Processing • Cloud for HPC and BigData – Virtualization with SR-IOV and Containers HPC, Deep Learning, and Cloud
- 41. HPCAC-Switzerland (April ’18) 41Network Based Computing Laboratory • Deep Learning frameworks are a different game altogether – Unusually large message sizes (order of megabytes) – Most communication based on GPU buffers • Existing State-of-the-art – cuDNN, cuBLAS, NCCL --> scale-up performance – NCCL2, CUDA-Aware MPI --> scale-out performance • For small and medium message sizes only! • Proposed: Can we co-design the MPI runtime (MVAPICH2- GDR) and the DL framework (Caffe) to achieve both? – Efficient Overlap of Computation and Communication – Efficient Large-Message Communication (Reductions) – What application co-designs are needed to exploit communication-runtime co-designs? Deep Learning: New Challenges for MPI Runtimes Scale-upPerformance Scale-out Performance cuDNN NCCL gRPC Hadoop Proposed Co- Designs MPI cuBLAS A. A. Awan, K. Hamidouche, J. M. Hashmi, and D. K. Panda, S-Caffe: Co-designing MPI Runtimes and Caffe for Scalable Deep Learning on Modern GPU Clusters. In Proceedings of the 22nd ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming (PPoPP '17) NCCL2
- 42. HPCAC-Switzerland (April ’18) 42Network Based Computing Laboratory 0 10000 20000 30000 40000 50000 512K 1M 2M 4M Latency(us) Message Size (Bytes) MVAPICH2 BAIDU OPENMPI 0 1000000 2000000 3000000 4000000 5000000 6000000 8388608 16777216 33554432 67108864 134217728 268435456 536870912 Latency(us) Message Size (Bytes) MVAPICH2 BAIDU OPENMPI 1 10 100 1000 10000 100000 4 16 64 256 1024 4096 16384 65536 262144 Latency(us) Message Size (Bytes) MVAPICH2 BAIDU OPENMPI • 16 GPUs (4 nodes) MVAPICH2-GDR vs. Baidu-Allreduce and OpenMPI 3.0 MVAPICH2: Allreduce Comparison with Baidu and OpenMPI *Available with MVAPICH2-GDR 2.3a ~30X better MV2 is ~2X better than Baidu ~10X better OpenMPI is ~5X slower than Baidu ~4X better
- 43. HPCAC-Switzerland (April ’18) 43Network Based Computing Laboratory 1 10 100 1000 10000 100000 Latency(us) Message Size (Bytes) NCCL2 MVAPICH2-GDR MVAPICH2-GDR vs. NCCL2 – Broadcast Operation • Optimized designs in MVAPICH2-GDR 2.3b* offer better/comparable performance for most cases • MPI_Bcast (MVAPICH2-GDR) vs. ncclBcast (NCCL2) on 16 K-80 GPUs *Will be available with upcoming MVAPICH2-GDR 2.3b 1 10 100 1000 10000 100000 Latency(us) Message Size (Bytes) NCCL2 MVAPICH2-GDR ~10X better ~4X better 1 GPU/node 2 GPUs/node Platform: Intel Xeon (Broadwell) nodes equipped with a dual-socket CPU, 2 K-80 GPUs, and EDR InfiniBand Inter-connect
- 44. HPCAC-Switzerland (April ’18) 44Network Based Computing Laboratory MVAPICH2-GDR vs. NCCL2 – Reduce Operation • Optimized designs in MVAPICH2-GDR 2.3b* offer better/comparable performance for most cases • MPI_Reduce (MVAPICH2-GDR) vs. ncclReduce (NCCL2) on 16 GPUs *Will be available with upcoming MVAPICH2-GDR 2.3b 1 10 100 1000 10000 100000 Latency(us) Message Size (Bytes) MVAPICH2-GDR NCCL2 ~5X better Platform: Intel Xeon (Broadwell) nodes equipped with a dual-socket CPU, 1 K-80 GPUs, and EDR InfiniBand Inter-connect 1 10 100 1000 4 8 16 32 64 128 256 512 1K 2K 4K 8K 16K 32K 64K Latency(us) Message Size (Bytes) MVAPICH2-GDR NCCL2 ~2.5X better
- 45. HPCAC-Switzerland (April ’18) 45Network Based Computing Laboratory MVAPICH2-GDR vs. NCCL2 – Allreduce Operation • Optimized designs in MVAPICH2-GDR 2.3b* offer better/comparable performance for most cases • MPI_Allreduce (MVAPICH2-GDR) vs. ncclAllreduce (NCCL2) on 16 GPUs *Will be available with upcoming MVAPICH2-GDR 2.3b 1 10 100 1000 10000 100000 Latency(us) Message Size (Bytes) MVAPICH2-GDR NCCL2 ~1.2X better Platform: Intel Xeon (Broadwell) nodes equipped with a dual-socket CPU, 1 K-80 GPUs, and EDR InfiniBand Inter-connect 1 10 100 1000 4 8 16 32 64 128 256 512 1K 2K 4K 8K 16K 32K 64K Latency(us) Message Size (Bytes) MVAPICH2-GDR NCCL2 ~3X better
- 46. HPCAC-Switzerland (April ’18) 46Network Based Computing Laboratory • Caffe : A flexible and layered Deep Learning framework. • Benefits and Weaknesses – Multi-GPU Training within a single node – Performance degradation for GPUs across different sockets – Limited Scale-out • OSU-Caffe: MPI-based Parallel Training – Enable Scale-up (within a node) and Scale-out (across multi-GPU nodes) – Scale-out on 64 GPUs for training CIFAR-10 network on CIFAR-10 dataset – Scale-out on 128 GPUs for training GoogLeNet network on ImageNet dataset OSU-Caffe: Scalable Deep Learning 0 50 100 150 200 250 8 16 32 64 128 TrainingTime(seconds) No. of GPUs GoogLeNet (ImageNet) on 128 GPUs Caffe OSU-Caffe (1024) OSU-Caffe (2048) Invalid use case OSU-Caffe publicly available from http://hidl.cse.ohio-state.edu/ Support on OPENPOWER will be available soon
- 47. HPCAC-Switzerland (April ’18) 47Network Based Computing Laboratory High Productivity and High Performance Out-of-Core DNN Training • Large Size Deep Neural Networks (DNNs) cannot be trained on GPUs due to memory limitation! – ResNet-50 - state-of-the-art DNN architecture for Image Recognition (Trainable with a small batch size of 45) – Next generation models like Neural Machine Translation (NMT) are emerging that require even more memory • Can we design Out-of-core DNN training support using new features in CUDA 8/9 and hardware mechanisms in Pascal/Volta GPUs? • The proposed framework called OC-Caffe (Out-of-Core Caffe) shows the potential of managed memory designs that can provide performance with negligible/no overhead. – OC-Caffe eliminates 3,000 lines of code for a high- productivity design by exploiting Unified Memory features Submission Under Review
- 48. HPCAC-Switzerland (April ’18) 48Network Based Computing Laboratory • Comparable performance to Caffe-Default for “in-memory” batch sizes • OC-Caffe-Opt: up to 5X improvement over Intel-MKL-optimized CPU-based AlexNet training on Volta V100 GPU with CUDA9 and CUDNN7 Performance Trends for OC-Caffe OC-Caffe will be released by the HiDL Team@OSU hidl.cse.ohio-state.edu Out-of-core (over-subscription)Trainable (in-memory) 0 200 400 600 800 1000 1200 1400 1600 1800 Images/sec(Higherisbetter) caffe-gpu oc-caffe-naïve oc-caffe-opt oc-caffe-opt - only 2.9% degradation Submission Under Review 0 200 400 600 800 1000 1200 1400 1600 Img/sec(Higherisbetter) caffe-gpu oc-caffe-naïve oc-caffe-opt caffe-cpu intel-caffe intel-caffe-opt oc-caffe-opt is 5X better than intel-caffe-opt caffe-gpu cannot run X
- 49. HPCAC-Switzerland (April ’18) 49Network Based Computing Laboratory • Traditional HPC – Message Passing Interface (MPI), including MPI + OpenMP – Support for PGAS and MPI + PGAS (OpenSHMEM, UPC) – Exploiting Accelerators • Deep Learning – Acceleration of Caffe, CNTK, TensorFlow, and many more – Out-of-core Processing • Cloud for HPC and BigData – Virtualization with SR-IOV and Containers HPC, Deep Learning, and Cloud
- 50. HPCAC-Switzerland (April ’18) 50Network Based Computing Laboratory • Virtualization has many benefits – Fault-tolerance – Job migration – Compaction • Have not been very popular in HPC due to overhead associated with Virtualization • New SR-IOV (Single Root – IO Virtualization) support available with Mellanox InfiniBand adapters changes the field • Enhanced MVAPICH2 support for SR-IOV • MVAPICH2-Virt 2.2 supports: – OpenStack, Docker, and singularity Can HPC and Virtualization be Combined? J. Zhang, X. Lu, J. Jose, R. Shi and D. K. Panda, Can Inter-VM Shmem Benefit MPI Applications on SR-IOV based Virtualized InfiniBand Clusters? EuroPar'14 J. Zhang, X. Lu, J. Jose, M. Li, R. Shi and D.K. Panda, High Performance MPI Library over SR-IOV enabled InfiniBand Clusters, HiPC’14 J. Zhang, X .Lu, M. Arnold and D. K. Panda, MVAPICH2 Over OpenStack with SR-IOV: an Efficient Approach to build HPC Clouds, CCGrid’15
- 51. HPCAC-Switzerland (April ’18) 51Network Based Computing Laboratory 0 50 100 150 200 250 300 350 400 milc leslie3d pop2 GAPgeofem zeusmp2 lu ExecutionTime(s) MV2-SR-IOV-Def MV2-SR-IOV-Opt MV2-Native 1% 9.5% 0 1000 2000 3000 4000 5000 6000 22,20 24,10 24,16 24,20 26,10 26,16 ExecutionTime(ms) Problem Size (Scale, Edgefactor) MV2-SR-IOV-Def MV2-SR-IOV-Opt MV2-Native 2% • 32 VMs, 6 Core/VM • Compared to Native, 2-5% overhead for Graph500 with 128 Procs • Compared to Native, 1-9.5% overhead for SPEC MPI2007 with 128 Procs Application-Level Performance on Chameleon SPEC MPI2007Graph500 5%
- 52. HPCAC-Switzerland (April ’18) 52Network Based Computing Laboratory 0 500 1000 1500 2000 2500 3000 22,16 22,20 24,16 24,20 26,16 26,20 BFSExecutionTime(ms) Problem Size (Scale, Edgefactor) Graph500 Singularity Native 0 50 100 150 200 250 300 CG EP FT IS LU MG ExecutionTime(s) NPB Class D Singularity Native • 512 Processes across 32 nodes • Less than 7% and 6% overhead for NPB and Graph500, respectively Application-Level Performance on Singularity with MVAPICH2 7% 6% J. Zhang, X .Lu and D. K. Panda, Is Singularity-based Container Technology Ready for Running MPI Applications on HPC Clouds?, UCC ’17, Best Student Paper Award
- 53. HPCAC-Switzerland (April ’18) 53Network Based Computing Laboratory 0 5000 10000 15000 20000 Bandwidth(MB/s) Message Size (Bytes) GPU-GPU Inter-node Bi-Bandwidth Docker Native 0 2000 4000 6000 8000 10000 12000 Bandwidth(MB/s) Message Size (Bytes) GPU-GPU Inter-node Bandwidth Docker Native 1 10 100 1000 1 4 16 64 256 1K 4K 16K 64K 256K 1M 4M Latency(us) Message Size (Bytes) GPU-GPU Inter-node Latency Docker Native MVAPICH2-GDR-2.3a Intel Haswell (E5-2687W @ 3.10 GHz) node - 20 cores NVIDIA Volta V100 GPU Mellanox Connect-X4 EDR HCA CUDA 9.0 Mellanox OFED 4.0 with GPU-Direct-RDMA MVAPICH2-GDR on Container with Negligible Overhead
- 54. HPCAC-Switzerland (April ’18) 54Network Based Computing Laboratory MVAPICH2 – Plans for Exascale • Performance and Memory scalability toward 1-10M cores • Hybrid programming (MPI + OpenSHMEM, MPI + UPC, MPI + CAF …) • MPI + Task* • Enhanced Optimization for GPU Support and Accelerators • Taking advantage of advanced features of Mellanox InfiniBand • Tag Matching* • Adapter Memory* • Enhanced communication schemes for upcoming architectures • Knights Landing with MCDRAM* • NVLINK* • CAPI* • Enhanced Support for Deep Learning • Extended topology-aware collectives • Extended Energy-aware designs and Virtualization Support • Extended Support for MPI Tools Interface (as in MPI 3.0) • Extended FT support • Support for * features will be available in future MVAPICH2 Releases
- 55. HPCAC-Switzerland (April ’18) 55Network Based Computing Laboratory Funding Acknowledgments Funding Support by Equipment Support by
- 56. HPCAC-Switzerland (April ’18) 56Network Based Computing Laboratory Personnel Acknowledgments Current Students (Graduate) – A. Awan (Ph.D.) – R. Biswas (M.S.) – M. Bayatpour (Ph.D.) – S. Chakraborthy (Ph.D.) – C.-H. Chu (Ph.D.) – S. Guganani (Ph.D.) Past Students – A. Augustine (M.S.) – P. Balaji (Ph.D.) – S. Bhagvat (M.S.) – A. Bhat (M.S.) – D. Buntinas (Ph.D.) – L. Chai (Ph.D.) – B. Chandrasekharan (M.S.) – N. Dandapanthula (M.S.) – V. Dhanraj (M.S.) – T. Gangadharappa (M.S.) – K. Gopalakrishnan (M.S.) – R. Rajachandrasekar (Ph.D.) – G. Santhanaraman (Ph.D.) – A. Singh (Ph.D.) – J. Sridhar (M.S.) – S. Sur (Ph.D.) – H. Subramoni (Ph.D.) – K. Vaidyanathan (Ph.D.) – A. Vishnu (Ph.D.) – J. Wu (Ph.D.) – W. Yu (Ph.D.) Past Research Scientist – K. Hamidouche – S. Sur Past Post-Docs – D. Banerjee – X. Besseron – H.-W. Jin – W. Huang (Ph.D.) – W. Jiang (M.S.) – J. Jose (Ph.D.) – S. Kini (M.S.) – M. Koop (Ph.D.) – K. Kulkarni (M.S.) – R. Kumar (M.S.) – S. Krishnamoorthy (M.S.) – K. Kandalla (Ph.D.) – M. Li (Ph.D.) – P. Lai (M.S.) – J. Liu (Ph.D.) – M. Luo (Ph.D.) – A. Mamidala (Ph.D.) – G. Marsh (M.S.) – V. Meshram (M.S.) – A. Moody (M.S.) – S. Naravula (Ph.D.) – R. Noronha (Ph.D.) – X. Ouyang (Ph.D.) – S. Pai (M.S.) – S. Potluri (Ph.D.) – J. Hashmi (Ph.D.) – H. Javed (Ph.D.) – P. Kousha (Ph.D.) – D. Shankar (Ph.D.) – H. Shi (Ph.D.) – J. Zhang (Ph.D.) – J. Lin – M. Luo – E. Mancini Current Research Scientists – X. Lu – H. Subramoni Past Programmers – D. Bureddy – J. Perkins Current Research Specialist – J. Smith – M. Arnold – S. Marcarelli – J. Vienne – H. Wang Current Post-doc – A. Ruhela – K. Manian Current Students (Undergraduate) – N. Sarkauskas (B.S.)
- 57. HPCAC-Switzerland (April ’18) 57Network Based Computing Laboratory Thank You! Network-Based Computing Laboratory http://nowlab.cse.ohio-state.edu/ [email protected] The High-Performance MPI/PGAS Project http://mvapich.cse.ohio-state.edu/ The High-Performance Deep Learning Project http://hidl.cse.ohio-state.edu/ The High-Performance Big Data Project http://hibd.cse.ohio-state.edu/