Using Docker containers for scientific environments - on-premises and in the cloud - HEPiX 2017
- 1. Using Docker Containers for Scientific Environments — On- Premises and in the Cloud Sergey Yakubov, Martin Gasthuber, Birgit Lewendel KEK, Tsukuba, 18.10.2017
- 2. Page 2 Contents Introduction Scientific environments on-premises • IT-Managed containers • Custom user containers Scientific environments in hybrid clouds • HNSciCloud project • Using cloud to extend local resources Conclusions and outlook | Sergey Yakubov | 18.10.2017 | Hepix Fall 2017 | KEK, Tsukuba
- 3. Page 3 Introduction • Batch farm (HTCondor) – see talk by T. Finnern • HPC cluster Maxwell (SLURM) • Large storage, fast network and CPUs • 12,000 cores, Infiniband, 76 TB memory, 3.3 PB storage • Used mostly for offline data analyses/numerical simulations • But also for online analyses (more in the future) • Docker containers | Sergey Yakubov | 18.10.2017 | Hepix Fall 2017 | KEK, Tsukuba Compute resources at DESY
- 4. Page 4 Introduction • Using Docker container technology we can create environments that allow to: • separate IT and user requirements/dependencies • separate responsibilities - IT focus on scaling and container template construction, physicist on application development • provide compute resources dynamically and quickly, whether on top of existing local resources or in the cloud • control provisioned resources - storage, CPUs, memory, networks, … • Can we do this with OpenStack & Co? Probably yes, but … | Sergey Yakubov | 18.10.2017 | Hepix Fall 2017 | KEK, Tsukuba Containerized scientific environments
- 5. Page 5 Scientific environments on-premises • A Dockerfile is created by IT/ group admins (e.g. Debian image with software for a specific experiment) and stored as Puppet resource • Puppet automatically creates an image on Dockerfile changes and pushes it to DESY’s Docker registry • Compute resources are reserved via SLURM • At a specified time SLURM job starts Docker containers on each of the allocated compute nodes with sshd daemon. • Users with corresponding rights can login and do their work. IT-Managed Containers | Sergey Yakubov | 18.10.2017 | Hepix Fall 2017 | KEK, Tsukuba admin admin user ssh
- 6. Page 6 Scientific environments on-premises • User submits a SLURM job script with Docker commands • Compute resources are allocated via SLURM • SLURM execute specified Docker containers on each of the allocated compute nodes • Any Docker images can be used • Docker authorization plugin takes care about security. Custom user containers | Sergey Yakubov | 18.10.2017 | Hepix Fall 2017 | KEK, Tsukuba user
- 7. Page 7 Scientific environments on-premises Example - SIMEX | Sergey Yakubov | 18.10.2017 | Hepix Fall 2017 | KEK, Tsukuba SimEx - photon science simulation platform https://github.com/eucall-software/simex_platform
- 8. Page 8 Scientific environments on-premises Example - SIMEX | Sergey Yakubov | 18.10.2017 | Hepix Fall 2017 | KEK, Tsukuba X-ray wavefront propagation calculator • Propagation of light through optical elements • Utilizes SRW (Synchrotron Radiation Workshop) library • C++ core + python wrappers • Hybrid OpenMP/MPI parallelization 0 2 4 6 8 10 12 14 0 10 20 30 40 Speed-up N cores Threads x MPI processes Number of nodes Total time Time/file 1x1 1 11h 1031 s 40x1 1 65 min 98 s 4x10 4 7.5 min 45 s 8x5 8 4.2 min 51 s Single source file 40 source files
- 9. Page 9 Scientific environments on-premises Example - SIMEX | Sergey Yakubov | 18.10.2017 | Hepix Fall 2017 | KEK, Tsukuba X-ray wavefront propagation calculator • Propagation of light through optical elements • Utilizes SRW (Synchrotron Radiation Workshop) library • C++ core + python wrappers • Hybrid OpenMP/MPI parallelization 0 2 4 6 8 10 12 14 0 10 20 30 40 Speed-up N cores Threads x MPI processes Number of nodes Total time Time/file 1x1 1 11h 1031 s 40x1 1 65 min 98 s 4x10 4 7.5 min 45 s 8x5 8 4.2 min 51 s Single source file 40 source files 160x speed-up
- 10. Page 10 Helix Nebula Science Cloud Joint Pre-Commercial Procurement | Sergey Yakubov | 18.10.2017 | Hepix Fall 2017 | KEK, Tsukuba Procurers: CERN, CNRS, DESY, EMBL- EBI, ESRF, IFAE, INFN, KIT, STFC, SURFSara Experts: Trust-IT & EGI.eu The group of procurers have committed • Procurement funds • Manpower for testing/evaluation • Use-cases with applications & data • In-house IT resources Resulting services will be made available to end-users from many research communities Co-funded via H2020 Grant Agreement 687614 Total procurement budget >5M€ * Thanks to the CERN IT Group for the provided HNSciCloud slides *
- 11. Page 11 Helix Nebula Science Cloud • Compute and storage • support a range of virtual machine and container configurations including HPC working with datasets in the petabyte range • Transparent Data Access • provide transparent for user on-premise’s data access from the cloud • Network connectivity • provide high-end network capacity via GEANT for the whole platform • Federated Identity Management • provide common identity and access management | Sergey Yakubov | 18.10.2017 | Hepix Fall 2017 | KEK, Tsukuba * Technical challenges
- 12. Page 12 Helix Nebula Science Cloud Preparation • Analysis of requirements, current market offers and relevant standards • Build stakeholder group • Develop tender material Implementation & Sharing Jan’16 Dec’18 Each step is competitive - only contractors that successfully complete the previous step can bid in the next 4 Designs 3 Prototypes 2 Pilots Call-off Feb’17 Call-off Dec’17 Tender Jul’16 We are here | Sergey Yakubov | 18.10.2017 | Hepix Fall 2017 | KEK, Tsukuba * Project phases
- 13. Page 13 Scientific environments in hybrid clouds Resources, Fast network, Transparent Data Access from HNSciCloud and SLURM Elastic Computing | Sergey Yakubov | 18.10.2017 | Hepix Fall 2017 | KEK, Tsukuba control node compute nodes Using cloud to extend local resources
- 14. Page 14 Scientific environments in hybrid clouds | Sergey Yakubov | 18.10.2017 | Hepix Fall 2017 | KEK, Tsukuba cloud compute nodes Using cloud to extend local resources control node compute nodes
- 15. Page 15 Scientific environments in hybrid clouds test.sh Example | Sergey Yakubov | 18.10.2017 | Hepix Fall 2017 | KEK, Tsukuba #!/bin/sh #SBATCH --partition=cloudXXX #SBATCH --workdir=/test_id #SBATCH --nodes=1 id –u > cloud_id.txt dockerrun centos:7 id –u > cloud_docker_id.txt local-node$ sbatch test.sh local-node$ id –u 12345 local-node$ cat cloud_id.txt 12345 local-node$ cat cloud_docker_id.txt 12345
- 16. Page 16 Conclusions and outlook Containerized scientific environment • Implemented via Docker • Isolates work of different users/groups • Same performance as on underlying infrastructure • Portable • More user experience to be gained Hybrid clouds • Dynamical cloud resource allocation/deallocation • Transparent to the user • user submits job to local scheduler • transparent data access from the cloud • thanks to Docker no need to install user software on the cloud VM • Performance to be tested | Sergey Yakubov | 18.10.2017 | Hepix Fall 2017 | KEK, Tsukuba
- 17. Thank you for you attention!