The Case For Docker In Multi-Cloud Enabled Bioinformatics Applications
4 likes516 views
The document discusses the implementation of Docker for multi-cloud bioinformatics applications, highlighting its advantages over traditional virtual machine virtualization in terms of efficiency and portability. It presents a system called elastichpc-docker that enables users to create and manage clusters of containers across different cloud platforms, facilitating complex bioinformatics workflows. The evaluation of elastichpc-docker shows it to perform favorably against existing solutions, particularly in the context of medical variant detection workflows.
The Case For Docker In Multi-Cloud Enabled Bioinformatics Applications
- 1. The case for Docker in multi- cloud enabled bioinformatics applications Ahmed Ali, Mohamed M. ElKalioby, Mohamed Abouelhoda Nile University, Egypt Presented By Mohamed M. El-Kalioby, MSc 1
- 2. Introduction ● Next generation sequencing technology has changed the traditional bioinformatics practice ● Sophisticated multi-step workflows used to transform the raw sequence data into knowledge. ● One NGS workflow can include tens of tasks and hundreds of information sources integrated together to achieve the analysis goals. ● Medical Variant Detection Workflow is an example of such workflows. 2
- 3. Medical Variant Detection Workflow (MVDW) 3
- 4. Medical Variant Detection Workflow (2) ● Multiple Versions and Instances of the workflow needed ● Tools and parameters can be changed ● per user, where each one may require certain modules, annotation databases, and special post-processing; ● per experiment type, e.g., whole genome, whole exome, or RNAseq in a single or multiplexed mode ● per sequencing platforms, illumina, IonTorrent, or any other one. 4
- 5. Requirements5 ● Efficient Dynamic Deployment Strategy ● The deployed system should use HPC resources ● Able to consume cloud computing resources (private and public clouds)
- 6. Virtualization Technology ● the whole system with all modules, databases and the related dependencies are packaged in a virtual machine (VM) image. ● These images can be then used to instantiate a virtual machine running in private or public cloud. ● Examples from sequence analysis ● Crossbow for NGS read alignment & SNP calling, ● RSD-Cloud for comparative genomics ● … many more 6
- 7. Virtual Technology (2) ● The traditional engine for running the virtual machine instances is based either on ● Oracle Virtual Box, ● KVM, ● Xen Hypervisor ● VMware 7
- 8. Docker8 ● Docker provides a new level of virtualization ● the computing machine (including the operating system) is not virtualized, ● Only the application and the related dependencies are encapsulated in a ’virtual’ isolated process INFRASTRUCTURE Operating System Virtual Machine Hypervisor VM1 VM2 … VMn APP1 APP2 …. APPn INFRASTRUCTURE Operating System Container Container … Container APPnAPP1 APP2 … Container Engine Software Stack with Virtual Machines Software Stack with Containers (a) (b)
- 9. Usage of Docker 9 Dockerclient DockerServer (Daemon) Pull Image Download/upload Images Build Image Run Container Build/Push container images to local registry Terminate Container Docker public registry Local registry Infrastructure Operating System container container Run containers
- 10. Why Docker10 ● Reduced execution overhead compared to traditional whole machine virtualization ● Provides an effective solution to the image portability problem. ● Virtual machine images running in Amazon are not compatible with those running in Google and vice versa which directly lead to duplication of work to prepare new images with each deployment.
- 11. Challenges ● Extra layers need to be built on top of Docker to enable the use of HPC resources (computer cluster) and multi-cloud platforms ● Deployment in different commercial clouds is not an easy task. ● Each cloud platforms has different APIs and different business models. ● Images are compatible with different providers 11
- 12. Contribution ● Define use case scenario for using Docker within a computer cluster for bioinformatics workflows. ● Evaluate its performance in comparison to the use of native hardware and usual virtual machines, in private and public cloud. ● We also present a new version of our multicloud elasticHPC, referred to as elasticHPC-Docker 1. enable the user deploy and run multi-step whole analysis workflows, 2. create computer cluster with Docker based applications and define a use case scenario for that 3. support the use of private clouds as well as commercial clouds like Amazon and Google. 12
- 13. Containers in the Cloud13
- 14. Google ● Google Cloud offers a container service in the form of two products 1. container-optimized virtual machine images, which includes programs to run standard Docker images, according to a user defined file in YAML format. 2. Google Kubernetes Engine (GKE) to create a cluster of virtual machines that can run Docker images. GKE is based on pods, ● Google has established Google container registry (GCR). ● Cost: ● The optimized container images and GKE run at no extra cost. pays usual price of virtual machines. ● GKE charges an extra fee of $0.15 per hour per cluster on top of the usual machine price (for cluster size > 5 nodes). ● GKE has two limitations: 1. It does not support Docker’s private images. 2. The cluster size in GKE cannot exceed 100 nodes. 14
- 15. Amazon ● Amazon provides Elastic Container Service (ECS). ● ECS enables the deployment of Docker containers on Amazon EC2. ● Amazon uses docker-compose to manage docker containers. ● Docker-compose facilitates the process of setting up a multi-container application by defining the application and all its dependencies in a single file using YAML format. ● The instantiated machines include programs to automatically configure the Docker environment. ● Amazon has its own images registry. ● Cost: ● the user pays for same as that of the usual instance types. ● If the load balancing service is selected, the user pays an extra small cost of $0.025 per hour and $0.008 per GB transferred between instances ● Limitations: ● It does not support attaching EBS volumes to the running containers. 15
- 16. ElasticHPC-Docker Features ● Ability to port and run any docker image to either private or commercial clouds. ● Creation and management of a cluster of containers. The cluster can use single or multiple machines. ● The computer cluster can have nodes from different cloud providers; i.e. some nodes can come from Amazon and some can come from Google. ● Ability to create and destroy containers in the run-time. This makes it possible to run multiple containers on the same machine, one at a time. ● The package supports scaling up/down of virtual machines (worker nodes) in a running clusters. 16
- 17. ElasticHPC-Docker Features (2) 17 ● The package allows mounting of virtual disks and establishment of a shared file system to the containers (Default option is the NFS). In AWS, we use EBS volumes and in Google we use persistent storage disks. ● elasticHPC-Docker automatically configures a job scheduler (including security settings among the different providers) among the containers. The default job schedule is PBS Torque, but SGE is also supported. ● The current package includes many Docker specification files (DockerFile) for the most important tools for NGS data analysis. These include Fastx, BWA, GATK . ● It includes a number of structural bioinformatics tools, including AutoDock, Frodock, and AMBER GROMACS,, among others;.
- 18. EHPC-Docker (Use Case)18 EHPC-Client EHPC-VM Manager Port 5000 Communication with VM Manager Port 5555 Ports1:4999, 5001:65535 Container Communication with Container service Master Node Communication Among conainer Service Communication Among Containerized Services Attached Data Volume Shared File System (Block Storage) Running on Users PC EHPC-VM Manager Port 5000 Port 5555 Ports1:4999, 5001:65535 Container Slave Node Worker Node Attached Data Volume EHPC-VM Manager Port 5000 Port 5555 Ports1:4999, 5001:65535 Container Slave Node Worker Node Attached Data Volume EHPC-VM Manager Port 5000 Port 5555 Ports1:4999, 5001:65535 Container Slave Node Worker Node Attached Data Volume 1. User downloads the EHPC-Docker client2. User runs the client to create a cluster on a supported clouda. The client starts Master nodeb. Master node creates the rest of the cluster in parallelc. Master node distributes the URL of the image registryd. Master and worker nodes retrieve the image and start the containers. e. Once done, the master node sets up the ports and finalizes the configuration of in terms of setting up the job scheduler and the shared storage.Cluster is ready
- 19. Experiments ● We conducted two experiments: 1. Measure the time for establishing container clusters over different cloud platforms. 2. Measure the performance of using Docker when running the variant detection workflow. 19
- 20. Experiment 120 1. GKE is faster than ECS 2. elasticHPC is faster than GKE 3. elasticHPC is close to ECS
- 21. Experiment 2 ● For this experiment, we used an exome dataset from DePristo et al. of size ~ 9 GB. ● The exome is a set of NGS reads sequenced only from the whole coding regions of a genome.) ● The workflow was executed three times independently on Google, AWS, and private cloud based on OpenStack. ● In each cloud, the 9 GB input data is divided into blocks to be processed in parallel over the cluster nodes. ● For fair comparison, we used machines of as similar specifications as possible. ● Amazon: m3.2xlarge (8 C, Intel 2.5 GHz, 30 GB RAM, SSD disks, $0.532/hour), ● Google: n1-highmem-8(8 C, Intel 2.5 GHz, 52 GB RAM, SSD disks,$0.504/hour) ● OpenStack: we used local machine with 8 Cores, 56 GB RAM. 21
- 22. Experiment 2 Physical Servers 22 Docker is too close to physical
- 23. Experiment 2 Google Cloud 23 ElasticHPC is faster than GCE Containers
- 24. Experiment 2 Amazon Cloud 24 ElasticHPC is very close to Amazon ECS
- 25. Conclusion ● We introduced elasticHPC-Docker based on container technology. ● Our package enables the creation of a computer cluster with containerized applications and workflows in private and in different commercial clouds using single interface. ● It includes options to run bioinformatics applications and workflows for large datasets ● Through the container technology, elasticHPC-Docker provides an efficient solution to the inter-operability among commercial clouds, ● It is efficient in practice with reduced overhead especially on local infrastructures. ● It is available on http://www.elastichpc.org 25
- 26. 26 Thank You