High Performance Cloud Computing
5 likes•1,653 views
This document discusses high performance cloud computing. It covers 1) cloud infrastructure like EC2 instance types, 2) tools for provisioning and managing cloud resources such as AWS CloudFormation and Chef, 3) example HPC applications that can run on the cloud including genomics tools, and 4) best practices around optimizing costs and resources for high performance computing workloads in the cloud.
![include_recipe "packages"
include_recipe "ruby"
include_recipe "apache2"
if platform?("centos","redhat")
if dist_only?
# just the gem, we'll install the apache module within apache2
package "rubygem-passenger"
return
else
package "httpd-devel"
end
else
%w{ apache2-prefork-dev libapr1-dev }.each do |pkg|
package pkg do
action :upgrade
end
end
end
gem_package "passenger" do
version node[:passenger][:version]
end
execute "passenger_module" do
command 'echo -en "nnnn" | passenger-install-apache2-module'
creates node[:passenger][:module_path]
end](https://image.slidesharecdn.com/hpcgenomics-110922162517-phpapp02/85/High-Performance-Cloud-Computing-41-320.jpg)
High Performance Cloud Computing
- 1. High Performance Cloud Computing Deepak Singh P r i n c i p a l P r o d u c t M a n a g e r
- 2. Via butteryflysha under a CC-BY license
- 3. Image: Simon Cockell under CC-BY
- 4. “High Performance” Computing
- 5. using a large number of computers at the same time for a single task
- 7. batch
- 10. 4
- 15. instance types
- 16. t1.micro standard (m1) high memory (m2) high CPU (c1)
- 17. elastic
- 19. programmable
- 22. MPI
- 23. Cluster Compute Instance
- 24. 2*Intel Xeon 5570 23 GB RAM 1.7 TB disk
- 25. 10 gig E
- 26. Placement Group
- 27. Placement group
- 28. linpack
- 29. Cores 7040 R max 41.82 R peak 82.51
- 30. 231
- 31. 450
- 32. WIEN2K Parallel Performance H size 56,000 (25GB) Runtime (16x8 processors) Local (Infiniband) 3h:48 Cloud (10Gbps) 1h:30 ($40) 1200 atom unit cell; SCALAPACK+MPI diagonalization, matrix size 50k-100k Credit: K. Jorissen, F. D. Villa, and J. J. Rehr (U. Washington)
- 33. HPC is evolving
- 35. 2*Intel Xeon 5570 22 GB RAM 1.7 TB disk 2*NVidia M2050
- 36. 2. Provision & Manage
- 39. bootstrap
- 40. chef/puppet
- 41. include_recipe "packages" include_recipe "ruby" include_recipe "apache2" if platform?("centos","redhat") if dist_only? # just the gem, we'll install the apache module within apache2 package "rubygem-passenger" return else package "httpd-devel" end else %w{ apache2-prefork-dev libapr1-dev }.each do |pkg| package pkg do action :upgrade end end end gem_package "passenger" do version node[:passenger][:version] end execute "passenger_module" do command 'echo -en "nnnn" | passenger-install-apache2-module' creates node[:passenger][:module_path] end
- 42. familiar tools
- 44. LSF
- 45. Moab/Torque
- 46. Condor
- 47. StackIQ Rocks+
- 48. combine worlds
- 49. MIT Starcluster
- 50. $ starcluster start mycluster $ starcluster listclusters
- 55. 30,472 cores
- 56. ‘nuff said
- 59. S3 Input data Code Elastic Name Output MapReduce node S3 + SimpleDB Queries HDFS + BI Via JDBC, Pig, Hive Elastic cluster
- 60. 3. Applications
- 63. mapreduce for genomics http://bowtie-bio.sourceforge.net/crossbow/index.shtml http://contrail-bio.sourceforge.net http://bowtie-bio.sourceforge.net/myrna/index.shtml
- 67. 4. People Credit: Pieter Musterd a CC-BY-NC-ND license
- 68. most valuable
- 70. Tasks Instances
- 73. Tasks Queue Instances Increase instance count
- 81. on-demand
- 82. reserved
- 83. spot
- 87. Credit: Angel Pizzaro, U. Penn
- 88. NASA JPL
- 89. Stochastic Dual Dynamic Programming 44,000 CPU hrs in Oct 2010 http://aws.amazon.com/solutions/case-studies/psr/
- 91. Credit: Angel Pizzaro, U. Penn
- 93. 4
- 95. 2. Provision & Manage
- 96. 3. Applications
- 97. 4. People
- 98. [email protected] Twitter:@mndoci http://slideshare.net/mndoci http://mndoci.github.com Inspiration and ideas from Matt Wood, James Hamilton & Larry Lessig Credit” Oberazzi under a CC-BY-NC-SA license