XSEDE April 2017
- 1. XSEDE for Computational Research Jeff Wereszczynski
- 2. What is XSEDE? • NSF Funded project (current allocation: 2016-2021, $115 million) • Provides access to multiple supercomputers with distinct hardware configurations • TACC: Stampede, Wrangler, Ranch • SDSC: Comet, Data Oasis • PSC: Bridges • Stanford: XStream • IU/TACC: Jetstream
- 3. Stampede 2 (18 Petaflops) Stampede is currently undergoing an upgrade and will include: § 4200 ”Knights Landing” nodes (68 cores each, with 4 threads/node, 96 GB RAM) § 1736 Intel Xeon Nodes § Omni-path interconnect (100 Gbps) § Job limitations § 48 hours § 80 nodes (5440 cores)
- 4. Other TACC Resources § Wrangler: § Visualization and Data Analytics § Fast I/O § 132 nodes (10 Ivy Bridge CPUs + 1 NVIDIA K40 GPU) § Ranch § Long term storage § Tape system (currently 61 PB of space)
- 5. Comet at SDSC (2 Petaflops) § 1984 nodes with Xeon E5-2680v3 processors (24 CPUs/node), 128 GB DDR4 DRAM, and 320 GB of SSD local scratch memory. § 36 GPU nodes with 4X NVIDIA K80 GPUs each (soon 36 4xP100 GPU nodes) § Large memory nodes with 1.5 TB of DRAM. § 56 Gbps FDR InfiniBand § Gateway hosting nodes and a Virtual Machine repository. § Job Limits: § 48 hours § 72 nodes (1728 cores) § Long term storage provided by Data Oasis
- 6. Bridges at PSC Bridges is made up of four types of nodes: § Four Integrity Superdome X servers, which are scale-up products that let users lock data once into their 12 terabytes of shared memory and then conduct analytics § May be run for up to 14 days § 42 HPE ProLiant DL580 servers, each of which has 3 terabytes of shared memory and provides virtualization and remote visualization. § 800 HPE Apollo 2000 nodes, each with 128 gigabytes of shared memory apiece, servicing capacity workloads. § 48 GPU nodes with either K80 or P100 GPUs
- 7. XStream at Stanford GPU Computing System: § Each of the 65 nodes has 8 NVIDIA K80 cards or 16 NVIDIA Kepler GPUs, interconnected through PCI-Express PLX-based switches. Each GPU has 12 GB of GDDR5 memory. § Compute nodes also feature 2 Intel Ivy-Bridge 10-core CPUs, 256 GB of DRAM and 450 GB of local SSD storage. § The system features 1.4 PB of Lustre storage (22 GB/s aggregate).
- 8. Jetstream at IU/TACC VM SIZE VCPUS RAM (GB) LOCAL STORAGE (GB) SU COST PER HOUR Tiny 1 2 8 1 Small 2 4 20 2 Medium 6 16 60 6 Large 10 30 120 10 XLarge 22 60 240 22 XX Large 44 120 480 44 Jetstream can be used in several different virtual machine (VM) sizes which are charged in service units (SUs) based on how much of the total system resource is used. The table below outlines the VM sizes created for Jetstream.
- 10. Extended Collaborative Support Service (ECSS) Support for: Performance analysis Petascale optimization Efficient use of accelerators I/O optimization Data analytics Visualization Use of XSEDE by science gateways Workflows
- 11. Allocation Types Trial allocation: 1000 SUs on Comet (1 day turnaround) Startup: 1k-50k SUs (varies by resource) Required material: Abstract + CV Allocation within a few days Education: 1k-50k SUs (varies by resource, ~1k-1.5k per student) Required material: Abstract + CV, justification for hardware Allocation within a few days
- 12. Allocation Types: Research• No limit (millions of SUs), one year allocation request • Required material: • Main proposal: • 10 page max (15 for requests >10 Million SUs) • Must justify calculations to be performed (length/number of runs, scientific results, etc.) • If not federally funded, should also justify the science • Be sure to list other computational resources • Scaling document • Benchmarking for all codes/systems on systems you plan to run on. • Also justify your storage requests! • Quarterly submissions, allocations begin ~3 months after submission • Request and justify ECSS support if needed Figure 3: Scaling of Octanucleosome Simulations on Stampede