The Pacific Research Platform: Leading Up to the National Research Platform
Download as PPTX, PDF1 like721 views
The Pacific Research Platform (PRP) aims to create a high-performance, science-driven big data superhighway across California by enhancing collaborations among universities and connecting various research facilities through efficient data transfer nodes. Launched with NSF funding, PRP focuses on addressing challenges in data-intensive research by integrating dedicated systems and specialized network architectures. Its progress includes connecting multiple campuses, improving data transfer speeds, and fostering community engagement in scientific initiatives.
The Pacific Research Platform: Leading Up to the National Research Platform
- 1. “The Pacific Research Platform: Leading Up to the National Research Platform” Opening Keynote The National Research Platform Workshop Montana State University Bozeman, MT August 7, 2017 Dr. Larry Smarr Director, California Institute for Telecommunications and Information Technology Harry E. Gruber Professor, Dept. of Computer Science and Engineering Jacobs School of Engineering, UCSD http://lsmarr.calit2.net 1
- 2. 30 Years Ago NSF Brought to University Researchers a DOE HPC Center Model NCSA Was Modeled on LLNL SDSC Was Modeled on MFEnet 1985/6
- 3. Launching the Nation’s Information Infrastructure: NSFnet Supernetwork and the Six NSF Supercomputers NCSA NSFNET 56 Kb/s Backbone (1986-8) PSCNCAR CTC JVNC SDSC PRP’s Backbone is One Million Times Faster!
- 4. I-WAY: Information Wide Area Year Supercomputing ‘95 • The First National 155 Mbps Research Network – 65 Science Projects – Into the San Diego Convention Center • I-Way Featured: – Networked Visualization Applications – Large-Scale Immersive Displays – I-Soft Programming Environment – Led to the Globus Project UIC http://archive.ncsa.uiuc.edu/General/Training/SC95/GII.HPCC.html See Session 1 Talk by Ian Foster
- 5. NSF’s PACI Program was Built on the vBNS to Prototype America’s 21st Century Information Infrastructure The PACI Grid Testbed National Computational Science 1997 vBNS led to
- 6. NSF’s OptIPuter Project: Using Supernetworks to Meet the Needs of Data-Intensive Researchers OptIPortal– Termination Device for the OptIPuter Global Backplane Calit2 (UCSD, UCI), SDSC, and UIC Leads—Larry Smarr PI Univ. Partners: NCSA, USC, SDSU, NW, TA&M, UvA, SARA, KISTI, AIST Industry: IBM, Sun, Telcordia, Chiaro, Calient, Glimmerglass, Lucent 2003-2009 $13,500,000 In August 2003, Jason Leigh and his students used RBUDP to blast data from NCSA to SDSC over the TeraGrid DTFnet, achieving18Gbps file transfer out of the available 20Gbps LS Slide 2005
- 7. We Have Been Working Toward PRP for 15 Years: NSF OptIPuter, Quartzite, Prism Awards PI Papadopoulos, 2013-2015 PI Smarr, 2002-2009 PI Papadopoulos, 2004-2007
- 8. Thirty Years After NSF Adopts DOE Supercomputer Center Model NSF Adopts DOE ESnet’s Science DMZ for High Performance Applications • A Science DMZ integrates 4 key concepts into a unified whole: – A network architecture designed for high-performance applications, with the science network distinct from the general-purpose network – The use of dedicated systems as data transfer nodes (DTNs) – Performance measurement and network testing systems that are regularly used to characterize and troubleshoot the network – Security policies and enforcement mechanisms that are tailored for high performance science environments http://fasterdata.es.net/science-dmz/ Science DMZ Coined 2010 The DOE ESnet Science DMZ and the NSF “Campus Bridging” Taskforce Report Formed the Basis for the NSF Campus Cyberinfrastructure Network Infrastructure and Engineering (CC-NIE) Program See Deep Dive 6 On High-Perf Networking
- 9. Based on Community Input and on ESnet’s Science DMZ Concept, NSF Has Funded Over 100 Campuses to Build Local Big Data Freeways Red 2012 CC-NIE Awardees Yellow 2013 CC-NIE Awardees Green 2014 CC*IIE Awardees Blue 2015 CC*DNI Awardees Purple Multiple Time Awardees Source: NSF
- 10. (GDC) Logical Next Step: The Pacific Research Platform Creates a Regional End-to-End Science-Driven “Big Data Superhighway” System NSF CC*DNI Grant $5M 10/2015-10/2020 PI: Larry Smarr, UC San Diego Calit2 Co-Pis: • Camille Crittenden, UC Berkeley CITRIS, • Tom DeFanti, UC San Diego Calit2, • Philip Papadopoulos, UCSD SDSC, • Frank Wuerthwein, UCSD Physics and SDSC Letters of Commitment from: • 50 Researchers from 15 Campuses • 32 IT/Network Organization Leaders See Deep Dive 3 on NSF Research CI
- 11. Developing Regional DMZs Require a Lot of Consultation-Sociotechnilogical Problem • May 2014 LS Gives Invited Presentation to UC IT Leadership Council – Strong Support from UC and UCOP CIOs • July 2014 LS Gives Invited Talk to CENIC Annual Retreat – CENIC/PW Agrees to Act as Backplane – CIO Support Extends to CA Private Research Universities • December 2014 UCOP CIO and VPR’s Provide PRP “Momentum Money” • January 2015 Kickoff of PRPv0 by Network Engineers – Begins Every Two Week Conference Calls, Now Weekly • March 2015 LS Invited “Blue Sky” Presentation to UC VCR/CIO Summit – NSF PRP Proposal Submitted With Letters of Commitment From: – 50 Researchers from 15 Campuses – 32 IT/Network Organization Leaders • July 2015 NSF Announces Funding for PRP • October 2015 PRP Grant Begins
- 12. Big Data Science Data Transfer Nodes (DTNs)- Flash I/O Network Appliances (FIONAs) UCSD Designed FIONAs To Solve the Disk-to-Disk Data Transfer Problem at Full Speed on 10G, 40G and 100G Networks FIONAS—10/40G, $8,000 FIONette—1G, $1,000 Phil Papadopoulos, SDSC & Tom DeFanti, Joe Keefe & John Graham, Calit2 John Graham, Calit2 See Deep Dive 4 How Does PRPv1 Work?
- 13. We Measure Disk-to-Disk Throughput with 10GB File Transfer 4 Times Per Day in Both Directions for All PRP Sites January 29, 2016 From Start of Monitoring 12 DTNs to 24 DTNs Connected at 10-40G in 1 ½ Years July 21, 2017 Source: John Graham, Calit2
- 14. Increasing Participation Through PRP Science Engagement Workshops Source: Camille Crittenden, UC Berkeley UC San Diego UC Merced UC Davis UC Berkeley See Session 3 on Engagement
- 15. The First National Research Platform Workshop on August 7-8, 2017 Announced in I2 Closing Keynote: Larry Smarr “Toward a National Big Data Superhighway” on Wednesday, April 26. Co-Chairs: Larry Smarr, Calit2 & Jim Bottum, Internet2 150 Attendees
- 16. PRP’s First 1.5 Years: Connecting Multi-Campus Application Teams and Devices
- 17. Particle Physics: Using the PRP to Fix the “Last Mile Problem” of the LHC in California ATLASCMS U.S. Institutions Participating in LHC LHC Data Generated by CMS & ATLAS Detectors Analyzed on OSG Maps from www.uslhc.us See Talks by Harvey Newman, Frank Wuerthwein, & Rob Gardner
- 18. Global Scientific Instruments Will Produce Ultralarge Datasets Continuously Requiring Dedicated Optic Fiber and Supercomputers Square Kilometer Array Large Synoptic Survey Telescope https://tnc15.terena.org/getfile/1939 3.2 Gpixel Camera Tracks ~40B Objects, Creates 10M Alerts/Night Within 1 Minute of Observing 2x100Gb/s See Session 5 Talk By Heidi Morgan “First Light” In 2019
- 19. 100 Gbps FIONA at UCSC Allows for Downloads to the UCSC Hyades Cluster from the LBNL NERSC Supercomputer for DESI Science Analysis 300 images per night. 100MB per raw image 30GB per night 120GB per night 250 images per night. 530MB per raw image 150 GB per night 800GB per night Source: Peter Nugent, LBNL Professor of Astronomy, UC Berkeley Precursors to LSST and NCSA NSF-Funded Cyberengineer Shaw Dong @UCSC Receiving FIONA Feb 7, 2017
- 20. Cancer Genomics Hub (UCSC) Was Housed in SDSC, But NIH Moved Dataset From SDSC to Uchicago - So the PRP Deployed a FIONA to Chicago’s MREN 1G 8G Data Source: David Haussler, Brad Smith, UCSC 15G Jan 2016 See Deep Dive 5 on Community Data Resources
- 21. PRP Is Linking the Laboratories of the Pacific Earthquake Engineering Research Center http://peer.berkeley.edu/ PEER Labs: UC Berkeley, Caltech, Stanford, UC Davis, UC San Diego, and UC Los Angeles John Graham Installing FIONette at PEER Feb 10, 2017
- 22. 40G FIONAs 20x40G PRP-connected WAVE@UC San Diego PRP Now Enables Distributed Virtual Reality (and Distributed Machine Learning) PRP WAVE @UC Merced Transferring 5 CAVEcam Images from UCSD to UC Merced: 2 Gigabytes now takes 2 Seconds (8 Gb/sec)
- 23. Visualizing Enormous Datasets Resulting from Supercomputer Simulations Using EVL’s DTN GPU Services (PRP Provided EVL DTN) Cosmological Volume Simulation • Computed on the ALCF Mira – 29 Billion Particles – 40 TB per Snapshot and 500 Snapshots • Then Sent to NCSA Blue Waters – 1st -Level Data Analytics & Visualization – Using “yt” • Web Services on EVL’s DTN – Used to Convert a Snapshot – To a DZI Multi-Resolution Image – Displayed Using SAGE2 – In the SC16 NCSA Booth Cosmologist Katrin Heitmann of ANL Standing in Front of the SAGE2 Display at SC16 Slide from Maxine Brown, EVL Director See Deep Dive 8 on SC17 Demos
- 24. The Prototype PRP Has Attracted New Application Drivers Scott Sellars, Marty Ralph Center for Western Weather and Water Extremes Frank Vernon - Expansion of HPWREN Tom Levy, Cultural Heritage Cryo EM
- 25. GPU JupyterHub: 2 x 14-core CPUs 256GB RAM 1.2TB FLASH 3.8TB SSD Nvidia K80 GPU Dual 40GbE NICs And a Trusted Platform Module GPU JupyterHub: 1 x 18-core CPUs 128GB RAM 3.8TB SSD Nvidia K80 GPU Dual 40GbE NICs And a Trusted Platform Module PRP UC-JupyterHub Backbone Next Step: Deploy Across PRP Source: John Graham, Calit2 UCB UCSD
- 26. Cryo-electron Microscopy (cryo-EM) Has Driven a “Resolution Revolution” in the Last Five Years Exposure (every 60 seconds): X & Y dimensions: 7420 x 7676 Pixels Frames per Movie: 10 - 50 Size: 3 - 10 GB per Movie Every 24 hours: Number of Movies: ~1400 Data Size: ~5 TB Typical Datasets: Length of Time: 2 - 6 Days Total size: 10 - 30 TB Each Cryo-EM ‘Image’ is Actually a Movie Source: Michael A. Cianfrocco, Elizabeth Villa, & Andres Leschziner, UCSD
- 27. Using PRP to Connect Cryo-EM across California With End Users and Computational Facilities Long term: ‣ Partner with Cryo-EM Facilities to Stream Data Straight from Microscopes (over PRP) to SDSC ‣ Perform All Cryo-EM Analysis (from Micrographs to 3D Models) via Web Browser on SDSC ‣ Expand Computing to Other XSEDE Resources (e.g. Xstream) and DOE’s NERSC Short term: ‣ Provide 2D and 3D Analysis on Particle Stacks on Comet at SDSC Source: Michael A. Cianfrocco, UCSD * * SDSC NERSC Xstream 3 Supercomputer Centers cosmic-cryoem.org ~20 Microscopes in CA UCLA UC Davis UC Santa Cruz SF Bay UC Berkeley, LBNL, UCSF, Stanford San Diego UCSD, TSRI, Salk* Extending to MSU
- 28. HPWREN Real-Time Network Cameras on Mountains for Environmental Observations and Fires San Diego County Red Mountain Fire Cameras • Southeast (left) “Highway” Fire • Southwest (center rear) “Poinsettia” Fire • West (right) “Tomahawk” Fire Source: Frank Vernon, Hans Werner Braun HPWREN May 14, 2014
- 29. PRP Backbone Sets Stage for 2017 Expansion of HPWREN, Connected to CENIC, into Orange and Riverside Counties • PRP CENIC 100G Link UCSD to SDSU – DTN FIONAs Endpoints – Data Redundancy – Disaster Recovery – High Availability – Network Redundancy • CENIC Enables PRP 10G Links Between UCSD, SDSU, & UCI HPWREN Servers • Potential Future UCR CENIC Anchor UCR UCI UCSD SDSU Source: Frank Vernon, Hans Werner Braun HPWREN UCI Antenna Dedicated June 27, 2017
- 30. UC Catalyst Project: Linking Cultural Heritage and Archaeology Datasets at UCB, UCLA, UCM and UCSD with CAVEkiosks 48 Megapixel CAVEkiosk UCSD Library 48 Megapixel CAVEkiosk UCB Library 24 Megapixel CAVEkiosk UCM Library See Session 2 Talk by Chris Hoffman
- 31. Director: F. Martin Ralph Website: cw3e.ucsd.edu Big Data Collaboration with: Source: Scott Sellers, CW3E Collaboration on Atmospheric Water in the West Between UC San Diego and UC Irvine Director, Soroosh Sorooshian, UCSD Website http://chrs.web.uci.edu
- 32. Calit2’s FIONA SDSC’s COMET Calit2’s FIONA Pacific Research Platform (10-100 Gb/s) GPUsGPUs Complete workflow time: 20 days20 hrs20 Minutes! UC, Irvine UC, San Diego Major Speedup in Scientific Work Flow Using the PRP Source: Scott Sellers, CW3E
- 33. New NSF CHASE-CI Grant Creates a Community Cyberinfrastructure Adding a Machine Learning Layer Built on Top of the Pacific Research Platform Caltech UCB UCI UCR UCSD UCSC Stanford MSU UCM SDSU NSF Grant for High Speed “Cloud” of 256 GPUs For 30 ML Faculty & Their Students at 10 Campuses for Training AI Algorithms on Big Data See Session 3 Talk by Tom DeFanti
- 34. We are Now Investigating How the PRP Prototype Might Be Extended to National-Scale From the text of the PRP cooperative agreement: After approximately 18 (or TBD) months, a site visit and comprehensive review of progress towards meeting project milestones and goals and overall performance and management processes will take place, including user community relationships, scientific impacts, and the status of the project as a model for potential future national-scale, network-aware, data-focused cyberinfrastructure attributes, approaches, and capabilities.
- 35. PRP is Partnering with the Advanced CyberInfrastructure – Research and Education Facilitators (ACI-REF) NSF Grant to Explore Extension PRP Connected ACI-REF has also spawned the 28- member Campus Research Computing consortium (CaRC), funded by the NSF as a Research Coordination Network (RCN). CaRC is dedicated to sharing best practices, expertise, and resources, enabling the advancement of campus- based research computing activities around the nation. Jim Bottum, Principal Investigator ACI-REF CaRC See Session 1 Talk by Tom Cheatham
- 36. Expanding to the Global Research Platform Via CENIC/Pacific Wave, Internet2, and International Links See Session 5 on GRP PRP PRP’s Current International Partners Korea Shows Distance is Not the Barrier to Above 5Gb/s Disk-to-Disk Performance Netherlands Guam Australia Korea Japan
- 37. Our Support: • US National Science Foundation (NSF) awards CNS 0821155, CNS-1338192, CNS-1456638, CNS-1730158, ACI-1540112, & ACI-1541349 • University of California Office of the President CIO • UCSD Chancellor’s Integrated Digital Infrastructure Program • UCSD Next Generation Networking initiative • Calit2 and Calit2 Qualcomm Institute • CENIC, PacificWave and StarLight • DOE ESnet