Introduction to Software Defined Visualization (SDVis)
1 like1,111 views
This document provides an overview of Intel's Software Defined Visualization (SDVis) initiative and updates on its current status. SDVis aims to enable scalable, flexible visualization that can run on a variety of systems from laptops to large clusters. It utilizes several open source libraries developed by Intel including Embree for ray tracing, OSPRay as a rendering engine, and OpenSWR for rasterization. The document discusses how SDVis addresses challenges of large-scale, high performance visualization. It provides examples of scientific visualization projects using SDVis and performance comparisons of Embree and OSPRay to GPU-based solutions. In addition, the document outlines several active integrations of SDVis technologies in visualization software including ParaView and
Introduction to Software Defined Visualization (SDVis)
- 2. INTEL® HPCDEVELOPERCONFERENCE FUELYOURINSIGHT VISUALIZATIONTRACKWELCOMEANDSDVISUPDATE Intel HPC Developer Conference November 2016
- 3. Outline § The Pendulum of Computing § State of The Union Software Defined Visualization(SDVis) – Quick Refresher – What is SDVis? Why? – We are we today? (Hint: Launched and Active Integrations) § Visualization @ HPCDC and SC’16 Overview § Summary 3
- 4. PENDULUM
- 5. The Challenge with ‘Traditional’ Large Scale Visualization HPC cluster performs modeling and simulations Dedicated Visualization HW (GPUs) and SW Client devices view the final images HW Visualization Using Dedicated hardware and specialized software Bottlenecks… I/O, Scheduling, Memory Size, Power,… Circa 2010 - 2015
- 6. 6 The Pendulum of Computing General Purpose Software + Integration Special Purpose Fixed Function Hybrid
- 7. 7 The Pendulum of Computing Example: Graphics/Visualization – 1970’s->80’s Special Purpose Fixed Function Vector Displays Tektronics Silicon Graphics Gfx Workstations 10’s-100’s of Dev Engs Digital Graphics starting to emerge but in the realm of “specialty” uses General Purpose Software + Integration Hybrid
- 8. 8 The Pendulum of Computing Example: Graphics/Visualization – ~1980’s-90’s General Purpose Software + Integration Special Purpose Fixed Function Hybrid Vector Displays Tektronics Silicon Graphics Gfx Workstations 100’s of SW Engs VGA/SVGA ”Frame Buffers”. Personal Computers DOS-Apps / Games 1000’s SW Engs Limited Standards High SW Innovation Inflection Point: PCs Democratize Computing and Development
- 9. 9 The Pendulum of Computing Example: Graphics/Visualization – ~1990-’95 General Purpose Software + Integration Special Purpose Fixed Function Hybrid Vector Displays Tektronics Silicon Graphics Gfx Workstations 100’s of SW Engs HW Cursor Windowing GUI 2D FB HW Copy Rect Fill; Line Draw Device Drivers 100’s -1000’s SW Engs Inflection Point: GUI Systems + Standards Emerge - Algo’s Mature VGA/SVGA F.B. Personal Computers DOS-Apps / Games 1000’s SW Engs Limited Standards High SW Innovation
- 10. 10 The Pendulum of Computing Example: Graphics/Visualization – ~1995->2000 General Purpose Software + Integration Special Purpose Fixed Function Hybrid Video / Audio Compression Graphics/Video Integrated 3D Fixed Function Pipelines ”High Speed Buses – PCI” 100’s SW Engs Limited Software Innovation HW Cursor Windowing GUI 2D FB HW Copy Rect Fill; Line Draw Inflection Point: Windows 95, DirectX/OpenGL, 3D Gaming VGA/SVGA F.B. Personal Computers DOS-Apps / Games 1000’s SW Engs Limited Standards High SW Innovation
- 11. 11 The Pendulum of Computing Example: Graphics/Visualization – ~2000-‘14 General Purpose Software + Integration Special Purpose Fixed Function Hybrid Video / Audio Compression Graphics/Video Integrated 3D Fixed Function Pipelines ”High Speed Buses – PCI” Shaders; Limited Compute Advanced 3D Texture Modes Integrated M.B. Graphics High Speed RAM (GDDR) 3D Scientific Vis CUDA, Multicore CPU’s 1000’s SW Engs Inflection Point: 3D Gaming “Realism” drives programmable shaders, improved flexibility HPC Compute Adjacency + CUDA + Multicore -> Parallel Computing Emerges VGA/SVGA F.B. Personal Computers DOS-Apps / Games 1000’s SW Engs Limited Standards High SW Innovation
- 12. 12 The Pendulum of Computing Example: Graphics/Visualization – 2014 -> ? General Purpose Software + Integration Special Purpose Fixed Function Hybrid Video / Audio Compression Graphics/Video Integrated 3D Fixed Function Pipelines ”High Speed Buses – PCI” Shaders; Limited Compute Advanced 3D Texture Modes Integrated M.B. Graphics High Speed RAM (GDDR) 3D Scientific Vis CUDA, Multicore CPU’s 1000’s SW Engs A ”New” Inflection Point?: Big Data, I/O Bottlenecks, Power, TCO challenges 8+ cores, SIMD CPUs, On-Package High Bandwidth Memory Multicore and Many-Core CPU’s Integrated CPU Graphics HBW Memory and Caches Big Data High SW innovation (perf+fidelity) SciVis analysis growth 1000’s - 10,000’s+ “
- 13. 13 So what does this mean for Visual Analysis and Workflows?
- 14. Example: EPFL Blue Brain Project “Brayns” + OSPRay Brain Neuron Growth 3D Visualization Web-Based GUI Large Display or Display Wall Output Real-time Simulation of Brain Neuron Formation and Electrical Impulses ~70 GB of Input Data @ 15-20fps; Shown using 4 Knights Landing Processors @ ISC’16 Intel Booth IMPOSSIBLE FOR A TETHERED GPU WITH LIMITED MEMORY AND Bandwidth!
- 15. Example:GR-Chombo Black Hole Collision Stephen Hawking CTC LIGO Black Hole Collision simulation and resulting Gravitational Waves 1200 Timesteps over 0.2s – Collide @ ‘Speed of Light’ 3.6 TB Postprocessed (from 36 TB) Dataset IMPOSSIBLE FOR A TETHERED GPU WITH LIMITED MEMORY AND Bandwidth!
- 17. 17 Refresher – Why SDVis and What is it?
- 18. Our Vision: Scalable, Flexible Vis Rendering that Runs Anywhere! Standalone Laptops or Workstations Big Memory Nodes or Rendering Focused Clusters Large Compute+Vis clusters with Local or Remote Clients Cloud or Network
- 19. How? Intel® SSF and Intel-Supported Software Defined Visualization (SDVis)! Embree - CPU Optimized Ray Tracing Algorithms - ‘Tool kit’ for Building Ray Tracings Apps - Broadly Adopted by 3rd Party ISVs - More at http://embree.github.io OSPRay - Rendering Engine Based on Embree - API Designed to Ease Creation of Visualization Software - More at http://ospray.org OpenSWR - High Performance CPU Vis Rasterization - Fully Integrated into MESA v12.0+ - Supports ParaView, Visit, VTK, EnSight, VL3 - More at http://mesa3d.org Standard OpenGL Image Image Rendered by OSPRay HW Visualization Software Defined Visualization
- 20. Addressing Large-scale, High Performance, and High Fidelity Visualization with SDVis Gain deeper understanding of data impacting science & discovery High fidelity, more realistic images even as data sets become increasingly larger, and more complex; no need to compromise data resolution Solve computing + modeling problem together (in-situ vis) Essential SW development suite that makes concurrent simulation and visualization efficient – users can work interactively and get results quicker One system Use same system for both simulation and visualization, avoid data transfer delays and memory size constraints – faster insights for solving toughest problems
- 21. Your Work. No Compromises. Benefits of SDVis • Open-sourced technology delivering vivid visualization of complex, enormous data sets • Innovative software libraries for visualizing results with high performance by unlocking the parallelism already in your system • High-fidelity images for gaining deeper insights in science and industry, faster • Software Only solution lowers costs – no card cost, no card maintenance, lower power bills Magnetic Reconnection Model, Courtesy Bill Duaghton(LANL) and Berc Geveci(Kitware) Gravational Waves : GR-Chombo AMR Data, Stephen Hawking CTC, UCambridge; Queens College, London; visualization, Carson Brownlee, Intel, ParaView) Ribosome: Data: Max-Planck Institute for Biophysical Chemistry
- 22. Hi-Fidelity Visualization with… § … scalable image quality § … scalable model size § … scalable in rendering cost Data set provided by Florida International University
- 23. OpenSWR Software Rasterizer (www.mesa3d.org www.openswr.org) • High performance open source software implementation of OpenGL* rasterizer ‑ Fully multi-threaded and vectorized for Intel® processors ‑ Can access full system memory - highest resolution data ‑ Leverages community development effort (MESA) • Drop in replacement for OpenGL library • Available since July’16 in Mesa v12.0+ targeting features and performance for leading ScIVis Apps VL3
- 24. Ray Tracing Foundation: Embree Ray Tracing Kernel Library 24 Provides highly optimized and scalable ray tracing kernels § Acceleration structure build and ray traversal § Single Ray, Ray Packets(4,8,16), Ray Streams(N) Targets up to photorealistic professional and scientific rendering applications Highest ray tracing performance on CPUs § 1.5–6× speedup reported by users Support for latest CPUs / ISAs § Intel® Xeon Phi™ Processor (codenamed Knights Landing) – AVX-512 API for easy integration into applications Free and open source under Apache 2.0 license § http://embree.github.com 24
- 25. Performance: Embree vs. NVIDIA* OptiX* (Pascal) 0 5 10 15 20 25 30 35 40 45 50 Bentley (2.3M Tris) Crown (4.8M Tris) Dragon (7.4M Tris) Karst Fluid Flow (8.4M Tris) Power Plant (12.8M Tris) Intel® Xeon® Processor E5-2699 v4 2 x 22 cores, 2.2 GHz Intel® Xeon Phi™ Processor 7250 68 cores, 1.4 GHz NVIDIA TITAN X (Pascal) Coprocessor 12 GB RAM Frames Per Second (Higher is Better), 1024x1024 image resolution Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark* and MobileMark*, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more information go to http://www.intel.com/performance. Embree 2.12.0, ICC 2016 Update 1, Intel® SPMD Program Compiler (Intel® ISPC) 1.9.1 NVIDIA* OptiX* 4.0.1, CUDA* 8.0.44 Source: Intel
- 26. 26 Intel Path Tracer Renderer Sample Application: Path Tracer Renderer using Embree and OptiX* RT Libraries Description: Path tracing app for use in benchmarking RT libraries Availability: § Code: embree.github.io § Recipe: embree.github.io. Usage Model: § TBB, ISPC and Intrinsics Highlights: § The code represents a typical ray tracing rendering pipeline used throughout DCC to show comparative performance on different types of hardware with a variety of input 3D data models. It has been optimized for all Intel ISA;s and delivered to the community and available for download from embree.github.io § End-User benefits: Ability to achieve competitive performance and the flexibility of IA for rendering and render farm applications Results: § Embree on dual socket (44 cores total) Intel® Xeon® E5-2699 v4 Processor performs 20% - 30% faster than Intel® Xeon Phi™ 7250 Processor – Path tracing causes low SIMD utilization (better for Xeon) • Embree on dual socket (44 cores total) Intel® Xeon® E5-2699 v4 Processor more than 2x faster than OptiX on NVIDIA TITAN X (Pascal) GPU. – Path tracing causes low SIMD utilization (better for Xeon) – Large path tracing kernel requires many registers (thus fewer threads executed on GPU) * Other names and brands may be claimed as the property of others. Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. See benchmark tests and configurations in the speaker notes. For more information go to http://www.intel.com/performance 0 5 10 15 20 25 30 35 40 45 50 Bentley (2.3M Tris) Crown (4.8M Tris) Dragon (7.4M Tris) Karst Fluid Flow (8.4M Tris) Power Plant (12.8M Tris) Intel® Xeon® Processor E5-2699 v4 2 x 22 cores, 2.2 GHz Intel® Xeon Phi™ Processor 7250 68 cores, 1.4 GHz NVIDIA TITAN X (Pascal) Coprocessor 12 GB RAM Frames Per Second (Higher is Better), 1024x1024 image resolution
- 27. Diffuse Path Tracing Performance: Embree vs. NVIDIA* OptiX* Prime 0 20 40 60 80 100 120 140 160 180 Mazda (5.7M Tris) Villa (37.7M Tris) Art Deco (10.7M Tris) Power Plant (12.8M Tris) San Miguel (10.5M Tris) Intel® Xeon® Processor E5-2699 v4 2 x 22 cores, 2.2 GHz Intel® Xeon Phi™ Processor 7250 68 cores, 1.4 GHz NVIDIA TITAN X (Pascal) Coprocessor 12 GB RAM Million Rays Per Second (Higher is Better), 3840x2160 image resolution Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark* and MobileMark*, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more information go to http://www.intel.com/performance. Embree 2.12.0, Intel® C++ Compiler 17.0 NVIDIA* OptiX* Prime 4.0.1, CUDA* 8.0.44 Source: Intel
- 28. Embree Adoption* 28 Image rendered with FluidRay RT Rendered with StingRay, SURVICE Engineering Courtesy of Jeff Patton, Rendered with Corona Renderer pCon.planner rendered courtesty EasternGraphics *Many other announced users incl.: Pixar, Weta Digital, Activision, Chaos V-Ray, Ready At Dawn, FrostBite, EpicGames UnReal, High Moon, Blue Sky, UBISoft MP, Framestore,….
- 29. 29 § Build on top of Embree; Launched June 2016 § Scalable Visualization targeted features – Surfaces (both polygonal and non- polygonal) – Volumes, and volume rendering – High-Fidelity rendering/shading methods – Scalable Cluster Wide Rendering § Packed it up in an ‘easy-to-use’ rendering library for visualization – Same "spirit" as OpenGL, but different API VL3 Brayns PowerCT OSPRay: A Ray-Tracing based Rendering Engine for High-Fidelity Visualization NASA
- 31. ParaView v5.2 with integrated OSPRay and OpenSWR 31 • Brain Tumor monitoring and treatment • 3D interactive @ 10-20fps • Intel® Xeon Phi™ processor cluster • Ambient occlusion plus shadows • Stop by the Intel SC’16 booth to see it live! • Data courtesy Kitware. Visualization, Carson Brownlee, Intel
- 32. 32 NASA – Custom OSPRay App • Simulated on Pleiades supercomputer • Rendered on attached ‘hyperwall’ cluster • Dataset: SRB booster separation from SLS Simulation: Jeff Onufer and Tom Pulliam, NASA Ames Visualization: Tim Sandstrom and Pat Moran, NASA Ames
- 33. 33 Stephen Hawking Centre for Theoretical Cosmology – ParaView / VTK with OSPRay Gravational Waves : GR-Chombo AMR Data, Stephen Hawking CTC, UCambridge; Queens College, London; visualization, Carson Brownlee, Intel) • 600 GB Memory Footprint • 36 TB Simulation Data Set • 4 Intel® Xeon Phi™ 7230 Processors • 1 Intel® Xeon® E5 v4 Dual Socket node • Intel® Omni-Path Fabric • ~10 fps • See a demo in the SC’16 Intel “Discovery Zone”
- 34. 34 Stephen Hawking Centre for Theoretical Cosmology – ‘Walls’ in situ with OSPRay Rendering • 10 TB Memory Footprint • SGI UV-300 16TB SMP • >1000 Shared memory Intel® Xeon® E5 v3 processors • ~15 fps • Domain Wall formation in the universe from Big Bang to today (13.8 billion years) • Simulation code by Shellard et al, Visualizaiton by Johannes Gunther (Intel)
- 35. 35 Argonne VL3 Distributed Volume Renderer VL3 with Mesa v13.0 with OpenSWR VL3 Compositing with OSPRay Visualizations: Silvio Rizzi, Joe Insley: Argonne; Aaron Knoll: SCI @ UUtah
- 37. SDVis Track Schedule (SATURDAY) 37 Technical Sessions SW Visualization (Powder Mountain) Lab Sessions Lab Room 3 (Capacity 50) Start End Technical Sessions Start End Hands On Lab 12:00 PM 1:00 PM Registration Opens 1:00 PM 1:50 PM Welcome Kick Off 1:50 PM 2:05 PM Break 2:05 PM 2:55 PM Talk 1 - SDVis Update (Jim Jeffers) Talk 2 – OpenSWR Update (Jeff Amstutz) 2:05 PM 3:30 PM 2:55 PM 3:10 PM Break 3:10 PM 4:00 PM Talk 1 - OSPRay 1.0 and Beyond (Jeff A, Intel) Talk 2 - MPI Data-Parallel Rendering w/OSPRay (Carson B, Intel) 3:30 PM 3:40PM Break 3:40PM 4:30PM Software Defined Visualization : Getting the most out of ParaView OSPRay (Paul A. Navrátil & David E. DeMarle, Kitware) 4:00 PM 4:15 PM Break 4:15 PM 5:05 PM Talk 1 - Realizing Multi-Hit Ray Tracing in Embree and OSPRay (Christiaan Gribble, Intel/SURVICE) Talk 2 - Visualization w/Visit on Knights Landing (Jian Huang & Hank Childs, UOregon / UTennessee) 4:30PM 5:05PM
- 38. SDVis Track Schedule (SUNDAY) 38 Technical Sessions SW Visualization (Powder Mountain) Lab Sessions Lab Room 2 (Capacity 25) 7:00 AM 9:00 AM Registration andBreakfast 8:45 AM 9:30 AM Keynote 9:30 AM 9:45 AM Break 9:45 AM 10:35 AM Talk 1 - SDVis Efforts @ Intel® PCC Aaron Knoll, Unv. Of Utah) Talk 2 - OSPRay Integration into Pcon-Planner (Caglar Özgür & Frank Wicht, Eastern Graphics) 9:45 AM 10:35 AM Software Defined Visualization : Getting the most out of ParaView OSPRay (Kitware) 10:35 AM 10:50 AM Break 10:35 AM 11:25AM 10:50 AM 11:40 AM Bio-Molecular Vis on Knight Landing (John Stone, UIUC) 11:25 AM 11:40AM Break 11:40 AM 1:00 PM Lunch time Panel 11:40 AM 1:00 PM Lunch time Panel 1:00 PM 1:50 PM Paraview & VTK w/OSPRay and OpenSWR (David DeMarle, Kitware) 1:00 PM 2:00PM 1:50 PM 2:05 PM Break 2:05 PM 2:55PM SDVIs and In-Situ Visualization on TACC's Stampede (Paul Navratil) 2:00PM 2:40 AM 2:40AM 2:50PM Break 2:55 PM 3:10 PM Break 2:50PM 4:00 PM 3:10 PM 4:00 PM Live Demos and Open Discussion on Software Defined Visualization (All Vis Track Presenters) 4:00 PM 4:15 PM Break 4:15 PM 4:45 PM Closing Keynote 7:00 PM 10:00 PM Intel® Networking Reception
- 39. SC’16 Software Defined Visualization Demos Intel Main Booth (#1819): Intel® SSF Cluster (Intel® Xeon Phi™ Processors, Intel Xeon® v4 Processors, Intel® Omni- Path Fabric, Intel® HPC Orchestrator, Intel® Lustre • 1) ParaView v5.2 w/OSPRay&OpenSWR: Brain Tumor Analysis • 2) VMD v1.9.x w/OSPRay: Cryo-EM Reconstruction with ROME • 3) VMD v1.9.x w/OSPRay: LAMMPS for Cancer Research Intel Discover Zone (#2121) – Intel® Xeon Phi™ Processor DAPs • Argonne VL3 w/OSPRay: HACC Dark Matter Analysis • ParaView v5.2 w/OSPRay: Stephen Hawking CTC – Ligo based Black Hole collision Partner Booths • Dell, SuperMicro, Kitware, NASA, Univ of Utah, NCSA, … 39
- 40. SCI-X Open House Univ. of Utah 40 Weds Nov. 16 1:00 p.m. -‐ 7:00 p.m. 1-‐5 p.m.: Open House: Cont. Buses between the Salt Palace and the University (10 minutes each way). 5 p.m.: Keynote presentaKon by Jim Clark -‐ Warnock Engineering Building L104 (overflow -‐ WEB 2230) 6:00 p.m. -‐ RecepKon -‐ Catmull Gallery
- 41. Summary: Software Defined Visualization www.sdvis.org • Addresses ever-growing HPC challenges for data size, flexibility, reliability and maintainability • OpenSWR, OSPRay and Embree rendering libraries optimize use of CPUs and main memory • Integrating into prominent Vis tools, ParaView*, VisIt, EnSight*, VMD, Brayns, VL3, and more …. • All freely available (Open Source), developed and maintained by Intel SDVis = Performance, Fidelity and Lower Cost!!
- 42. QUESTIONS?
- 45. Legal Notices and Disclaimers Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Performance varies depending on system configuration. No computer system can be absolutely secure. Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. Copyright © 2016 Intel Corporation. All rights reserved. Intel, Intel Inside, the Intel logo, Intel Xeon and Intel Xeon Phi are trademarks of Intel Corporation in the United States and other countries. *Other names and brands may be claimed as the property of others. Copyright © 2016 Intel Corporation, All Rights Reserved 45