Your interactive computing
0 likes120 views
The document discusses using Python as an interactive computing platform for tasks like data analysis, visualization, workflow automation, and parallel computing. It provides examples of Python libraries and tools for tasks like interactive notebooks, 2D/3D plotting, data manipulation, command line parsing, concurrency, and connecting to cloud services. It also covers strategies for high performance computing with Python, leveraging C/C++ libraries, and approaches for building platforms with Python.
Your interactive computing
- 1. Your Own Interactive Computing Platform 2017/10/16 at TKU Yung-Yu Chen https://www.linkedin.com/in/yungyuc/
- 2. Computing for Everything ❖ Solve problems that can only be solved by computers. ❖ Save resources, shorten cycles, better this world. ❖ Let machines repeat it.
- 3. DeepMind AI Reduces Google Data Centre Cooling Bill by 40% ❖ https://deepmind.com/blog/deepmind-ai-reduces- google-data-centre-cooling-bill-40/
- 4. Black Hole Simulation https://go.nasa.gov/2xhd5xD
- 5. Supersonic Jet in Cross Flow density contours
- 6. Optical Proximity Correction Optical proximity correction for semiconductor manufacturing https://commons.wikimedia.org/wiki/File:Optical_proximity_correction_structures.svg
- 8. Python as Your Computing Platform ❖ Interactive analysis and visualization. ❖ Data preparation and processing. ❖ Workflow automation. ❖ Distributed processing and parallel computing.
- 9. Interactive ❖ http://jupyter.org: run Python everywhere and code it through browser. ❖ https://notebooks.azure.com: Azure sets it up for you already ❖ https://matplotlib.org: de facto 2D plotting library ❖ https://www.vtk.org: versatile 3D visualization toolbox ❖ https://www.paraview.org: if you only want a frontend
- 10. Data Manipulation ❖ http://pandas.pydata.org: de facto tool for data analytics ❖ Comma-Separated Value: “csv” standard module. ❖ http://www.pytables.org: HDF5 hierarchical data access. ❖ http://unidata.github.io/netcdf4-python/: netCDF, yet another data storage based on HDF5.
- 11. Workflow Automation ❖ Offload routine to computers. ❖ https://docs.python.org/3/library/argparse.html: standard command-line argument processing ❖ https://github.com/saltstack/salt: cloud-oriented automation for management and configuration ❖ AWS, GCE, Azure all offer SDK for Python. ❖ Lead to system admin and/or devop.
- 12. Concurrency ❖ https://docs.python.org/3/library/asyncio.html ❖ support native asynchronous constructs ❖ https://docs.python.org/3/library/multiprocessing.html ❖ parallel computing and distributed processing using multiple processes ❖ http://zeromq.org/bindings:python ❖ socket communication ❖ http://pythonhosted.org/mpi4py/ ❖ use MPI in Python
- 13. Interact
- 14. How to Learn ❖ Download Anaconda and set it up: https:// www.anaconda.com/download ❖ Share your notebooks at Anaconda Cloud: https:// anaconda.org ❖ Free for public projects
- 15. WWI. source: https://www.youtube.com/watch?v=K0Wp7Y3Tbiw HPC (high-performance computing) is hard. Physics is harder. Don’t rely on the Schlieffen Plan.
- 16. Optimization ❖ Memory access is expensive. ❖ Branching may be expensive too. ❖ Think like a machine: see through high-level source code all the way to assembly. ❖ If you don’t write your own compiler, learn C++. “There are only two hard things in Computer Science: cache invalidation and naming things.” Phil Karlton
- 17. HPC Architecture ❖ Scientific computing takes tremendous computing power. We are interested in big problems. ❖ Some may be divided to smaller, self-contained sub- problems, e.g., data analytics. ❖ Some are unavoidably big. A reasonably big problem may use thousands of CPU cores for days. ❖ Accelerators like GPGPU sometimes speed up, but at a cost of complicated code.
- 18. Python for Building Platform ❖ It's impossible to get it right the first time. ❖ Architecture design takes many iterations. ❖ Python allows quick prototyping. ❖ There is almost always a package for you. ❖ Python is either the best or the second best language for anything.
- 19. NumPy ❖ N-dimensional array (ndarray). ❖ ndarray is typed and offers very fast speed. Oftentimes faster than naive C code. ❖ Efficient data storage and flexible access to memory. ❖ Linear algebra (MKL is supported), FFT, etc. ❖ SciPy: application-specific toolbox.
- 20. Python Is Designed for C/C++ ❖ Everything may be replaced by C/C++. ❖ Python is a C library for a dynamically-typed runtime. ❖ Python is slow, but using Python makes the whole HPC system faster. ❖ Performance hotspots. ❖ High-level abstraction in low-level code. ❖ Plain C: Python C API or Cython. ❖ C++: pybind11 (C++11) or boost.python (pre-C++11). ❖ Fortran: f2py (part of numpy).
- 21. Two Types of Platform ❖ Top-down: lay out everything in Python and replace hotspots using C/C++ ❖ Pro: Fast development. Reach results early. ❖ Con: Python dynamicity weakens robustness. ❖ Bottom-up: lay out core parts in C++ and glue in Python ❖ Pro: Highly robust (if coded right.) ❖ Con: Hard to get it right and take long time to code. ❖ Equally high-performance. Python scripts work as input files.
- 22. Beyond the Concepts: Real Work ❖ What problems do you want to solve? ❖ Software skills: ❖ Coding and HPC: Python, C, C++. ❖ Testing. ❖ Version control. ❖ Operation and automation, including building system. ❖ Debugging. ❖ Documenting.