Parsl: Pervasive Parallel Programming in Python
Download as PPTX, PDF1 like697 views
The document summarizes Parsl, a Python library for pervasive parallel programming. Parsl allows users to naturally express parallelism in Python programs and execute tasks concurrently across different computing platforms while respecting data dependencies. It supports various use cases from small machine learning workloads to extreme-scale simulations involving millions of tasks and thousands of nodes. Parsl provides simple, scalable, and flexible parallel programming while hiding complexity of parallel execution.
![Expressing a many task workflow in Parsl
1) Wrap the science applications as Parsl Apps:
@bash_app
def simulate(outputs=[]):
return './simulation_app.exe {outputs[0]}’
@bash_app
def merge(inputs=[], outputs=[]):
i = inputs; o = outputs
return './merge {1} {0}'.format(' '.join(i), o[0])
@python_app
def analyze(inputs=[]):
return analysis_package(inputs)](https://image.slidesharecdn.com/parsl-standrews-191018195037/85/Parsl-Pervasive-Parallel-Programming-in-Python-12-320.jpg)
![Expressing a many task workflow in Parsl
2) Execute the parallel workflow by calling Apps:
sims = []
for i in range (nsims):
sims.append(simulate(outputs=['sim-%s.txt' % i]))
all = merge(inputs=[i.outputs[0] for i in sims],
outputs=['all.txt'])
result = analyze(inputs=[all.outputs[0]])
simulate simulate simulate
…
merge
analyze
sim-1.txt sim-2.txt sim-N.txt
all.txt](https://image.slidesharecdn.com/parsl-standrews-191018195037/85/Parsl-Pervasive-Parallel-Programming-in-Python-13-320.jpg)
Parsl: Pervasive Parallel Programming in Python
- 1. Parsl: Pervasive Parallel Programming in Python 18 October 2019 Daniel S. Katz ([email protected], http://danielskatz.org, @danielskatz) Assistant Director for Scientific Software & Applications, NCSA Research Associate Professor, CS, ECE, iSchool Parsl Team: Y. Babuji, A. Woodard, Z. Li, D. S. Katz, B. Clifford, R. Kumar, L. Lacinski, R. Chard, J. M. Wozniak, I. Foster, M. Wilde, K. Chard
- 2. Supporting composition and parallelism in Python Software is increasingly assembled rather than written • High-level language (e.g., Python) to integrate and wrap components from many sources Parallel and distributed computing is no longer a niche area • Increasing data sizes combined with plateauing sequential processing power • Parallel hardware (e.g., accelerators) and distributed computing systems Parsl allows for the natural expression of parallelism in such a way that programs can express opportunities for parallelism that can then be realized, at execution time, using different execution models on different parallel platforms
- 3. Traditional workflow • A set of tasks and dependencies between them • Perhaps expressed as data structure, e.g. graph (DAG or cyclic) • How is this different than a procedural computer program? • At the level of a set of instructions: • Dependencies are often explicit, perhaps like a compiled intermediate representation • Tasks are much longer (running time O(sec) – O(hr)) • At level of set of functions: • Tasks as more well-defined (inputs, outputs) • Tasks are often longer (running time O(sec) – O(hr)) https://danielskatzblog.wordpress.com/2018/01/08/expressing-workflows-as-code-vs-data/ https://danielskatzblog.wordpress.com/2019/02/05/using-workflows-expressed-as-code-and-workflows-expressed-as-data-together/
- 4. Traditional workflow expression • Why express a workflow differently than a program? • Program (script) is a natural way of expressing a workflow • Easy to understand, easy to change • Examples: shell scripts, programs in Parsl • Parsl: functions used to identify components • Expressing it as data corresponds to the compiled (assembly) version of the workflow • Maybe easier to execute? • Perhaps easier to reproduce? https://danielskatzblog.wordpress.com/2018/01/08/expressing-workflows-as-code-vs-data/ https://danielskatzblog.wordpress.com/2019/02/05/using-workflows-expressed-as-code-and-workflows-expressed-as-data-together/
- 5. Workflow as code vs workflow as data • Goes back to workflow lifecycle concept • Workflows follow a cycle: • Experimentation/exploration phase (scientific hacking): workflow is extension of thought processes of workflow maker • Productization/ dissemination phase: developer (or someone else) prepares for wider and repeated use by documentation and optimization then disseminates • This use by others can simply be use, or it can be further development. • Different types of users have different needs • Experts want to be able to do as much as possible • Other users trade away complex features for simpler user interface C. Wroe, C. Goble, et al., “Recycling workflows and services through discovery and reuse,” CCPE v19, pp. 181-194, 2007. doi:10.1002/cpe.1050
- 6. Representative Parsl Use Cases Input Output DLHub SwiftSeq LSST-DESC
- 7. DESC image simulation Catalog Simulator Image Simulator Atmosphere , Telescope, Camera... LSST Data Management Stack Fake Observations!SCIENCE! NASA/JPL- Caltech/ESO /R. Hurt HSC Project / NAOJ LSST Project Credit: Antonio Villarreal
- 8. ImSim workflow Bundler 189 sensors x ~10,000s of instance catalogs Node-sized bundles (64 tasks each) JSON description Catalog 1 189 tasks x 4000 nodes Parsl Extreme Scale Executor x 189 x ~10,000s 256K cores for 3 days Catalog 2 Catalog 3 128K cores for 3.5 days x 2000 nodes
- 9. Representative Parsl use cases DLHub Machine Learning Inference SwiftSeq DNA Sequence Analysis LSST-DESC Simulated Sky Survey O(Tasks) Thousands Thousands Millions O(Nodes) Tens Hundreds Thousands O(Duration) Milliseconds-Seconds Hours-Days Hours-Day Pattern Bag-of-tasks Dataflow Dataflow Requirements Low latency bounds High throughput Extreme scale
- 10. Parsl Basics
- 11. Parsl: Interactive parallel programming in Python Apps define opportunities for parallelism • Python apps call Python functions • Bash apps call external applications Apps return “futures”: a proxy for a result that might not yet be available Apps run concurrently respecting data dependencies. Natural parallel programming! Parsl scripts are independent of where they run. Write once run anywhere! pip install parsl Try parsl via binder at bottom left of http://parsl-project.org
- 12. Expressing a many task workflow in Parsl 1) Wrap the science applications as Parsl Apps: @bash_app def simulate(outputs=[]): return './simulation_app.exe {outputs[0]}’ @bash_app def merge(inputs=[], outputs=[]): i = inputs; o = outputs return './merge {1} {0}'.format(' '.join(i), o[0]) @python_app def analyze(inputs=[]): return analysis_package(inputs)
- 13. Expressing a many task workflow in Parsl 2) Execute the parallel workflow by calling Apps: sims = [] for i in range (nsims): sims.append(simulate(outputs=['sim-%s.txt' % i])) all = merge(inputs=[i.outputs[0] for i in sims], outputs=['all.txt']) result = analyze(inputs=[all.outputs[0]]) simulate simulate simulate … merge analyze sim-1.txt sim-2.txt sim-N.txt all.txt
- 14. Decomposing dynamic parallel execution into a task-dependency graph Parsl
- 15. Parsl scripts are execution provider independent The same script can be run locally, on grids, clouds, or supercomputers Growing support for various schedulers and cloud vendors From Parsl docs
- 16. Separation of code and execution Choose execution environment at runtime. Parsl will direct tasks to the configured execution environment(s).
- 17. Authentication and authorization Authn/z is hard… • 2FA, X509, GSISSH, etc. Integration with Globus Auth to support native app integration for accessing Globus (and other) services Using scoped access tokens, refresh tokens, delegation support
- 18. Transparent (wide area) data management Implicit data movement to/from repositories, laptops, supercomputers Globus for third-party, high performance and reliable data transfer • Support for site-specific DTNs HTTP/FTP direct data staging parsl_file = File(globus://EP/path/file) www.globus.org
- 20. Different types of scientific workloads High-throughput workloads • Protein docking, image processing, materials reconstructions • Requirements: 1000s of tasks, 100s of nodes, reliability, usability, monitoring, elasticity, etc. Extreme-scale workloads • Cosmology simulations, imaging the arctic, genomics analysis • Requirements: millions of tasks, 1000s of nodes (100,000s cores) Interactive and real-time workloads • Materials science, cosmic ray shower analysis, machine learning inference • Requirements: 10s of nodes, rapid response, pipelining
- 21. Different types of execution High-throughput executor (HTEX) • Pilot job-based model with multi-threaded manager deployed on workers • Designed for ease of use, fault-tolerance, etc. • <2000 nodes (~60K workers), ms tasks, task duration/nodes > 0.01 Extreme-scale executor (EXEX) • Distributed MPI job manages execution. Manager rank communicates workload to other worker ranks directly • Designed for extreme scale execution on supercomputers • >1000 nodes (>30K workers), ms tasks, >1 m task duration Low-latency Executor (LLEX) • Direct socket communication to workers, fixed resource pool, limited features • 10s nodes, <1M tasks, <1m tasks
- 22. Short tasks scale to thousands of workers Strong scaling: 50,000 tasks submitted with increasing number of workers * Fireworks only 5,000 tasks HTEX and EXEX outperform other Python-based approaches when >256 workers Other approaches are limited to fewer than 128 nodes; HTEX and EXEX continue to scale 0s tasks 1s tasks
- 23. Executors scale to 2M tasks/256K workers 0s tasks 1s tasks Weak scaling: 10 tasks per worker HTEX and EXEX again outperform other Python-based approaches up to ~2M tasks HTEX and EXEX scale to 2K nodes (~65k workers) and 8K nodes (~262K workers), respectively, with >1K tasks/s
- 24. Parsl executors can provide low latency • LLEX achieves low (3.47ms) and consistent latency • HTEX (6.87ms) and EXEX (9.83) are less consistent
- 25. Scalability summary • EXEX scales to over 250,000 workers across 8,000 nodes • Both EXEX and HTEX deliver ~1200 tasks/s • LLEX achieves an average latency of 3.47 ms with tight bounds Framework Max. number of workers Max. number of nodes Max tasks/sec Parsl-IPP 2048 64 330 Parsl-HTEX 65 536 2048 1181 Parsl-EXEX 262 144 8192 1176 FireWorks 1024 32 4 Dask distributed 4096 128 2617
- 27. Interactive supercomputing in Jupyter notebooks
- 29. DOE Distributed Computing & Data Ecosystem (DCDE) • A DOE group is identifying best practices and research challenges to create and operate a DOE/SC wide federated Distributed Computing & Data Ecosystem (DCDE) • Future Lab Computing Working Group (FLC-WG) • Initially working towards a pilot • Using OAuth, working with Globus • Test deployment at BNL • Parsl is part of this effort, via initial work in linking ORNL and BNL • We’ve added support for an OAuthSSHChannel • Now being tested on test deployment
- 30. Multi-site execution 1.Loading Parsl configuration triggers: a) Creation of SSH channels b) Deployment of an interchange process onto login nodes c) Submission of pilot jobs that will connect to the interchange 2.Parsl submits tasks directly to interchange 3.Parsl uses Globus to stage data Interchange Interchange Parsl
- 31. Multi-site execution Too much small code See demo instead https://bit.ly/2Wsjlep (code in https://github.com/Parsl/demo_multifacility)
- 32. Other functionality provided by Parsl Globus. Delegated authentication and wide area data management Fault tolerance. Support for retries, checkpointing, and memoization Containers. Sandboxed execution environments for workers and tasks Data management. Automated staging with HTTP, FTP, and Globus Multi site. Combining executors/providers for execution across different resources Elasticity. Automated resource expansion/retraction based on workload Monitoring. Workflow and resource monitoring and visualization Reproducibility. Capture of workflow provenance in the task graph Jupyter integration. Seamless description and management of workflows Resource abstraction. Block-based model overlaying different providers and resources
- 33. Summary
- 34. Parsl’s parallelism in Python • Simple: minimal new constructs • Safe: deterministic parallel programs through immutable input/output objects, dependency task graph, etc. • Scalable: efficient execution from laptops to the largest supercomputers • Flexible: programs composed from existing components and then applied to different resources/workloads
- 36. Questions? http://parsl-project.org U . S . D E P A R T M E N T O F ENERGY