Swift Parallel Scripting for High-Performance Workflow

Swift Parallel Scripting for
High-Performance Workflow
April 16, 2015
Michael Wilde wilde@anl.gov
Daniel S. Katz dsk@uchicago.edu
http://swift-lang.org

Domain of Interest
“Time”
“Complexity”
Increasing capabilities in computational science

Workflow needs
 Application Drivers
– Applications that are many-task in nature: parameters sweeps, UQ,
inverse modeling, and data-driven applications
– Analysis of capability application outputs
– Analysis of stored or collected data
– Increase productivity at major experiment facilities (light source, etc.)
– Urgent computing
– These applications are all many-task in nature
 Requirements
– Usability and ease of workflow expression
– Ability to leverage complex architecture of HPC and HTC systems
(fabric, scheduler, hybrid node and programming models), individually
and collectively
– Ability to integrate high-performance data services and volumes
– Make use of the system task rate capabilities from clusters to
extreme-scale systems
 Approach
– A programming model for programming in the large
3

When do you need HPC workflow?
Example application: protein-ligand docking for drug screening
(B)
O(100K)
drug
candidates
…then hundreds of
detailed MD
models to find
10-20 fruitful
candidates for
wetlab & APS
crystallography
O(10) proteins
implicated in a disease
= 1M
docking
tasks…
X
…

5
For protein docking workflow:
foreach p, i in proteins {
foreach c, j in ligands {
(structure[i,j], log[i,j]) =
dock(p, c, minRad, maxRad);
}
scatter_plot = analyze(structure)
To run:
swift –site tukey,blues dock.swift
Expressing this many task workflow in Swift

The Swift runtime system has drivers and algorithms to efficiently support and aggregate diverse runtime environments
Swift enables execution of simulation campaigns across
multiple HPC and cloud resources
6
Local datamarkApps
Swift host: login node, laptop, …
Scripts
Data servers
Data servers
Data servers
Campus systems
Cloud resources
Petascale systems
National infrastructure

Swift in a nutshell
 Data types
string s = “hello world”;
int i = 4;
int A[];
 Mapped data types
type image;
image file1<“snapshot.jpg”>;
 Mapped functions
app (file o) myapp(file f, int i)
{ mysim "-s" i @f @o; }
 Conventional expressions
if (x == 3) {
y = x+2;
s = @strcat(“y: ”, y);
}
 Structured data
image A[]<array_mapper…>;
 Loops
foreach f,i in A {
B[i] = convert(A[i]);
}
 Data flow
analyze(B[0], B[1]);
analyze(B[2], B[3]);
Swift: A language for distributed parallel scripting, J. Parallel Computing, 2011

Pervasively parallel
 Swift is a parallel scripting system for grids, clouds and clusters
 F() and G() are computed in parallel
– Can be Swift functions, or leaf tasks (executables or scripts in shell, python, R,
Octave, MATLAB, ...)
 r computed when they are done
 This parallelism is automatic
 Works recursively throughout the program’s call graph
(int r) myproc (int i)
{
int f = F(i);
int g = G(i);
r = f + g;
}

Data-intensive example:
Processing MODIS land-use data
analyze
colorize
x 317
landUse
x 317
mark
Swift loops process hundreds of images in parallel
assemble
Image processing pipeline for land-use data from the MODIS satellite instrument…

foreach raw,i in rawFiles {
land[i] = landUse(raw,1);
colorFiles[i] = colorize(raw);
}
(topTiles, topFiles, topColors) =
analyze(land, landType, nSelect);
gridMap = mark(topTiles);
montage =
assemble(topFiles,colorFiles,webDir);

Example of Swift’s implicit parallelism:
analyze
colorize
x 317
landUse
x 317
mark
Swift loops process hundreds of images in parallel
assemble
Image processing pipeline for land-use data from the MODIS satellite instrument…

Swift provides 4 important benefits:
13
Makes parallelism more transparent
Implicitly parallel functional dataflow programming
Makes computing location more transparent
Runs your script on multiple distributed sites and
diverse computing resources (desktop to petascale)
Makes basic failure recovery transparent
Retries/relocates failing tasks
Can restart failing runs from point of failure
Enables provenance capture
Tasks have recordable inputs and outputs

Swift/T: productive extreme-scale scripting
 Script-like programming with “leaf” tasks
– In-memory function calls in C++, Fortran, Python, R, … passing in-memory objects
– More expressive than master-worker for “programming in the large”
– Leaf tasks can be MPI programs, etc. Can be separate processes if OS permits.
 Distributed, scalable runtime manages tasks, load balancing, data movement
 User function calls to external code run on thousands of worker nodes
Swift
control
process
Swift
control
process
Parallel
evaluator
and
data store
Swift worker process
C C++ Fortran
C C++ Fortran
C C++ Fortran
MPI
Scripts

Parallel tasks in Swift/T
 Swift expression: z = @par=32 f(x,y);
 ADLB server finds 8 available workers
– Workers receive ranks from ADLB server
– Performs comm = MPI_Comm_create_group()
 Workers perform f(x,y)communicating on comm

LAMMPS parallel tasks
 LAMMPS provides a
convenient C++ API
 Easily used by Swift/T
parallel tasks
foreach i in [0:20] {
t = 300+i;
sed_command = sprintf("s/_TEMPERATURE_/%i/g", t);
lammps_file_name = sprintf("input-%i.inp", t);
lammps_args = "-i " + lammps_file_name;
file lammps_input<lammps_file_name> =
sed(filter, sed_command) =>
@par=8 lammps(lammps_args);
}
Tasks with varying sizes packed into big MPI run
Black: Compute Blue: Message White: Idle

Swift/T-specific features
 Task locality: Ability to send a task to a process
– Allows for big data –type applications
– Allows for stateful objects to remain resident in the workflow
– location L = find_data(D);
int y = @location=L f(D, x);
 Data broadcast
 Task priorities: Ability to set task priority
– Useful for tweaking load balancing
 Updateable variables
– Allow data to be modified after its initial write
– Consumer tasks may receive original or updated values when they emerge
from the work queue
17
Wozniak et al. Language features for scalable distributed-memory
dataflow computing. Proc. Dataflow Execution Models at PACT, 2014.

Swift/T: scaling of trivial foreach { } loop
100 microsecond to 10 millisecond tasks
on up to 512K integer cores of Blue Waters
18

Large-scale applications using Swift
 Simulation of super-
cooled glass materials
 Protein and biomolecule
structure and interaction
 Climate model analysis and
decision making for global
food production & supply
 Materials science at the
Advanced Photon Source
 Multiscale subsurface
flow modeling
 Modeling of power grid
for OE applications
All have published science
results obtained using
Swift
E
C
A
B
A
B
C
D
E
F
F
D

Assess
Red indicates higher statistical
confidence in data
Impact and Approach Accomplishments ALCF Contributions
• HEDM imaging and analysis
shows granular
material structure, of
non-destructively
• APS Sector 1 scientists use
Mira to process data from live
HEDM experiments, providing
real-time feedback to correct
or improve in-progress
experiments
• Scientists working with
Discovery Engines LDRD
developed new Swift analysis
workflows to process APS data
from Sectors 1, 6, and 11
• Mira analyzes experiment in
10 mins vs. 5.2 hours on APS
cluster: > 30X improvement
• Scaling up to ~ 128K cores
(driven by data features)
• Cable flaw was found and
fixed at start of experiment,
saving an entire multi-day
experiment and valuable user
time and APS beam time.
• In press: High-Energy Synchrotron X-
ray Techniques for Studying Irradiated
Materials, J-S Park et al, J. Mat. Res.
• Big data staging with MPI-IO for
interactive X-ray science, J Wozniak et
al, Big Data Conference, Dec 2014
• Design, develop, support, and trial
user engagement to make Swift
workflow solution on ALCF
systems a reliable, secure and
supported production service
• Creation and support of the Petrel
data server
• Reserved resources on Mira for
APS HEDM experiment at Sector
1-ID beamline (8/10/2014 and
future sessions in APS 2015 Run 1)
Boosting Light Source Productivity with Swift ALCF Data Analysis
H Sharma, J Almer (APS); J Wozniak, M Wilde, I Foster (MCS)
Analyze
Fix
Re-analyze
Valid
Data!
2 3
4
5
1

Conclusion: parallel workflow scripting is practical,
productive, and necessary, at a broad range of scales
 Swift programming model demonstrated feasible and
scalable on XSEDE, Blue Waters, OSG, DOE systems
 Applied to numerous MTC and HPC application domains
– attractive for data-intensive applications
– and several hybrid programming models
 Proven productivity enhancement in materials,
genomics, biochem, earth systems science, …
 Deep integration of workflow in progress at XSEDE, ALCF
Workflow through implicitly parallel dataflow is
productive for applications and systems at many scales,
including on highest-end system

What’s next?
 Programmability
– New patterns ala Van Der Aalst et al (workflowpatterns.org)
 Fine grained dataflow – programming in the smaller?
– Run leaf tasks on accelerators (CUDA GPUs, Intel Phi)
– How low/fast can we drive this model?
 PowerFlow
– Applies dataflow semantics to manage and reduce energy usage
 Extreme-scale reliability
 Embed Swift semantics in Python, R, Java, shell, make
– Can we make Swift “invisible”? Should we?
 Swift-Reduce
– Learning from map-reduce
– Integration with map-reduce

GeMTC: GPU-enabled Many-Task Computing
Goals:
1) MTC support 2) Programmability
3) Efficiency 4) MPMD on SIMD
5) Increase concurrency to warp level
Approach:
Design & implement GeMTC middleware:
1) Manages GPU 2) Spread host/device
3) Workflow system integration (Swift/T)
Motivation: Support for MTC on all accelerators!
S. J. Krieder, J. M. Wozniak, T. Armstrong, M. Wilde, D. S. Katz, B. Grimmer, I. T. Foster, I. Raicu, "Design and Evaluation of
the GeMTC Framework for GPU-enabled Many-Task Computing,” HPDC'14

Further research directions
 Deeply in-situ processing for extreme-scale analytics
 Shell-like Read-Evaluate-Print Loop ala iPython
 Debugging of extreme-scale workflows
Deeply in-situ analytics of a
climate simulation

25
U . S . D E P A R T M E N T O F
ENERGY
Swift gratefully acknowledges support from:
http://swift-lang.org

Swift Parallel Scripting for High-Performance Workflow

More Related Content

What's hot (10)

Similar to Swift Parallel Scripting for High-Performance Workflow (20)

More from Daniel S. Katz (20)

Recently uploaded (20)

Swift Parallel Scripting for High-Performance Workflow