The genesis of clusterlib - An open source library to tame your favourite supercomputer

The genesis of clusterlib
An open source library to tame your
favourite supercomputer
Arnaud Joly
May 2015, Phd hour discussions

Use case for the birth of clusterlib
Solving supervised learning tasks
The goal of supervised learning is to learn a function from
input-output pairs in order to predict the output for any new
input.
A supervised learning task
1.5 1.0 0.5 0.0 0.5 1.0 1.5
1.5
1.0
0.5
0.0
0.5
1.0
1.5
A learnt function
1.5 1.0 0.5 0.0 0.5 1.0 1.5
1.5
1.0
0.5
0.0
0.5
1.0
1.5

Time is running out
A huge set of tasks and a practical upper bound
O(#datasets×#algorithms×#parameters) ≤ Time before deadline
Embarrassingly parallel tasks.

Supercomputer = scheduler + workers
Supercomputer = cluster of computers

CECI is in !
Thanks to the CECI1 at the University of Liège, we have access
to
7 supercomputers (or clusters of computers);
≈ 20 000 cores;
>60 000 GB ram;
12 high performance scientiﬁc GPU.
1
The CECI (http: // www. ceci-hpc. be/ ) is a supercomputer
consortium funded by the FNRS.

A glimpse to the SLURM scheduler user interface
How to sumit a job?
First, we need to write a bash ﬁle job.sh which speciﬁes
resource requirements and calls the program.
#!/bin/bash
#SBATCH --job-name=job-name
#SBATCH --time=10:00
#SBATCH --mem=1000
srun hostname
Then you can launch the job in the queue
$ sbatch job.sh
How to launch easily many jobs?

With clusterlib, how to launch easily many jobs??
Let’s generate jobs submission command on the ﬂy!
>>> from clusterlib.scheduler import submit
>>> script = submit(job_command="srun hostname",
... job_name="test",
... time="10:00",
... memory=1000,
... backend="slurm")
>>> print(script)
echo ’#!/bin/bash
srun hostname’ | sbatch --job-name=test --time=10:00 --mem=1000
>>> # Let’s launch the job
>>> os.system(script)

A glimpse to the SLURM scheduler user interface
How to check if a job is running?
To check if a job is running, you can use the squeue command.
$ squeue -u ‘whoami‘
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
1225128 defq job-9999 someone PD 0:00 1 (Priority)
1225129 defq job-9998 someone PD 0:00 1 (Priority)
...
1224607 defq job-0003 someone R 7:39:16 1 node025
How to avoid launching running, queued or completed jobs?

With clusterlib, how to avoid re-launching ... ?
... running and queued jobs?
Let’s give to jobs a unique name, then let’s retrieve the names
of running / queued jobs.
>>> from clusterlib.scheduler import queued_or_running_jobs
>>> queued_or_running_jobs()
["job-0001", "job-0002", ..., "job-9999"]
... completed jobs?
A job must indicate through the ﬁle system that it has been
completed e.g. by creating a ﬁle or by registering completion into a
database.
clusterlib provides a small NO-SQL database based on sqlite3.

Simplicity beats complexity
With only 4 functions
from clusterlib.scheduler import queued_or_running_jobs
from clusterlib.scheduler import submit
from clusterlib.storage import sqlite3_dumps
from clusterlib.storage import sqlite3_loads
We launch easily thousands of jobs.
We are task DRY (Don’t repeat yourself).
We are smoothly working on SLURM and SGE schedulers.
We have only a dependency on python,

Are we done?
Let’s go open source!

Why making an open source library?
Give back to the open source community.
Open source initiative aﬃliate communities

Bug reports are great!

Welcome new contributors !
From left to right: Olivier Grisel, Antonio Sutera, Loic Esteve (no
photo) and Konstantin Petrov (no photo).

Why making open source in sciences? Reproducibility!

Open source way: Host your code publicly
www.myproject.com
Another awesome host platform
Tip Sit on the shoulders of a giant!
Bonus Use a control version system such as git (clusterlib
choice), mercurial,. . .

Open source way: Choose a license
No license = closed source
In short
You can’t use or even read my code.

GPL-like license = copyleft
In short
You can read / use / share / modify my code, but derivatives
must retains those rights.

BSD / MIT-style license = permissive
In short
Do whatever you want with the code, but keep my name with it.

For the sake of open source, pick a popular open source license.

For the sake of wisdom or money, choose carefully.

For the sake of wisdom or money, choose carefully.
For the sake of science, go with BSD / MIT-style license.
Clusterlib is BSD Licensed.

Open source way: Start an issue tracker
An issue tracker allows managing and maintaining a list of
issues.
Tip Sit on the shoulders of giants! (again)

Open source way: Let users discuss with core contributors
1. Issue tracker (only viable for small project)
2. Mailing list : sourceforge, google groups, . . .
3. Stack overﬂow tag for big projects
Tip Sit on the shoulders of giants! (again * 2)

Are we done?
The grand seduction!

Know who you are!
Vision
The goal of the clusterlib is to ease the creation, launch and
management of embarrassingly parallel jobs on supercomputers
with schedulers such as SLURM and SGE.
Core values
Pure python, simple, user-friendly!

Attrative documentation
Readme

API documentation is nice.
Tip Follow a standard such as "PEP 0257 – Docstring
Conventions".

Beautiful doc is better.
clusterlib uses sphinx for building its doc.

And even better with examples.

A narrative documentation is awesome.

What’s new?
Thanks to all clusterlib contributors!

Appealing test suite
How good is your test suite (code coverage)?
All code lines are hit by the tests (100% line coverage), but not
all code paths (branches) are tested.
$ make test
nosetests clusterlib doc
Name Stmts Miss Branch BrMiss Cover Missing
------------------------------------------------------------------
clusterlib 1 0 0 0 100%
clusterlib.scheduler 82 0 40 16 87%
clusterlib.storage 35 0 14 0 100%
------------------------------------------------------------------
TOTAL 118 0 54 16 91%
------------------------------------------------------------------
Ran 12 tests in 0.189s
OK (SKIP=3)

Are we done?
Not yet! Let’s make our live easier!

Automation, automation, . . .
Continuous testing
clusterlib uses Travis CI (works for many languages).

Continuous integration pays oﬀ in the long term!
Ensure test suite is often run. Awesome during development.
Tip Continuous integration can be enhanced with code
test coverage and code quality coverage.

Continuous doc building

Create and join an open source projects!
Join a community! Learn the best practices and technologies!
Make your code survive more than the one project! Give your
code to the world!
scikit-learn sprint 2014

A full example of clusterlib usage
# main.py
import sys, os
from clusterlib.storage import sqlite3_dumps
NOSQL_PATH = os.path.join(os.environ["HOME"], "job.sqlite3")
def main(argv=None):
# For ease here, function parameters are sys.argv
if argv is None:
argv = sys.argv
# Do heavy computation
# Save script evaluation on the hard disk
if __name__ == "__main__":
main()
# Great, the jobs is done!
sqlite3_dumps({" ".join(sys.argv): "JOB DONE"}, NOSQL_PATH)

A full example of clusterlib usage
# launcher.py
import sys
from clusterlib.scheduler import queued_or_running_jobs
from clusterlib.scheduler import submit
from clusterlib.storage import sqlite3_loads
from main import NOSQL_PATH
if __name__ == "__main__":
scheduled_jobs = set(queued_or_running_jobs())
done_jobs = sqlite3_loads(NOSQL_PATH)
for param in range(100):
job_name = "job-param=%s" % param
job_command = ("%s main.py --param %s"
% (sys.executable, param))
if (job_name not in scheduled_jobs and
job_command not in done_jobs):
script = submit(job_command, job_name=job_name)

The genesis of clusterlib - An open source library to tame your favourite supercomputer

More Related Content

What's hot (19)

Viewers also liked (7)

Similar to The genesis of clusterlib - An open source library to tame your favourite supercomputer (20)

Recently uploaded (20)

The genesis of clusterlib - An open source library to tame your favourite supercomputer