6-9-2017-slides-vFinal.pptx

Introduction to Parallel
Programming
Center for Institutional Research
Computing
Slides for the book "An introduction to Parallel
Programming", by Peter Pacheco (available from the
publisher
website): http://booksite.elsevier.com/9780123
742605/
Copyright © 2010, Elsevier Inc. All rights
Reserved
1

Serial hardware and software
Copyright © 2010, Elsevier
Inc. All rights Reserved
input
output
programs
Computer runs one
program at a time.
2

Why we need to write parallel
programs
• Running multiple instances of a serial
program often isn’t very useful.
• Think of running multiple instances of your
favorite game.
• What you really want is for
it to run faster.
3

How do we write parallel programs?
• Task parallelism
– Partition various tasks carried out solving the
problem among the cores.
• Data parallelism
– Partition the data used in solving the problem
among the cores.
– Each core carries out similar operations on
it’s part of the data.
4

Professor P
15 questions
300 exams
5

Professor P’s grading assistants
TA#1
TA#2 TA#3
6

Division of work –
data parallelism
TA#1
TA#2
TA#3
100 exams
100 exams
100 exams
7

task parallelism
TA#1
TA#2
TA#3
Questions 1 - 5
Questions 6 - 10
Questions 11 - 15
Questions 1 - 7
Questions 8 - 11
Questions 12 - 15
Partitioning strategy:
- either by number
- Or by workload
or
or
or
8

Coordination
• Cores usually need to coordinate their work.
• Communication – one or more cores send
their current partial sums to another core.
• Load balancing – share the work evenly
among the cores so that one is not heavily
loaded.
• Synchronization – because each core works
at its own pace, make sure cores do not get
too far ahead of the rest.
9

What we’ll be doing
• Learning to write programs that are
explicitly parallel.
• Using the C language.
• Using the OpenMP extension to C
(multi-threading for shared memory)
– Others you can investigate after this
workshop:
• Message-Passing Interface (MPI)
• Posix Threads (Pthreads)
10

Essential concepts
• Memory
• Process execution terminology
• Configuration of Kamiak
• Coding concepts for parallelism - Parallel
program design
• Performance
• OpenMP
Reserved
11

.c programs we will use
• loop.c
• sync.c
• sumcomp.c
• matrix_vector.c (for independent study)
• Step 1: log into kamiak
• Step 2: run command: training
Reserved
12

Memory
• Two major classes of parallel programming
models:
– Shared Memory
– Distributed Memory
13

Shared Memory Architecture
CPU core #1
Memory
(Shared address
space)
disk
Memory bus
Compute node
I/O bus
Cache ($)
CPU core
Cache ($)
…
CPU core
Cache ($)
Threads
Threads Threads
All threads can see a single
shared address space.
Therefore, they can see each
other’s data.
Each compute node has 1-2
CPUs each with 10-14 cores
14

Multi-Threading (for shared
memory architectures)
• Threads are contained within processes
– One process => multiple threads
• All threads of a process share the same
address space (in memory).
• Threads have the capability to run
concurrently (executing different
instructions and accessing different
pieces of data at the same time)
• But if the resource is occupied by another
thread, they form a queue and wait.
– For maximum throughput, it is ideal to
map each thread to a unique/distinct core
15
CPU cores
Cache ($)
Threads
Memory
(Shared
address
space)

A process and two threads
the “master” thread
starting a thread
Is called forking
terminating a thread
Is called joining
16

Distributed Memory Architecture
Local Memory
Compute node 1
Local Memory
Compute node 2
Local Memory
Compute node m
………
Network Interconnect
Processes running
on cores
Processes running
on cores
Processes running
on cores
Processes cannot see each other’s memory address space.
They have to send inter-process messages (using MPI).
17

Distributed Memory System
• Clusters (most popular)
– A collection of commodity systems.
– Connected by a commodity interconnection
network.
• Nodes of a cluster are individual
computers joined by a communication
network.
a.k.a. hybrid systems
Kamiak provides an Infiniband
interconnect between all compute nodes
18

Single Program Models:
SIMD vs. MIMD
19
• SP: Single Program
– Your parallel program is a single program that you execute
on all threads (or processes)
• SI: Single Instruction
– Each thread should be executing the same line of code at
any given clock cycle.
• MI: Multiple Instruction
– Each thread (or process) could be independently running a
different line of your code (instruction) concurrently
• MD: Multiple Data
– Each thread (or process) could be operating/accessing a
different piece of the data from the memory concurrently

Single Program Models:
SIMD vs. MIMD
20
// Begin: parallel region of the code
…
..
..
..
..
..
..
..
..
..
..
// End: parallel region of the code
// Begin: parallel region of the code
…
..
..
..
..
..
..
..
..
..
..
// End: parallel region of the code
All threads executing the
same line of code.
They may be accessing
different pieces of data.
Thread 1
Thread 2
Thread 3
SIMD MIMD

Foster’s methodology
1. Partitioning: divide the computation to be
performed and the data operated on by
the computation into small tasks.
The focus here should be on identifying
tasks that can be executed in parallel.
21

2. Communication: determine what
communication needs to be carried out
among the tasks identified in the previous
step.
22

3. Agglomeration or aggregation: combine
tasks and communications identified in
the first step into larger tasks.
For example, if task A must be executed
before task B can be executed, it may
make sense to aggregate them into a
single composite task.
23

4. Mapping: assign the composite tasks
identified in the previous step to
processes/threads.
This should be done so that
communication is minimized, and each
process/thread gets roughly the same
amount of work.
24

Example from sum.c
• Open sum.c
• What can be parallelized here?
Reserved
25

OPENMP FOR SHARED MEMORY
MULTITHREADED PROGRAMMING
Reserved
26

Roadmap
• Writing programs that use OpenMP.
• Using OpenMP to parallelize many serial for
loops with only small changes to the source
code.
• Task parallelism.
• Explicit thread synchronization.
• Standard problems in shared-memory
programming.
27

Pragmas
• Special preprocessor instructions.
• Typically added to a system to allow
behaviors that aren’t part of the basic C
specification.
• Compilers that don’t support the pragmas
ignore them.
#pragma
28

OpenMp pragmas
• # pragma omp parallel
• # include omp.h
– Most basic parallel directive.
– The number of threads that run
the following structured block of code
is determined by the run-time system.
29

clause
• Text that modifies a directive.
• The num_threads clause can be added to
a parallel directive.
• It allows the programmer to specify the
number of threads that should execute the
following block.
# pragma omp parallel num_threads ( thread_count )
30

Some terminology
• In OpenMP parlance the collection of
threads executing the parallel block — the
original thread and the new threads — is
called a team, the original thread is called
the master, and the additional threads are
called worker.
31

In case the compiler doesn’t
support OpenMP
# include <omp.h>
#ifdef _OPENMP
# include <omp.h>
#endif
32

In case the compiler doesn’t
support OpenMP
# ifdef _OPENMP
int my_rank = omp_get_thread_num ( );
int thread_count = omp_get_num_threads ( );
# e l s e
int my_rank = 0;
int thread_count = 1;
# endif
33

#include <stdio.h>
int main()
{
printf("Hello worldn");
return 0;
}
Reserved
34
Serial version of “hello world”
How do we invoke omp in this case?
Compile it: gcc hello.c

After invoking omp
35
Compile it: gcc –fopenmp hello.c

Parallel Code Template (OpenMP)
#include <omp.h>
main(…) {
… // let p be the user-specified #threads
omp_set_num_threads(p);
#pragma omp parallel for
{
…. // openmp parallel region where p threads are
active and running concurrently
}
Reserved
36

Scope
• In serial programming, the scope of a
variable consists of those parts of a
program in which the variable can be
used.
• In OpenMP, the scope of a variable refers
to the set of threads that can access the
variable in a parallel block.
37

Scope in OpenMP
• A variable that can be accessed by all the
threads in the team has shared scope.
• A variable that can only be accessed by a
single thread has private scope.
• The default scope for variables
declared before a parallel block
is shared.
38

Loop.c
• Lets go over our first code – loop-serial.c
• Now edit to employ omp, save as loop-
parallel.c
Reserved
39

Performance
Reserved
40

Taking Timings
• What is time?
• Start to finish?
• A program segment of interest?
• CPU time?
• Wall clock time?
41

Taking Timings
theoretical
function
MPI_Wtime omp_get_wtime
42

• Number of threads = p
• Serial run-time = Tserial
• Parallel run-time = Tparallel
Tparallel = Tserial
/ p
Speedup
Tserial
Tparallel
S =
43

Scalability
• In general, a problem is scalable if it can handle
ever increasing problem sizes.
• If we increase the number of processes/threads
and keep the efficiency fixed without increasing
problem size, the problem is strongly scalable.
• If we keep the efficiency fixed by increasing the
problem size at the same rate as we increase
the number of processes/threads, the problem is
weakly scalable.
44

Studying Scalability
Input
size (n)
Number of threads (p)
1 2 4 8 16
1,000
2,000
4,000
8,000
16,000
Table records the parallel runtime (in seconds) for varying values of n and p.
45
It is conventional to test scalability in powers of two (or by doubling n and p).

Input
size (n)
Number of threads (p)
1 2 4 8 16
1,000 800 410 201 150 100
2,000 1,601 802 409 210 120
4,000 3,100 1,504 789 399 208
8,000 6,010 3,005 1,500 758 376
16,000 12,000 6,000 3,001 1,509 758
It is conventional to test scalability in powers of two (or by doubling n and p).
Table records the parallel runtime (in seconds) for varying values of n and p.
Strong scaling
behavior
Weak scaling
behavior
46

0
2000
4000
6000
8000
10000
12000
14000
1 2 4 8 16
n=1,000
n=2,000
n=4,000
n=8,000
n=16,000
p
Time
(sec)
0
2
4
6
8
10
12
14
16
18
1 2 4 8 16
n=1,000
n=2,000
n=4,000
n=8,000
n=16,000
Ideal
Speedup
p
Reserved
47

Timings with loop-parallel.c
(your code) or loop.c (our code)
Reserved
48
You try!
< 10 minutes

x1 x1
+2
x3 x4 x5 x6 x7
+1 +2 +3 +4 +5 +6
x2
x1+2
+3
x1+2+
3+4
x1+2+3+
4+5
x1+2+3+4
+5+6
x1+2+3+4+5
+6+7
x1+2+3+4+5+
6+7+8
x8
Serial Process (1 thread, 7 operations)
Serial vs. Parallel Reduction
49
+7

x1+2+3+4+5+
6+7+8
x5+6+7+8
x1+2+3+4
x1+2
x1+2
x1+2
x1 x2 x3 x4 x5 x6 x7 x8
t1 t2 t3 t4 t5 t6 t7 t8
x1+2
+1
+2
+3
Parallel Process (8 threads, 3 operations)
50
log2 p
stages

Reduction operators
• A reduction operator is a binary operation
(such as addition or multiplication).
• A reduction is a computation that
repeatedly applies the same reduction
operator to a sequence of operands in
order to get a single result.
• All of the intermediate results of the
operation should be stored in the same
variable: the reduction variable.
51

Computing a sum
• Open sumcompute-serial.c
• Lets go over it
Reserved
52
< 5 minutes

Mutual exclusion
# pragma omp critical
global_result += my_result ;
only one thread can execute
the following structured block at a time
53

Example
• Open sync-unsafe.c
Reserved
54

Synchronization
• Synchronization imposes order constraints
and is used to protect access to shared data
• Types of synchronization:
– critical
– atomic
– locks
– others (barrier, ordered, flush)
• We will work on an exercise involving critical,
atomic, and locks
Reserved
55

Critical
#pragma omp parallel for schedule(static) shared(a)
for(i = 0; i < n; i++)
{
#pragma omp critical
{
a = a+1;
}
}
Threads wait here: only one thread at a
time does the operation: “a = a+1”. So
this is a piece of sequential code inside
the for loop.
Reserved
56

Atomic
• Atomic provides mutual exclusion but only applies to the load/update of a
memory location
• It is applied only to the (single) assignment statement that immediately
follows it
• Atomic construct may only be used together with an expression statement
with one of operations: +, *, -, /, &, ^, |, <<, >>
• Atomic construct does not prevent multiple threads from executing the
function() at the same time (see the example below)
Code example:
int ic, i, n;
ic = 0;
#pragma omp parallel shared(n,ic) private(i)
for (i=0; i++; i<n)
{
#pragma omp atomic
ic = ic + function(c);
}
Atomic only protects the
update of ic
Reserved
57

Locks
• A lock consists of a data structure and functions
that allow the programmer to explicitly enforce
mutual exclusion in a critical section.
58
Acquire lock(i)
// critical section
Release lock(i)
Threads queue up
Only one
thread executes
Difference from critical
section:
- You can have multiple
locks
- A thread can try for any
specific lock
- => we can use this to
acquire data-level
locks
e.g., two threads can access different array indices without waiting.

Illustration of Locking Operation
• The protected region contains the
update of a shared variable
• One thread acquires the lock and
performs the update
• Meanwhile, other threads perform
some other work
• When the lock is released again,
the other threads perform the
update
Reserved
59

A Locks Code Example
long long int a=0;
long long int i;
omp_lock_t my_lock;
// init lock
omp_init_lock(&my_lock);
#pragma omp parallel for
for(i = 0; i < n; i++)
{
omp_set_lock(&my_lock);
a+=1;
omp_unset_lock(&my_lock);
}
omp_destroy_lock(&my_lock);
1. Define lock variable
2. Initialize lock
3. Set lock
4. Unset lock
5. Destroy lock
Compiling and running sync.c:
gcc −g −Wall −fopenmp −o sync sync.c
./sync #of-iteration #of-threads
Reserved
60

Some Caveats
1. You shouldn’t mix the different types of
mutual exclusion for a single critical
section.
2. There is no guarantee of fairness in
mutual exclusion constructs.
3. It can be dangerous to “nest” mutual
exclusion constructs.
61

• Default schedule:
Loop.c example
#pragma omp parallel for schedule(static)
private(a)//creates N threads to run the
next enclosed block
for(i = 0; i < loops; i++)
{
a = 6+7*8;
}
62

The Runtime Schedule Type
• The system uses the environment variable
OMP_SCHEDULE to determine at run-
time how to schedule the loop.
• The OMP_SCHEDULE environment
variable can take on any of the values that
can be used for a static, dynamic, or
guided schedule.
63

schedule ( type , chunksize )
– Static: Assigned before the loop is executed.
– dynamic or guided: Assigned while the loop is
executing.
– auto/ runtime: Determined by the compiler and/or the
run-time system
Controls how loop iterations are assigned
• Consecutive iterations are broken into chunks
• Total number = chunksize
• Positive integer
• Default is 1
64

schedule types can prevent load
imbalance
Threads
Time
Static schedule Dynamic schedule
vs
0
1
2
3
4
5
6
7
0
1
2
3 4
5
6
7
Time
Reserved
65
Thread finishing
first
Thread finishing
last
Idle time

Static: default
Static, n: set chunksize
Static
Static, n
Dynamic
Guided
iteration number
0
0
0
0
N-1
N-1
N-1
N-1
Thread 0 Thread 1 Thread 2 Thread 3
Thre
ad 0
Thre
ad 1
Thre
ad 2
Thre
ad 3
Thre
ad 0
Thre
ad 1
Thre
ad 2
Thre
ad 3
Thre
ad 0
Thre
ad 1
T
2
Thre
ad 0
Thre
ad 1
Thre
ad 2
Thre
ad 3
Thre
ad 0
Thre
ad 1
Thre
ad 2
Thre
ad 3
T
0
Thre
ad 1
Thre
ad 1
Thre
ad 0
Thre
ad 1
Thre
ad 2
Thre
ad 3
Th
r0
Th
r1
Th
r2
Th
r3
T
0
T
1
T
2
T
3
T
0
T
1
2
chunksize
66

Dynamic: thread executes a chunk
when done, it requests another one
Static
Static, n
Dynamic
Guided
iteration number
0
0
0
0
N-1
N-1
N-1
N-1
Thre
ad 0
Thre
ad 1
Thre
ad 2
Thre
ad 3
Thre
ad 0
Thre
ad 1
Thre
ad 2
Thre
ad 3
Thre
ad 0
Thre
ad 1
T
2
Thre
ad 0
Thre
ad 1
Thre
ad 2
Thre
ad 3
Thre
ad 0
Thre
ad 1
Thre
ad 2
Thre
ad 3
T
0
Thre
ad 1
Thre
ad 1
Thre
ad 0
Thre
ad 1
Thre
ad 2
Thre
ad 3
Th
r0
Th
r1
Th
r2
Th
r3
T
0
T
1
T
2
T
3
T
0
T
1
2
67

Guided: thread executes a chunk
when done, it requests another one
new chunks decrease in size (until
chunksize is met)
Static
Static, n
Dynamic
Guided
iteration number
0
0
0
0
N-1
N-1
N-1
N-1
Thre
ad 0
Thre
ad 1
Thre
ad 2
Thre
ad 3
Thre
ad 0
Thre
ad 1
Thre
ad 2
Thre
ad 3
Thre
ad 0
Thre
ad 1
T
2
Thre
ad 0
Thre
ad 1
Thre
ad 2
Thre
ad 3
Thre
ad 0
Thre
ad 1
Thre
ad 2
Thre
ad 3
T
0
Thre
ad 1
Thre
ad 1
Thre
ad 0
Thre
ad 1
Thre
ad 2
Thre
ad 3
Th
r0
Th
r1
Th
r2
Th
r3
T
0
T
1
T
2
T
3
T
0
T
1
2
68

multiplication.c
• Go over this code at a conceptual level – show
where and how to parallelize
Reserved
69

Matrix-vector multiplication
70

M x V = X: Parallelization Strategies
71
=
x
Row Decomposition
Thread 0
Thread 1
Thread 2
Thread 3
Thre
ad 0
Thre
ad 1
Thre
ad 2
Thre
ad 3
m x n n x 1 m x 1
 Each thread gets m/p rows
 Time taken is proportional to: (mn)/p : per thread
 No need for any synchronization (static scheduling will do)

M x V = X: Parallelization Strategies
72
=
x
Column Decomposition
Threa
d 0
Thread
1
Thread
2
Thread
3
m x n n x 1 m x 1
 Each thread gets n/p columns
 Time taken is proportional to: (mn)/p + time for reduction : per thread
x0 + x1 + x2 + x3
X [i]
Reduction

Extra slides
Reserved
73

What happened?
1. OpenMP compilers don’t
check for dependences
among iterations in a loop
that’s being parallelized
with a parallel for directive.
2. A loop in which the results
of one or more iterations
depend on other iterations
cannot, in general, be
correctly parallelized by
OpenMP.
74

Estimating π
75

OpenMP solution #1
loop dependency
76

OpenMP solution #2
Insures factor has
private scope.
77

Thread-Safety
78

An operating system “process”
• An instance of a computer program that is
being executed.
• Components of a process:
– The executable machine language program.
– A block of memory.
– Descriptors of resources the OS has allocated
to the process.
– Security information.
– Information about the state of the process.
79

Shared Memory System
• Each processor can access each memory
location.
• The processors usually communicate
implicitly by accessing shared data
structures.
• Example: Multiple CPU cores on a single
chip
Kamiak compute nodes have multiple
CPUs each with multiple cores
80

UMA multicore system
Figure 2.5
Time to access all
the memory locations
will be the same for
all the cores.
81

NUMA multicore system
Figure 2.6
A memory location a core is
directly connected to can be
accessed faster than a memory
location that must be accessed
through another chip.
82

Input and Output
• However, because of the indeterminacy of
the order of output to stdout, in most cases
only a single process/thread will be used
for all output to stdout other than
debugging output.
• Debug output should always include the
rank or id of the process/thread that’s
generating the output.
83

Input and Output
• Only a single process/thread will attempt
to access any single file other than stdin,
stdout, or stderr. So, for example, each
process/thread can open its own, private
file for reading or writing, but no two
processes/threads will open the same file.
84

data parallelism
85

task parallelism
Tasks
1) Receiving
2) Addition
86

Distributed Memory on Kamiak
Figure 2.4
compute node 1 compute node N
compute node 3
compute node 2
Each compute node has 1-2
CPUs each with 10-14 cores
87

The burden is on software
• Hardware and compilers can keep up the
pace needed.
• From now on…
– In shared memory programs:
• Start a single process and fork threads.
• Threads carry out tasks.
– In distributed memory programs:
• Start multiple processes.
• Processes carry out tasks.
89

Writing Parallel Programs
double x[n], y[n];
…
for (i = 0; i < n; i++)
x[i] += y[i];
1. Divide the work among the
processes/threads
(a) so each process/thread
gets roughly the same
amount of work
(b) and communication is
minimized.
2. Arrange for the processes/threads to synchronize.
3. Arrange for communication among processes/threads.
I think we will
eventually
introduce this when
doing sum. I would
remove this slide
from here.
90

Ananth can you put in a simple
example here?
Remove Slide
Reserved
91

OpenMP (ask ananth to tailor more
for the loop.c code)
• An API for shared-memory parallel
programming.
• MP = multiprocessing
• Designed for systems in which each
thread or process can potentially have
access to all available memory.
• System is viewed as a collection of cores
or CPU’s, all of which have access to
main memory.
Remove Slide
92

A process forking and joining two
threads
93

Of note…
• There may be system-defined limitations on the
number of threads that a program can start.
• The OpenMP standard doesn’t guarantee that
this will actually start thread_count threads.
• Most current systems can start hundreds or
even thousands of threads.
• Unless we’re trying to start a lot of threads, we
will almost always get the desired number of
threads.
94

Have people take serial version and make into the openMP version
of loop.c
-have a small example code that has the pragma and openMP calls
in it
-then have them adapt the loop.c to have the openmp in it…give
them 10 minutes to do while we go around the room and
help….then walk them through our parallel version of loop.c
95

SCHEDULING LOOPS IN SYNC.C
96

Locks
• A lock implies a memory fence of all thread visible variables
• The lock routines are used to guarantee that only one thread
accesses a variable at a time to avoid race conditions
• C/C++ lock variables must have type “omp_lock_t” or
“omp_nest_lock_t” (will not discuss nested lock in this
workshop)
• All lock functions require an argument that has a pointer to
omp_lock_t or omp_nest_lock_t
• Simple Lock functions:
– omp_init_lock(omp_lock_t*);
– omp_set_lock(omp_lock_t*);
– omp_unset_lock(omp_lock_t*);
– omp_test_lock(omp_lock_t*);
– omp_destroy_lock(omp_lock_t*);
Reserved
97

How to Use Locks
1) Define the lock variables
2) Initialize the lock via a call to omp_init_lock
3) Set the lock using omp_set_lock or omp_test_lock. The
latter checks whether the lock is actually available
before attempting to set it. It is useful to achieve
asynchronous thread execution.
4) Unset a lock after the work is done via a call to
omp_unset_lock.
5) Remove the lock association via a call to
omp_destroy_lock.
Reserved
98

Matrix-vector multiplication
Run-times and efficiencies
of matrix-vector multiplication
(times are in seconds)
99

6-9-2017-slides-vFinal.pptx

More Related Content

Similar to 6-9-2017-slides-vFinal.pptx (20)

Recently uploaded (20)

6-9-2017-slides-vFinal.pptx

Editor's Notes