Parallelism - Parallel Computing

A codelsss
introduction to
GPU parallelism
Will Landau
A review of GPU
parallelism
Examples of
parallelism
Vector addition
Pairwise summation
Matrix multiplication
K-means clustering
Markov chain Monte
Carlo
A codelsss introduction to GPU parallelism
Will Landau
Iowa State University
September 23, 2013
Will Landau (Iowa State University) A codelsss introduction to GPU parallelism September 23, 2013 1 / 47
A codelsss
introduction to
GPU parallelism
Will Landau
A review of GPU
parallelism
Examples of
parallelism
Vector addition
Pairwise summation
K-means clustering
Markov chain Monte
Carlo
Outline
A review of GPU parallelism
Examples of parallelism
Vector addition
Pairwise summation
K-means clustering
Markov chain Monte Carlo
A codelsss
introduction to
GPU parallelism
Will Landau
A review of GPU
parallelism
Examples of
parallelism
Vector addition
Pairwise summation
K-means clustering
Markov chain Monte
Carlo
Outline
Vector addition
Pairwise summation
K-means clustering
A codelsss
introduction to
GPU parallelism
Will Landau
A review of GPU
parallelism
Examples of
parallelism
Vector addition
Pairwise summation
K-means clustering
Markov chain Monte
Carlo
The single instruction, multiple data (SIMD)
paradigm
SIMD: apply the same command to multiple places in a

dataset.
f o r ( i = 0; i < 1e6 ; ++i )
a [ i ] = b [ i ] + c [ i ] ;
On CPUs, the iterations of the loop run sequentially.
With GPUs, we can easily run all 1,000,000 iterations

simultaneously.
i = t hr e a dI dx . x ;
a [ i ] = b [ i ] + c [ i ] ;
We can similarly parallelize a lot more than just loops.

A codelsss
introduction to
GPU parallelism
Will Landau
A review of GPU
parallelism
Examples of
parallelism
Vector addition
Pairwise summation
K-means clustering
Markov chain Monte
Carlo
CPU / GPU cooperation
The CPU (host) is in charge.
The CPU sends computationally intensive instruction

sets to the GPU (device) just like a human uses a
pocket calculator.
A codelsss
introduction to
GPU parallelism
Will Landau
A review of GPU
parallelism
Examples of
parallelism
Vector addition
Pairwise summation
K-means clustering
Markov chain Monte
Carlo
How GPU parallelism works
1. The CPU sends a command called a kernel to a GPU.
2. The GPU executes several duplicate realizations of this
command, called threads.
These threads are grouped into bunches called blocks.
The sum total of all threads in a kernel is called a grid.
Toy example:
CPU says: Hey, GPU. Sum pairs of adjacent numbers.

Use the array, (1, 2, 3, 4, 5, 6, 7, 8).
GPU thinks: Sum pairs of adjacent numbers is a

kernel that I need to apply to the array, (1, 2, 3, 4, 5, 6,
8).
The GPU spawns 2 blocks, each with 2 threads:

Block 0 1
Thread 0 1 0 1
Action 1 + 2 3 + 4 5 + 6 7 + 8
I could have also used 1 block with 4 threads and given

the threads dierent pairs of numbers.
A codelsss
introduction to
GPU parallelism
Will Landau
A review of GPU
parallelism
Examples of
parallelism
Vector addition
Pairwise summation
K-means clustering
Markov chain Monte
Carlo
A codelsss
introduction to
GPU parallelism
Will Landau
A review of GPU
parallelism
Examples of
parallelism
Vector addition
Pairwise summation
K-means clustering
Markov chain Monte
Carlo
Outline
Vector addition
Pairwise summation
K-means clustering
A codelsss
introduction to
GPU parallelism
Will Landau
A review of GPU
parallelism
Examples of
parallelism
Vector addition
Pairwise summation
K-means clustering
Markov chain Monte
Carlo
Examples of parallelism Vector addition
Vector addition
Say I have 2 vectors,

a =
_
_
a
1
a
2
.
.
.
a
n
_
_
b =
_
_
b
1
b
2
.
.
.
b
n
_
I want to compute their component-wise sum,

c =
_
_
c
1
c
2
.
.
.
c
n
_
_
=
_
_
a
1
+ b
1
a
2
+ b
2
.
.
.
a
n
+ b
n
_
_
A codelsss
introduction to
GPU parallelism
Will Landau
A review of GPU
parallelism
Examples of
parallelism
Vector addition
Pairwise summation
K-means clustering
Markov chain Monte
Carlo
Vector addition
A codelsss
introduction to
GPU parallelism
Will Landau
A review of GPU
parallelism
Examples of
parallelism
Vector addition
Pairwise summation
K-means clustering
Markov chain Monte
Carlo
Vector addition
A codelsss
introduction to
GPU parallelism
Will Landau
A review of GPU
parallelism
Examples of
parallelism
Vector addition
Pairwise summation
K-means clustering
Markov chain Monte
Carlo
Vector addition
A codelsss
introduction to
GPU parallelism
Will Landau
A review of GPU
parallelism
Examples of
parallelism
Vector addition
Pairwise summation
K-means clustering
Markov chain Monte
Carlo
Examples of parallelism Pairwise summation
Pairwise summation
Lets take the pairwise sum of the vector,

(5, 2, 3, 1, 1, 8, 2, 6)
using 1 block of 4 threads.
A codelsss
introduction to
GPU parallelism
Will Landau
A review of GPU
parallelism
Examples of
parallelism
Vector addition
Pairwise summation
K-means clustering
Markov chain Monte
Carlo
Pairwise summation
5 2 -3 1 1 8 2 6
Thread 1
6
0
A codelsss
introduction to
GPU parallelism
Will Landau
A review of GPU
parallelism
Examples of
parallelism
Vector addition
Pairwise summation
K-means clustering
Markov chain Monte
Carlo
Pairwise summation
5 2 -3 1 1 8 2 6
6 10
Thread 2
1
A codelsss
introduction to
GPU parallelism
Will Landau
A review of GPU
parallelism
Examples of
parallelism
Vector addition
Pairwise summation
K-means clustering
Markov chain Monte
Carlo
Pairwise summation
5 2 -3 1 1 8 2 6
6 10 -1
Thread 32
A codelsss
introduction to
GPU parallelism
Will Landau
A review of GPU
parallelism
Examples of
parallelism
Vector addition
Pairwise summation
K-means clustering
Markov chain Monte
Carlo
Pairwise summation
5 2 -3 1 1 8 2 6
Thread 3
6 10 -1 7
A codelsss
introduction to
GPU parallelism
Will Landau
A review of GPU
parallelism
Examples of
parallelism
Vector addition
Pairwise summation
K-means clustering
Markov chain Monte
Carlo
Pairwise summation
5 2 -3 1 1 8 2 6
6 10 -1 7
Synchronize threads
A codelsss
introduction to
GPU parallelism
Will Landau
A review of GPU
parallelism
Examples of
parallelism
Vector addition
Pairwise summation
K-means clustering
Markov chain Monte
Carlo
Synchronizing threads
Synchronization: waiting for all parallel tasks to reach

a checkpoint before allowing any of then to continue.
Threads from the same block can be synchronized easily.
In general, do not try to synchronize threads from

dierent blocks. Its possible, but extremely inecient.
A codelsss
introduction to
GPU parallelism
Will Landau
A review of GPU
parallelism
Examples of
parallelism
Vector addition
Pairwise summation
K-means clustering
Markov chain Monte
Carlo
Pairwise summation
5 2 -3 1 1 8 2 6
6 10 -1 7
5
Thread 0
A codelsss
introduction to
GPU parallelism
Will Landau
A review of GPU
parallelism
Examples of
parallelism
Vector addition
Pairwise summation
K-means clustering
Markov chain Monte
Carlo
Pairwise summation
5 2 -3 1 1 8 2 6
6 10 -1 7
5 17
Thread 1
A codelsss
introduction to
GPU parallelism
Will Landau
A review of GPU
parallelism
Examples of
parallelism
Vector addition
Pairwise summation
K-means clustering
Markov chain Monte
Carlo
Pairwise summation
5 2 -3 1 1 8 2 6
6 10 -1 7
5 17
Synchronize Threads
A codelsss
introduction to
GPU parallelism
Will Landau
A review of GPU
parallelism
Examples of
parallelism
Vector addition
Pairwise summation
K-means clustering
Markov chain Monte
Carlo
Pairwise summation
5 2 -3 1 1 8 2 6
6 10 -1 7
Thread 0
22
5 17
A codelsss
introduction to
GPU parallelism
Will Landau
A review of GPU
parallelism
Examples of
parallelism
Vector addition
Pairwise summation
K-means clustering
Markov chain Monte
Carlo
Compare the pairwise sum to the sequential sum
The pairwise sum requires only log

2
(n) sequential steps,
while the sequential sum requires n 1 sequential steps.
A codelsss
introduction to
GPU parallelism
Will Landau
A review of GPU
parallelism
Examples of
parallelism
Vector addition
Pairwise summation
K-means clustering
Markov chain Monte
Carlo
Reductions and scans
Reductions
Pairwise sum and pairwise multiplication are examples

of reductions.
Reduction: an algorithm that applies some binary

operation on a vector to produce a scalar.
Scans
Scan (prex sum): an operation on a vector that

produces a sequence of partial reductions.
Example: computing the sequence of partial sums in

pairwise fashion.
A codelsss
introduction to
GPU parallelism
Will Landau
A review of GPU
parallelism
Examples of
parallelism
Vector addition
Pairwise summation
K-means clustering
Markov chain Monte
Carlo
Examples of parallelism Matrix multiplication
Take an m n matrix, A = (a
ij
), and an n p matrix, B = (b
jk
).
Compute C = A B:
Write A in terms of its rows: A =
a
1.
.
.
.
a
m.
where
a
i .
=
a
i 1
a
in
Write B in terms of its columns: B =
b
.1
b
.p
where
b
.k
=
b
1k
.
.
.
b
nk
Compute C = A B by taking the product of each row of A with

each column of B:
C = A B =
(a
1.
b
.1
) (a
1.
b
.p
)
.
.
.
.
.
.
.
.
.
(a
m.
b
.1
) (a
m.
b
.p
)
A codelsss
introduction to
GPU parallelism
Will Landau
A review of GPU
parallelism
Examples of
parallelism
Vector addition
Pairwise summation
K-means clustering
Markov chain Monte
Carlo
Parallelizing matrix multiplication
Entry (i , k) of matrix C is
c
ik
= a
i 1
b
1k
. .
+a
i 2
b
2k
. .
+ + a
in
b
nk
. .
= c
i 1k
+ c
i 2k
+ + c
ink
Assign block (i , k) to compute c

ik
.
1. Spawn n threads.
2. Tell the j th thread to compute c
ijk
= a
ij
b
jk
.
3. Synchronize threads to make sure we have nished
calculating c
i 1k
, c
i 2k
, . . . , c
ink
before continuing.
4. Compute c
ik
=
n
j =1
c
ijk
as a pairwise sum.
A codelsss
introduction to
GPU parallelism
Will Landau
A review of GPU
parallelism
Examples of
parallelism
Vector addition
Pairwise summation
K-means clustering
Markov chain Monte
Carlo
Say I want to compute A B, where:

A =
_
_
1 2
1 5
7 9
_
_
B =
_
8 8 7
3 5 2
_
I write the multiplication as an array of products:

C =
_
1 2
8
3

1 2
8
5

1 2
7
2
1 5
8
3

1 5
8
5

1 5
7
2
7 9
8
3

7 9
8
5

7 9
7
2
_
A codelsss
introduction to
GPU parallelism
Will Landau
A review of GPU
parallelism
Examples of
parallelism
Vector addition
Pairwise summation
K-means clustering
Markov chain Monte
Carlo
We dont need to synchronize blocks because they

operate independently.
A codelsss
introduction to
GPU parallelism
Will Landau
A review of GPU
parallelism
Examples of
parallelism
Vector addition
Pairwise summation
K-means clustering
Markov chain Monte
Carlo
Consider block (0, 0), which computes

_
1 2
_
8
3
_
A codelsss
introduction to
GPU parallelism
Will Landau
A review of GPU
parallelism
Examples of
parallelism
Vector addition
Pairwise summation
K-means clustering
Markov chain Monte
Carlo
A codelsss
introduction to
GPU parallelism
Will Landau
A review of GPU
parallelism
Examples of
parallelism
Vector addition
Pairwise summation
K-means clustering
Markov chain Monte
Carlo
A codelsss
introduction to
GPU parallelism
Will Landau
A review of GPU
parallelism
Examples of
parallelism
Vector addition
Pairwise summation
K-means clustering
Markov chain Monte
Carlo
Examples of parallelism K-means clustering
Lloyds K-means algorithm
Cluster N vectors in Euclidian space into K groups.

A codelsss
introduction to
GPU parallelism
Will Landau
A review of GPU
parallelism
Examples of
parallelism
Vector addition
Pairwise summation
K-means clustering
Markov chain Monte
Carlo
Step 1: choose initial cluster centers.
The circles are the cluster means, the squares are the
data points, and the color indicates the cluster.
A codelsss
introduction to
GPU parallelism
Will Landau
A review of GPU
parallelism
Examples of
parallelism
Vector addition
Pairwise summation
K-means clustering
Markov chain Monte
Carlo
Step 2: assign each data point (square) to its
closest center (circle).
A codelsss
introduction to
GPU parallelism
Will Landau
A review of GPU
parallelism
Examples of
parallelism
Vector addition
Pairwise summation
K-means clustering
Markov chain Monte
Carlo
Step 3: update the cluster centers to be the
within-cluster data means.
A codelsss
introduction to
GPU parallelism
Will Landau
A review of GPU
parallelism
Examples of
parallelism
Vector addition
Pairwise summation
K-means clustering
Markov chain Monte
Carlo
Repeat step 2: reassign points to their closest
cluster centers.
. . . and repeat until convergence.

A codelsss
introduction to
GPU parallelism
Will Landau
A review of GPU
parallelism
Examples of
parallelism
Vector addition
Pairwise summation
K-means clustering
Markov chain Monte
Carlo
Parallel K-means
Step 2: assign points to closest cluster centers.
Spawn N blocks with K threads each.
Let thread (n, k) compute the distance between data

point n and cluster center k.
Synchronize threads.
Let thread (n, 1) assign data point n to its nearest

cluster center.
Step 3: recompute cluster centers.
Spawn one block for each cluster.
Within each block, compute the mean of the data in the

corresponding cluster.
A codelsss
introduction to
GPU parallelism
Will Landau
A review of GPU
parallelism
Examples of
parallelism
Vector addition
Pairwise summation
K-means clustering
Markov chain Monte
Carlo
Examples of parallelism Markov chain Monte Carlo
Consider a bladder cancer data set:
Available from http://ratecalc.cancer.gov/.
Rates of death from bladder cancer of white males from 2000 to

2004 in each county in the USA.
Let:
y
k
= number of observed deaths in county k.
n
k
= the number of person-years in county k divided by 100,000.

k
= expected number of deaths per 100,000 person-years.
The model:
y
k
ind
Poisson(n
k

k
)
k
iid
Gamma(, )
Uniform(0, a
0
)
Uniform(0, b
0
)
Also assume and are independent and x a

0
and b
0
.
A codelsss
introduction to
GPU parallelism
Will Landau
A review of GPU
parallelism
Examples of
parallelism
Vector addition
Pairwise summation
K-means clustering
Markov chain Monte
Carlo
Full conditional distributions
We want to sample from the joint posterior,

p(, , | y) p(y | , , )p(, , )
p(y | , , )p( | , )p(, )
p(y | , , )p( | , )p()p()
k=1
[p(y
k
|
k
, n
k
)p(
k
| , )]p()p()
k=1
e
n
k
y
k
k
()
1
k
e
I (0 < < a
0
)I (0 < < b
0
)
We iteratively sample from the full conditional distributions.

p( | y, , )
p( | y, , )
k
p(
k
| y,
k
, , ) IN PARALLEL!
A codelsss
introduction to
GPU parallelism
Will Landau
A review of GPU
parallelism
Examples of
parallelism
Vector addition
Pairwise summation
K-means clustering
Markov chain Monte
Carlo
Full conditional distributions
p(
k
| y,
k
, , ) p(, , | y)
e
n
k
y
k
k

1
k
e
=
y
k
+1
k
e
k
(n
k
+)
Gamma(y
k
+ , n
k
+ )
A codelsss
introduction to
GPU parallelism
Will Landau
A review of GPU
parallelism
Examples of
parallelism
Vector addition
Pairwise summation
K-means clustering
Markov chain Monte
Carlo
Conditional distributions of and
p( | y, , ) p(, , | y)
k=1
_
1
k
()
_
I (0 < < a
0
)
=
_
K
k=1
k
_
K
()
K
I (0 < < a
0
)
p( | y, , ) p(, , | y)
k=1
_
e
I (0 < < b
0
)
=
K
e
K
k=1

k
I (0 < < b
0
)
Gamma
_
K + 1,
K
k=1
k
_
I (0 < < b
0
)
A codelsss
introduction to
GPU parallelism
Will Landau
A review of GPU
parallelism
Examples of
parallelism
Vector addition
Pairwise summation
K-means clustering
Markov chain Monte
Carlo
Summarizing the Gibbs sampler
1. Sample from from its full conditional.
Draw the
k
s in parallel from independent
Gamma(y
k
+ , n
k
+ ) distributions.
In other words, assign each thread to draw an individual
k
from its Gamma(y
k
+ , n
k
+ ) distribution.
2. Sample from its full conditional using a random walk
Metropolis step.
3. Sample from its full conditional (truncated Gamma)
using the inverse cdf method if b
0
is low or a
non-truncated Gamma if b
0
is high.
A codelsss
introduction to
GPU parallelism
Will Landau
A review of GPU
parallelism
Examples of
parallelism
Vector addition
Pairwise summation
K-means clustering
Markov chain Monte
Carlo
Preview: a bare bones CUDA C workow
#i n c l u d e <s t d i o . h>
#i n c l u d e <s t d l i b . h>
#i n c l u d e <cuda . h>
#i n c l u d e <cuda r unt i me . h>
g l o b a l voi d s ome ke r ne l ( . . . ) { . . . }
i n t mai n ( voi d ) {
// De c l ar e a l l v a r i a b l e s .
. . .
// Al l o c a t e hos t memory .
. . .
// Dynami cal l y a l l o c a t e de v i c e memory f o r GPU
r e s u l t s .
. . .
// Wr i t e t o hos t memory .
. . .
// Copy hos t memory t o de v i c e memory .
. . .
A codelsss
introduction to
GPU parallelism
Will Landau
A review of GPU
parallelism
Examples of
parallelism
Vector addition
Pairwise summation
K-means clustering
Markov chain Monte
Carlo
Preview: a bare bones CUDA C workow
// Execut e k e r n e l on t he de v i c e .
s ome ker nel <<< num bl ocks , num t he ads pe r bl oc k
>>>(...) ;
// Wr i t e GPU r e s u l t s i n de v i c e memory back t o
hos t memory .
. . .
// Fr ee dynami c al l y a l l o c a t e d hos t memory
. . .
// Fr ee dynami c al l y a l l o c a t e d de v i c e memory
. . .
}
A codelsss
introduction to
GPU parallelism
Will Landau
A review of GPU
parallelism
Examples of
parallelism
Vector addition
Pairwise summation
K-means clustering
Markov chain Monte
Carlo
Outline
Vector addition
Pairwise summation
K-means clustering
A codelsss
introduction to
GPU parallelism
Will Landau
A review of GPU
parallelism
Examples of
parallelism
Vector addition
Pairwise summation
K-means clustering
Markov chain Monte
Carlo
Resources
1. J. Sanders and E. Kandrot. CUDA by Example.
Addison-Wesley, 2010.
2. Prof. Jarad Niemis STAT 544 lecture notes.
A codelsss
introduction to
GPU parallelism
Will Landau
A review of GPU
parallelism
Examples of
parallelism
Vector addition
Pairwise summation
K-means clustering
Markov chain Monte
Carlo
Thats all for today.
Series materials are available at

http://will-landau.com/gpu.

Parallelism - Parallel Computing

Enviado por

Dados do documento

Título original

Direitos autorais

Formatos disponíveis

Compartilhar este documento

Compartilhar ou incorporar documento

Opções de compartilhamento

Você considera este documento útil?

Este conteúdo é inapropriado?

Direitos autorais:

Formatos disponíveis

Parallelism - Parallel Computing

Enviado por

Direitos autorais:

Formatos disponíveis

A codelsss

SIMD: apply the same command to multiple places in a

On CPUs, the iterations of the loop run sequentially.

With GPUs, we can easily run all 1,000,000 iterations

We can similarly parallelize a lot more than just loops.

The CPU (host) is in charge.

The CPU sends computationally intensive instruction

These threads are grouped into bunches called blocks.

The sum total of all threads in a kernel is called a grid.

CPU says: Hey, GPU. Sum pairs of adjacent numbers.

GPU thinks: Sum pairs of adjacent numbers is a

The GPU spawns 2 blocks, each with 2 threads:

I could have also used 1 block with 4 threads and given

Say I have 2 vectors,

I want to compute their component-wise sum,

Lets take the pairwise sum of the vector,

Synchronization: waiting for all parallel tasks to reach

Threads from the same block can be synchronized easily.

In general, do not try to synchronize threads from

The pairwise sum requires only log

Pairwise sum and pairwise multiplication are examples

Reduction: an algorithm that applies some binary

Scan (prex sum): an operation on a vector that

Example: computing the sequence of partial sums in

Write A in terms of its rows: A =

Write B in terms of its columns: B =

Compute C = A B by taking the product of each row of A with

Assign block (i , k) to compute c

Say I want to compute A B, where:

I write the multiplication as an array of products:

We dont need to synchronize blocks because they

Consider block (0, 0), which computes

Cluster N vectors in Euclidian space into K groups.

. . . and repeat until convergence.

Step 2: assign points to closest cluster centers.

Spawn N blocks with K threads each.

Let thread (n, k) compute the distance between data

Let thread (n, 1) assign data point n to its nearest

Step 3: recompute cluster centers.

Spawn one block for each cluster.

Within each block, compute the mean of the data in the

Consider a bladder cancer data set:

Available from http://ratecalc.cancer.gov/.

Rates of death from bladder cancer of white males from 2000 to

Also assume and are independent and x a

We want to sample from the joint posterior,

We iteratively sample from the full conditional distributions.

In other words, assign each thread to draw an individual

Series materials are available at

Você também pode gostar