Advances in the Solution of NS Eqs. in GPGPU Hardware. Second order scheme and experimental validation

Advances in NS solver on GPU por M.Storti et.al.
Advances in the Solution of NS Eqs. in GPGPU
Hardware
by Mario Stortiab
, Santiago Costarelliab
, Luciano Garellia
,
Marcela Cruchagac
, Ronald Ausensic
, S. Idelsohnabde
,
Gilles Perrotf
, Raphaël Couturierf
a
Centro de Investigación de Métodos Computacionales, CIMEC (CONICET-UNL)
Santa Fe, Argentina, mario.storti@gmail.com, www.cimec.org.ar/mstorti
b
Facultad de Ingenier´ıa y Ciencias H´ıdricas. UN Litoral. Santa Fe, Argentina, fich.unl.edu.ar
c
Depto Ing. Mecánica, Univ Santiago de Chile
d
International Center for Numerical Methods in Engineering (CIMNE)
Technical University of Catalonia (UPC), www.cimne.com
e
Institució Catalana de Recerca i Estudis Avançats
(ICREA), Barcelona, Spain, www.icrea.cat
f
FEMTO-ST Institute, Franche-Comté Meso-Centre
CIMEC-CONICET-UNL 1
((version texstuff-1.2.8-4-g7cfd85f Mon Apr 27 23:01:25 2015 -0300) (date Thu Apr 30 12:04:10 2015 -0300))

Scientific computing on GPUs
• Graphics Processing Units (GPU’s) are specialized hardware designed to
discharge computation from the CPU for intensive graphics applications.
• They have many cores (thread processors), currently the Tesla K40
(Kepler GK110) has 2880 cores at 745 Mhz (boost 875 Mhz).
• The raw computing power
is in the order of Teraflops
(4.3 Tflops in SP and
1.43 Tflops in DP).
• Memory Bandwidth
(GDDR5) 288 GiB/s.
Memory size 12 GiB.
• Cost: USD 3,100. Low end
version Geforce GTX Titan:
USD 1,500.
CIMEC-CONICET-UNL 2

Scientiﬁc computing on GPUs (cont.)
• The difference between the GPUs
architecture and standard multicore
processors is that GPUs have much
more computing units (ALU’s
(Arithmetic-Logic Unit) and SFU’s
(Special Function Unit), but few control
units.
• The programming model is SIMD
(Single Instruction Multiple Data).
• Branch-divergence happens when a
branch condition gives true in some in
some threads an false in others. Due
to the SIMD model all threads can’t be
executed simultaneously.
• GPUs compete with many-core
processors (e.g. Intel’s Xeon Phi)
Knights-Corner, Xeon-Phi 60 cores).
if (COND) {
BODY-TRUE;
} else {
BODY-FALSE;
}
CIMEC-CONICET-UNL 3

GPUs, MIC, and multi-core processors
• Computing power is limited by the number
of transistors we can put in a chip. This is
limited in turn to the largest silicon chip
you can grow (O(400 mm2
)), and the
minimum size of the transistors you can
make (O(22 nm)). So actually the limit is in
the order of 5 × 109
transistors per chip.
• So with that number of transistors you can
do one of the following
Have a few (6 to 12) very powerful
cores: Xeon E5-26XX processors.
Have 60 or more less powerful cores,
each one with 40 million transistors,
similar to a Pentium processor: Intel
Many Integrated Core (MIC), Xeon Phi
Have a very large (O(3000)) simple
(without control) cores: Nvidia GPUs.
Die shot of the Nvidia ”Kepler1”
GK104 GPU
CIMEC-CONICET-UNL 4

Solution of incompressible Navier-Stokes flows on GPU
• GPU’s are less efficient for algorithms that require access to the card’s
(device) global memory. In older cards shared memory is much faster but
usually scarce (16K per thread block in the Tesla C1060) .
• The best algorithms are those that make computations for one cell
requiring only information on that cell and their neighbors. These
algorithms are classified as cellular automata (CA).
• In newer cards there is a much larger memory cache. Anyway data
locality is fundamental.
• Lattice-Boltzmann and explicit F M (FDM/FVM/FEM) fall in this category.
• Structured meshes require less data to exchange between cells (e.g.
neighbor indices are computed, no stored), and so, they require less
shared memory. Also, very fast solvers like FFT-based (Fast Fourier
Transform) or Geometric Multigrid are available .
CIMEC-CONICET-UNL 5

Fractional Step Method on structured grids with QUICK
Proposed by Molemaker et.al. SCA’08: 2008 ACM SIGGRAPH, Low viscosity
ﬂow simulations for animation.
• Fractional Step Method
(a.k.a. pressure
segregation)
• u, v, w and continuity
cells are staggered
(MAC=Marker And
Cell).
• QUICK advection
scheme is used in the
predictor stage.
• Poisson system is
solved with IOP
(Iterated Orthogonal
Projection) (to be
described later), on top
of Geometric MultiGrid
j
j+1
j+2
i+1 i+2(ih,jh)
Vij
PijUij
x
y
i
y-momentum cell
x-momentum nodes
y-momentum nodes
continuity nodes
x-momentum cell
continuity cell
CIMEC-CONICET-UNL 6

Solution of the Poisson with FFT
• Solution of the Poisson equation is, for large meshes, the more CPU
consuming time stage in Fractional-Step like Navier-Stokes solvers.
• We have to solve a linear system Ax = b
• The Discrete Fourier Transform (DFT) is an orthogonal transformation
ˆx = Ox = ﬀt(x).
• The inverse transformation O−1
= OT
is the inverse Fourier Transform
x = OT ˆx = iﬀt(ˆx).
• If the operator matrix A is spatially invariant (i.e. the stencil is the same at
all grid points) and the b.c.’s are periodic, then it can be shown that O
diagonalizes A, i.e. OAO−1
= D.
• So in the transformed basis the system of equations is diagonal
(OAO−1
) (Ox) = (Ob),
Dˆx = ˆb,
(1)
• For N = 2p
the Fast Fourier Transform (FFT) is an algorithm that
computes the DFT (and its inverse) in O(N log(N)) operations.
CIMEC-CONICET-UNL 7

Solution of the Poisson with FFT (cont.)
• So the following algorithm computes the solution of the system to
machine precision in O(N log(N)) ops.
ˆb = fft(b), (transform r.h.s)
ˆx = D−1 ˆb, (solve diagonal system O(N))
x = ifft(ˆx), (anti-transform to get the sol. vector)
• Total cost: 2 FFT’s, plus one element-by-element vector multiply (the
reciprocals of the values of the diagonal of D are precomputed)
• In order to precompute the diagonal values of D,
We take any vector z and compute y = Az,
then transform ˆz = fft(z), ˆy = fft(y),
Djj = ˆyj/ˆzj.
CIMEC-CONICET-UNL 8

FFT computing rates in GPGPU. GTX-580 and Titan Black
1e+5 1e+6 1e+7 1e+8
0
100
200
300
400
500
600
580/DP
580/SP
TB/DP
TB/SP
GTX 580
GTX Titan Black
Launch Feb 2014, Arch: Kepler,
USD 1,500, 2880 cores, 5.12 Tflops SP
Launch June 2011, Arch: Fermi,
USD 1,000, 772 cores, 1.5 Tflops SP
GTX 580 DP
FFTproc.rate[Gflops/sec]
Ncell*t
64x64x64
128x64x64
128x128x64
128x128x128
256x128x128
256x256x128
256x256x256
512x256x256
GTX 580 SP
GTX TB DP
[*] Ncell=2*Nx*Ny*Nz because FFT is complex
GTX TB SP
CIMEC-CONICET-UNL 9

Upper bound on computing rate for the NS solver w/FFT
• For instance for a mesh of N = 2563
= 16Mcell the rate is 4.66 Gcell/s
(SP).
• So that one could perform one Poisson solution (two FFTs) in 4.2ms
(doesn’t take into account the computation of the RHS, which is
negligible).
• Poisson eq. is a large component of the computing time in the Fractional
Step Solver for Navier-Stokes (and the only one that is > O(N)) so that
this puts an upper limit on the computing rate achievable with our
algorithm (based on FFT).
CIMEC-CONICET-UNL 10

Solution of NS on embedded geometries: FVM/IBM1
• When solid bodies are
included in the fluid, we
have to deal with curved
surfaces that in general do
not fit with node planes.
• The first approach is to fit
the body surface by a
stair-case one (a.k.a. Lego
surface).
• Convergence is degraded
to 1st order.
• Stair-case acts like an
artificial roughness so it
tends to give higher drag.
solid body
fluid

Solving Poisson with embedded bodies
• FFT solver and GMG are very fast but have several restrictions: invariance
of translation, periodic boundary conditions.
• No holes (solid bodies) in the fluid domain.
• So they are not well suited for direct use on embedded geometries.
• One approach for the solution is the IOP (Iterated Orthogonal Projection)
algorithm.
• It is based on solving iteratively the Poisson eq. on the whole domain
(fluid+solid). Solving in the whole domain is fast, because algorithms like
Geometric Multigrid or FFT can be used. Also, they are very efficient
running on GPU’s .

The IOP (Iterated Orthogonal Projection) method
The method is based on succesively solve for the incompressibility condition
(on the whole domain: solid+ﬂuid), and impose the boundary condition.
on the whole
domain (fluid+solid)
violates impenetrability b.c.
satisfies impenetrability b.c.

The IOP (Iterated Orthogonal Projection) method (cont.)
• Fixed point iteration
wk+1
= ΠbdyΠdivwk
.
• Projection on the space of
divergence-free velocity fields:
u = Πdiv(u)
u = u − P,
∆P = · u,
• Projection on the space of velocity
fields that satisfy the
impenetrability boundary condition
u = Πbdy(u )
u = ubdy, in Ωbdy,
u = u , in Ωfluid.

Accelerated Global Preconditioning (AGP) algorithm
• Storti et.al. ”A FFT preconditioning technique for the solution of
incompressible flow on GPUs.” Computers & Fluids 74 (2013): 44-57.
• It shares some features with IOP scheme.
• Both solvers are based on the fact that an efficient preconditioning exists.
The preco. consists in solving the Poisson problem on the global domain
(fluid+solid, i.e. no “holes” in the fluid domain).
• IOP is a stationary method and its limit rate of convergence doesn’t
depend on refinement.
• However id does depend on (O(AR)) where AR is the maximum aspect
ratio of bodies inside the fluid. I.e. the method is bad when very slender
bodies are present.
• Our proposed method AGP is a preconditioned Krylov space method.
• The condition number of the preconditioned system doesn’t depend on
refinement but deteriorates with the AR of immersed bodies.
• IOP iterates over both the velocity and pressure fields, whereas AGP
iterates only on the pressure vector (which is better for implementation on
GPU’s).

MOC+BFECC
• The Method of Characteristics (MOC) is a Lagrangian method. The
position of the node (“particle”) is tracked backwards in time and the
values at the previous time step are interpolated into that position. These
projections introduce dissipation.
• BFECC Back and Forth Error Compensation and Correction can ﬁx this.
• Assume we have a low order (dissipative) operator (may be MOC, full
upwind, or any other) Φt+∆t
= L(Φt
, u).
• BFECC allows to eliminate the dissipation error.
Advance forward the state Φt+∆t,∗
= L(Φt
, u).
Advance backwards the state Φt,∗
= L(Φt+∆t,∗
, −u).

MOC+BFECC (cont.)
exact w/dissipation error
• If L introduces some dissipative error , then Φt,∗
= Φt
, in fact
Φt,∗
= Φt
+ 2 .
• So that we can compensate for the error:
Φt+∆t
= L(Φt
, ∆t) − ,
= Φt+∆t,∗
− 1/2(Φt,∗
− Φt
)
(2)

MOC+BFECC (cont.)
Nbr of Cells QUICK-SP BFECC-SP QUICK-DP BFECC-DP
64 × 64 × 64 29.09 12.38 15.9 5.23
128 × 128 × 128 75.74 18.00 28.6 7.29
192 × 192 × 192 78.32 17.81 30.3 7.52
Cubic cavity. Computing rates for the whole NS solver (one step) in [Mcell/sec] obtained with the
BFECC and QUICK algorithms on a NVIDIA GTX 580. 3 Poisson iterations were used.
Costarelli, et.al. ”GPGPU implementation of the BFECC algorithm for pure
advection equations.” Cluster Computing 17(2) (2014): 243-254.

Analysis of performance
• Regarding the performance results shown above, it can be seen that the
computing rate of QUICK is at most 4x faster than that of BFECC. So
BFECC is more efﬁcient than QUICK whenever used with CFL > 2, being
the critical CFL for QUICK 0.5. The CFL used in our simulations is typically
CFL ≈ 5 and, thus, at this CFL the BFECC version runs 2.5 times faster
than the QUICK version.
• Effective computing rate is more representative than the plain computing
rate
effective computing rate = CFLcrit ·
(number of cells)
(elapsed time)
• The speedup of MOC+BFECC versus QUICK increases with the number of
Poisson iterations. In the limit of very large number of iters (very low
tolerance in the tolerance for Poisson) we expect a speedup 10x (equal to
the CFL ratio).

Validation. A buoyant fully submerged sphere
• A fluid structure interaction problem has been
used for validation.
• Fluid is water ρfluid = 1000 [kg/m3
].
• The body is a smooth sphere D =10cm, fully
submerged in a box of L =39cm. Solid
density is ρbdy = 366 [kg/m3
] ≈ 0.4 ρfluid
so the body has positive buoyancy. The buoy
(sphere) is tied by a string to an anchor point
at the center of the bottom side of the box.
• The box is subject to horizontal harmonic
oscillations
xbox = Abox sin(ωt),

Spring-mass-damper oscillator
• The acceleration of the box creates a driving force on the sphere. Note
that as the sphere is positively buoyant the driving force is in phase with
the displacement. That means that if the box accelerates to the right, then
the sphere tends to move in the same direction. (launch video p2-f09)
• The buoyancy of the sphere produces a restoring force that tends to keep
aligned the string with the vertical direction.
• The mass of the sphere plus the added mass of the liquid gives inertia.
• The drag of the fluid gives a damping effect.
• So the system can be approximated as a (nonlinear) spring-mass-damper
oscillator
inertia
mtot ¨x +
damp
0.5ρflCdA|v| ˙x +
spring
(mflotg/L)x =
driving frc
mflotabox,
mtot = mbdy + madd, mflot = md − mbdy,
madd = Cmaddmd, Cmadd ≈ 0.5, md = ρflVbdy

Experimental results
• Experiments carried out atUniv Santiago de
Chile (Dra. Marcela Cruchaga).
• Plexiglas box mounted on a shake table,
subject to oscillations of 2cm amplitude in
freqs 0.5-1.5 Hz.
• Sphere position is obtained using Motion
Capture (MoCap) from high speed (HS) camera
(800x800 pixels @ 120fps). HS cam is AOS
Q-PRI, can do 3 MPixel @500fps, max frame
rate is 100’000 fps.
• MoCap is performed with a home-made code.
Finds current position by minimization of a
functional. Can also detect rotations. In this
exp setup 1 pixel=0.5mm (D=200 px). Camera
distortion and refraction have been evaluated
to be negligible. MoCap can obtain subpixel
resolution. (launch video mocap2)

Experimental results (cont.)
0
20
40
60
80
100
120
140
160
180
0.5 0.6 0.7 0.8 0.9 1 1.1 1.2 1.3
phase[deg]
freq[Hz]
anly(CD=0.3,Cmadd=0.55)
exp
0
0.02
0.04
0.06
0.08
0.1
0.4 0.6 0.8 1 1.2 1.4 1.6
Amplitude[m]
freq[Hz]
anly(CD=0.3,Cmadd=0.55)
exp
Best ﬁt of the experimental results with the simple spring-mass-damper
oscillator are obtained for Cmadd = 0.55 and Cd = 0.25. Reference values
are:
• Cmadd = 0.5 (potential ﬂow), Cmadd = 0.58 (Neill etal Amer J Phys
2013),
• Cd = 0.47 (reference value). (launch video description-buoy-resoviz)

abox
driving force
box desplacement
buoy displacement (f<<f0)
buoy displacement (f=f0)
buoy displacement (f>>f0) f
buoy (positive buoyancy)
pendulum (negative buoyancy)
abox
driving force
box desplacement
buoy displacement (f<<f0)
buoy displacement (f=f0)
buoy displacement (f>>f0)
f
driving force = (md − mbdy)abox (3)

Computational FSI model
• The fluid structure interaction is performed with a partitioned Fluid
Structure Interaction (FSI) technique.
• Fluid is solved in step [tn
, tn+1
= tn
+ ∆t] between the known position
xn
of the body at tn
and an extrapolated (predicted) position xn+1,P
at
tn+1
.
• Then fluid forces (pressure and viscous tractions) are computed and the
rigid body equations are solved to compute the true position of the solid
xn+1
.
• The scheme is second order precision. It can be put as a fixed point
iteration and iterated to convergence (enhancing stability).

Computational FSI model (cont.)
• The simulation is performed in a non-inertial frame attached to the box.
So, according to the Galilean equivalence principle we have just to add an
inertial force due to the acceleration of the box.
• Due to the incompressibility this is absorbed in an horizontal flotation
force on the sphere, which is the driving force for the system.
• The rigid body eqs. are now:
mbdy ¨x + (mflotg/L)x = Ffluid + mflotabox
where Ffluid = Fdrag + Fadd are now computed with the CFD FVM/IBM1
code.

Why use a positive buoyant body?
• Gov. equations for the body:
mbdy ¨x + (mflotg/L)x =
comp w/CFD
Ffluid({x}) +mflotabox
where Ffluid = Fdrag + Fadd are fluid forces computed with the FVM/IBM
code.
• The most important and harder from the point of view of the numerical
method is to compute the the fluid forces (drag and added mass).
• Buoyancy is almost trivial and can be easily well captured of factored out
from the simulation.
• Then, if the body is too heavy it’s not affected by the fluid forces, so we
prefer a body as lighter as possible.
• In the present example mbdy = 0.192[kg], md = 0.523[kg],
madd = 0.288[kg]

Comparison of experimental and numerical results
0
0.02
0.04
0.06
0.08
0.1
0.5 0.6 0.7 0.8 0.9 1 1.1 1.2 1.3
Amplitude[m]
freq[Hz]
experimental
FVM/IBM1
0
50
100
150
200
0.5 0.6 0.7 0.8 0.9 1 1.1 1.2 1.3Phase[deg]
freq[Hz]
experimental
FVM/IBM1

Comparison of experimental and numerical results
0
0.02
0.04
0.06
0.08
0.1
0.5 0.6 0.7 0.8 0.9 1 1.1 1.2 1.3
Amplitude[m]
freq[Hz]
experimental
FVM/IBM1
0
50
100
150
200
0.5 0.6 0.7 0.8 0.9 1 1.1 1.2 1.3
Phase[deg]
freq[Hz]
experimental
FVM/IBM1
• Results obtained with mesh 2563
= 16.7 Mcell.
• Maximum error 22% at resonance (f = 0.9Hz).
• Added mass an buoyancy cancel out at resonance and amplitude is
almost controlled by drag. Hence drag is not well computed.
• Numerical amplitude is lower than experimental =⇒ numerical drag is
higher than experimental.
• Probable cause: staircase representation of surface acts as artiﬁcial
roughness and increases drag.
• Resonance frequency seems correct =⇒ added mass and buoyancy
forces are correct.

Convergence of FVM/IBM1
• Plot shows amplitude vs mesh size at resonance (f = 0.9Hz).
• The scheme is O(h) convergent.
• Need for a O(h2
) scheme with computational efﬁciency comparable to
the 1st order one (FVM/IBM1).
0 0.001 0.002 0.003 0.004 0.005
4
5
6
7
8
9
10
mesh size h [m]
Abuoy[m]
FVM/IBM1
experimental
256^3 = 16.7 Mcell
192^3 = 7.1 Mcell
128^3 = 2.1 Mcell
96^3 = 0.9 Mcell
actual Lego sphere
with O(40^3) blocks

Immersed boundary second order scheme (FVM/IBM2)
• Fadlun, et al. ”Combined immersed-boundary...” JCP (2000)
• Hard to combine with staggered grids and Fractional Step schemes.
• So switched to Rhie-Chow interpolation for using co-located grids (i.e.
avoiding staggered grids)
• Switch to SIMPLE algorithm for iterating the pressure-velocity coupling.
• Bodies are represented by level set functions. This allows for easy
computation of boundary operators without branch-divergence.

Immersed boundary second order scheme (FVM/IBM2) (cont.)
1: Data initialization: u0
, p0
, q0
, ˙q0
2: for n = 0, Nstep do
3: u∗
← un
, p∗
← pn
4: for k = 0, Nsimple do
5: w ← u∗
, f ← 0
6: for l = 0, NIBM do
7: w = Gp∗
+ R(w) + f; ¯w = ΠBC(w, qn
S , ˙qn
S )
8: f = ¯w − Gp∗
− R(w), f = f − f / fl=0
9: f = f , w = ¯w
10: If f < tolf break
11: end for
12: u ← ¯w
13: Solve A(p − p∗
) = A p∗
+ Du for p
14: u = u − u∗
/ u − un
15: u∗
← αuu + (1 − αu)u∗
; p∗
← p∗
+ αpp
16: If u < tolu break
17: end for
18: un+1
← u∗
, pn+1
← p∗
19: Compute the tractions of the ﬂuid onto the solid tfl
20: Compute the new solid position qn+1
S and velocity ˙qn+1
S
21: end for

The boundary projection operator
• Given a solid cell node S we have to determine
the fictitious (a.k.a. parasitic) velocity uf
S such
that it guarantees the satisfaction of the
boundary condition at B
• The closest point xB to S is determined as
xS = xB − dnB.
• d and nB can be determined from the level set
function ψ
nB = ( ψ/ ψ )B ≈ ( ψ/ ψ )S ,
d = ψS,
• Mirror fluid point to S is xF = xB + dnS
• Velocity at mirror point is determined from
extrapolation uf
S = 2uB − uF (*)
• Forcing at S is iteratively found such that (*) is
satisfied.
Solid cell center Fluid cell center
ψ<0 (solid)
ψ>0 (fluid)
ψ=0
S
B
nB
^
F
d
d

Numerical results with FVM/IBM2
0.4 0.6 0.8 1 1.2 1.4 1.6
0
0.02
0.04
0.06
0.08
0.1
f [Hz]
Ab[m]
experimental [m]
FVM/IBM1[m]
FVM/IBM2[m]
experimental
FVM/IBM1 256^3 = 16.7 Mcell
0.4 0.6 0.8 1 1.2 1.4 1.6
0
50
100
150
f [Hz]
phi[deg]
experimental [m]
FVM/IBM1[m]
FVM/IBM2[m]
experimental

Convergence of FVM/IBM2
• FVM/IBM2 exhibits O(h2
) convergence.
0 0.001 0.002 0.003 0.004 0.005
4
5
6
7
8
9
10
mesh size h [m]
FVM/IBM1
FVM/IBM2
experimental
Abuoy[m]

Convergence of FVM/IBM2 (cont.)
• Results for FVM/IBM2 with 1923
are much better than FVM/IBM1 with a
finer mesh 2563
.
• Time step for FVM/IBM1 was chosen so that CFL = 0.2 (CFLcrit = 0.5,
QUICK version).
• Tolerance for Poisson 10−6
.
• Results are almost converged for NIBM = 3, Nsimple = 5.
• For FVM/IBM2 we used tolf = 1 × 10−3
, tolu = 2 × 10−3
(typically
NIBM = 1, Nsimple = 3). CFL = 0.1 (∆t = 2 × 10−4
s). (Remember
CFLcrit = 0.5 − 0.6).
• Results have been fitted with the analytic model giving Cd,num = 0.23,
experimental= 0.25 (reference value for sphere drag is Cd = 0.47).
• Added mass coefficient: FVM/IBM2: Cmadd = 0.58, experimental = 0.58,
reference: 0.55, potential theory: 0.5.

Computational efﬁciency of FVM/IBM2 on GPGPUs
• Efﬁciency of the above algorithm has been measured on Nvidia Tesla
K40c, and GTX Titan Black.
• K40c has 2880 CUDA cores @876 MHz, 12 GiB ECC RAM. GK110 model
(Kepler arch). (It’s equivalent with K40 but has builtin fansink). Cost:
O(USD 1500).
• GTX Titan Black has 2880 cores @980 GHz with 6 GiB RAM (no ECC).
Cost: O(USD 3100)

Tesla K40c vs GTX Titan Black
GTX Titan Black
2880 cores @ 980MHz2880 cores @ 875MHz
235W 250W
12 GiB ECC 6 GiB (no ECC)
7.10e9 7.08e9
4.29 Tflop/s 5.12 Tflop/s
336 GB/s288 GB/s
cores /Turbo speed
dissipation
device RAM
# transistors
memory bandwidth
SP performance
TESLA
K40
+12%
+7%
0.5x
+19%
=
+17%
cost (aprox.) USD 3,100 USD 1,500 0.5x

Computational efﬁciency of FVM/IBM2 on GPGPUs
2e+06 4e+06 6e+06 8e+06 1e+07 1.2e+07
200
400
600
800
1000
1200
1400
Ncell
computingrate[Mcell/s]
GTX,SP,NSIMPLE=1
K40c,SP,NSIMPLE=1
GTX,DP,NSIMPLE=1
K40c,DP,NSIMPLE=1
K40c,SP,NSIMPLE=3
GTX,DP,NSIMPLE=3
K40c,DP,NSIMPLE=3
GTX,SP,NSIMPLE=3

Computing rates for FVM/IBM2
• Maximum attained rate is
1.3 Gcell/s for GTX Titan Black in
SP, Nsimple = 1, mesh of
2243
= 11.2 Mcell.
• Computing rates are monotonically
increasing with reﬁnement in all
cases.
• Comp. rates are roughly
∝ (NsimpleNIBM)−1
• Titan Black outperforms K40c for
both SP and DP.
• Remarkable good performance of
GTX Titan Black specially in DP.
2e+06 4e+06 6e+06 8e+06 1e+07 1.2e+07
0
200
400
600
800
1000
1200
Ncell
computingrate[Mcell/s]
GTX,SP,NSIMPLE=1
K40c,SP,NSIMPLE=1
GTX,DP,NSIMPLE=1
K40c,DP,NSIMPLE=1
GTX,SP,NSIMPLE=3
K40c,SP,NSIMPLE=3
GTX,DP,NSIMPLE=3
K40c,DP,NSIMPLE=3

Computing rates for FVM/IBM2 (cont.)
• For 1923
(the reﬁnement used for the results presented above) we can
compute 1s simulation in 30s of computation (ST:RT=1:30).
• We can estimate un upper bound for the computing rate counting the
transactions to the device RAM.
• We have 66 transactions (ﬂoats/doubles) per cell. So the theoretical limit is
limit rate (SP) =
336GB/s
66 · 4 Bytes/cell
= 1.27 Gcell/s,
limit rate (DP) =
336GB/s
66 · 8 Bytes/cell
= 636 Mcell/s,
(4)

Time consumption breakdown

Numerical results
Vorticity isosurface
velocity at sym plane
tadpoles
colored vort isosurfaces
(launch video fsi-boya-gpu-vorticity), (launch video gpu-corte)
(launch video fsi-vort-3D), (launch video particles-buoyp5)

FVM/IBM vs. LBM on GPGPU
• In LBM the Poisson equation is not solved; it uses
pseudo-compressibility, hence LBM is a true Cellular Automata (CA)
algorithm. (FVM/IBM could so too, of course )
• So comparatively it can reach higher computing rates, but suffers from the
artificial Mach penalization effect.
• Suppose you have a maximum velocity umax in the flow field.
• If you use pseudo-compressibility then you must choose an artificial
speed of sound cart > umax.
• The compressibility effects are O(M2
art) (Mart = umax/cart) so you
choose, let’s say, Mart = 0.1 so that the error is O(10−2
).
∆t < CFLcrit · h/(umax + cart)
= Mart/(1 + Mart) · h/umax ≈ Mart · h/umax,
CFL < Mart · CFLcrit,
• So your CFL is penalized by a factor Mart (0.1 in the example).

FVM/IBM vs. LBM on GPGPU (cont.)
• So in fact not only Mcell/s must be reported, but also the critical CFL for
the algorithm.
• So if the computing rate in Mcell/s is doubled but have a penalization of
0.5x in the CFL the algorithm breaks even.
• Or rather the effective computing rate should be reported
effective computing rate = CFLcrit ·
(number of cells)
(elapsed time)
• The presented algorithm has (experimentally determined)
CFLcrit = 0.5 − 0.6.

Future work
• We are in the process of buying a HPC cluster (≈ 20 Tﬂops, w/o GPGPUs)
with funds from Provincia de Santa Fe Science Agency ASACTEI (Thanks!
). Includes nodes w/GTX Titan Z (2x2880=5760 cores @876 MHz
(boost), 12 GiB RAM, equiv to 2 Titan (not Black! ) on the same card).
We estimate reaching 2 Gcell/s on one card.
• Replace momentum eqs. solver with MOC+BFECC.
• Multi-GPU: (hybrid MPI+CUDA+OpenMP) Some experiments already done
with Poisson eq.
• Reﬁnement with multi-block strategy.
• LES Smagorinsky turbulence model.

Acknowledgments
• Chilean Council for Scientific and Technological Research (CONICYT-FONDECYT 1130278);
• The Scientific Research Projects Management Department of the Vice Presidency of
Research, Development and Innovation (DICYT-VRID) at Universidad de Santiago de Chile;
• Association of Universities: Montevideo Group (AUGM);
• Argentinean Council for Scientific Research (CONICET project PIP 112-20111-00978);
• Argentinean National Agency for Technological and Scientific Promotion, ANPCyT, (grants
PICT-E-2014-0191, PICT-2014-0191);
• Universidad Nacional del Litoral, Argentina (grant CAI+D-501-201101-00233-LI);
• Santa Fe Science Technology and Innovation Agency (grant ASACTEI-010-18-2014); and
• European Research Council (ERC) Advanced Grant Real Time Computational Mechanics
Techniques for Multi-Fluid Problems (REALTIME, Reference: ERC-2009-AdG).
• We made extensive use of Free Software as GNU/Linux OS, GCC/G++ compilers, Octave, and
Open Source software as VTK, ParaView, among many others.

Advances in the Solution of NS Eqs. in GPGPU Hardware. Second order scheme and experimental validation

More Related Content

What's hot (20)

Viewers also liked (6)

Similar to Advances in the Solution of NS Eqs. in GPGPU Hardware. Second order scheme and experimental validation (20)

More from Storti Mario (10)

Recently uploaded (20)

Advances in the Solution of NS Eqs. in GPGPU Hardware. Second order scheme and experimental validation