Myers_SIAMCSE15

Controlling Numerical Error in Particle-in-Cell
Simulations of Collisionless Dark Matter
Andrew Myers
atmyers@lbl.gov
Applied Numerical Algorithms Group, Computational Research Division
with Phillip Colella, Brian Van Straalen
SIAM-CSE Meeting
March 17th, 2015
Extreme Resilient Discretizations
Submitted to ApJ
Tuesday, March 17, 15

• Motivation - why is understanding these errors relevant for us here?
• Standard PIC methods don’t converge for Cosmology applications
• Two modifications:
– Regularization
– Adaptive Remapping
• Summary and Future Research
Controlling Numerical Error in Particle-in-Cell Simulations of Collisionless Dark Matter
Andrew Myers, LBNL
2
Talk outline

– Regularization
Andrew Myers, LBNL
3
Talk outline

Roofline Performance Model
Cori Node
Andrew Myers, LBNL
4
Roofline Model - High Arithmetic Intensity needed
for maximum performance

• Field solve: force is computed on the mesh by e.g.
solving Poisson’s Equation w/ 2nd order finite
differences.
• Interpolation: Force is interpolated back to particle
positions using same kernel.
• Particle Push: Particle positions and velocities are
updated. 2nd-order leapfrog.
• Deposition: Particle masses are deposited onto mesh:
Andrew Myers, LBNL
5
This is a problem for 2nd Order PIC Methods
⇢n+1
i =
X
p
✓
mp
Vi
◆
W
xi xn+1
p
x
!
2nd order: Piecewise linear, Cloud-in-Cell interpolation
Start
Deposition
Field Solve
Interpolation
Particle Push

• Poisson solve is a global bottleneck. Theoretical peak AI is bad.
• Even if we read in a chunk of particles and do all the work we possibly can before
moving on to the next chunk, by:
• Reading in a batch of particles
• Subtracting of their contribution to the density
• Interpolating the field to the particle positions
• Pushing the particles
• Depositing the particles at their new positions
AI .
1
1 + 1/nppc
Andrew Myers, LBNL
6
Performance problems: global bottleneck, poor AI
In 1D,
perfect cache:
24 Flops,
3 doubles per
particle,
3 doubles per
cell
1 for high ppc (convergence)
1/2 for 1 particle per cell

Andrew Myers, LBNL
7
Improving AI - Higher Order in Space
B a0 = 1
˜Wq(k)
Wq(x) =
q/2 1
X
p=0
a2p( 1)p
M(2p)
q (x),
B
[ q/2, q/2]
Wq q = 4, 6
W4(x) =
8
><
>:
|x|3
2 |x|2 |x|
2 + 1, |x| 2 [0, 1],
|x|3
6 + |x|2 11|x|
6 + 1, |x| 2 [1, 2],
0,
W6(x) =
8
>>>><
>>>>:
|x|5
12 + |x|4
4 + 5|x|3
12
5|x|2
4
|x|
3 + 1, |x| 2 [0, 1],
|x|5
24
3|x|4
8 + 25|x|3
24
5|x|2
8
13|x|
12 + 1, |x| 2 [1, 2],
|x|5
120 + |x|4
8
17|x|3
24 + 15|x|2
8
137|x|
60 + 1, |x| 2 [2, 3],
0,
k
k = 4
q
q
[ q/2, q/2]
B a0 = 1
˜Wq(k)
Wq(x) =
q/2 1
X
p=0
a2p( 1)p
M(2p)
q (x),
B
[ q/2, q/2]
Wq q = 4, 6
W4(x) =
8
><
>:
|x|3
2 |x|2 |x|
2 + 1, |x| 2 [0, 1],
|x|3
6 + |x|2 11|x|
6 + 1, |x| 2 [1, 2],
0,
W6(x) =
8
>>>><
>>>>:
|x|5
12 + |x|4
4 + 5|x|3
12
5|x|2
4
|x|
3 + 1, |x| 2 [0, 1],
|x|5
24
3|x|4
8 + 25|x|3
24
5|x|2
8
13|x|
12 + 1, |x| 2 [1, 2],
|x|5
120 + |x|4
8
17|x|3
24 + 15|x|2
8
137|x|
60 + 1, |x| 2 [2, 3],
0,
k
k = 4
q
q
[ q/2, q/2]
• Replace CIC with higher-order interpolation kernels
• Discrete delta approximations of any order we want (Lo, Minden, Colella
2015, in prep)
4 x AI 8 x AI

Andrew Myers, LBNL
8
High-order, time integrators with fewer bottlenecks -
Example: RK4
• 4th order RK methods require 3 or 4 force evaluations per time step, and 2
or 3 particle pushes. These must be done sequentially.
• An alternative is to store the force evaluations from the RK4 stages of the
previous time step at grid points, and extrapolate to get approximate forces
with which to compute the displacements for your next time step.
• Cheap with many p.p.c.
tn t tn
1
2
t tn +
1
2
t tn + ttn
f(tn t) ˜f(tn + t)
f(tn
1
2
t) ˜f(tn +
1
2
t)
f(tn)
- past
- future

Andrew Myers, LBNL
9
• Here is an example for a velocity-independent force using a 2nd order
interpolating polynomial to extrapolate the forces.
If we use these approximate force to compute the displacements, we have
kn+1
1 = F (tn
, xn
)
kn+1
2 = F
✓
tn
+
1
2
t, xn
+
1
2
vn
t +
1
8
˜f(1) t2
◆
kn+1
3 = F
✓
tn
+ t, xn
+ vn
t +
1
2
˜f
✓
3
2
◆
t2
◆
(7)
An alternative time-stepping scheme with better arithmetic intensity is
thus
kn+1
1 = F (tn
, xn
)
kn+1
2 = F
✓
tn
+
1
2
t, xn
+
1
2
vn
t +
1
8
kn
3 t2
◆
kn+1
3 = F
✓
tn
+ t, xn
+ vn
t +
1
2
(kn
1 3kn
2 + 3kn
3 ) t2
◆
xn+1
= xn
+ vn
t +
1
6
kn+1
1 + 2kn+1
2 t2
vn+1
= vn
+
1
6
kn+1
1 + 4kn+1
2 + kn+1
3 t. (8)
This is quite similar to the classical method, except that the displacements
for all three k values can be computed without doing any force solves.
Using Taylor expansions, one can verify that this is still 4th-order accu-
rate. In fact, one could also use linear extrapolation for ˜f(⌧) with any two
of the RK stages from time step n 1 and still retain 4th-order accuracy.
2 Stability Analysis
To check for stability, we consider the linearized model system:
˙x = v
˙v = x. (9)
The numerical scheme in equation (7), applied to (8), can be written in
a compact form if we consider the lagged forces kn
1 , kn
2 , and kn
3 to be part
The Method
goal is to solve the following system of equations for the particle posi-
and velocities x and v given the force F:
˙x = v
˙v = F(t, x). (1)
The standard, fourth-order Runge-Kutta method applied to this system
, for the special case of a force that does not depend on v:
xn+1
= xn
+ vn
t +
1
6
(k1 + 2k2) t2
vn+1
= vn
+
1
6
(kn
1 + 4k2 + k3) t, (2)
e
k1 = F (tn
, xn
)
k2 = F
✓
tn
+
1
2
t, xn
+
1
2
vn
t +
1
8
k1 t2
◆
k3 = F
✓
tn
+ t, xn
+ vn
t +
1
2
k2 t2
◆
. (3)
Note the sequential nature of this algorithm; k1 must be computed before
which must be computed before k3. An alternative is to extrapolate the
s from the previous time step. For example, the forces at the stages
sponding to times tn t, tn 1
2 t, and tn were kn
1 , kn
2 , and kn
3 .
ning
⌧ =
t (tn t)
t
, (4)
d order interpolating polynomial that passes through the required points
˜f(⌧) = (2kn
1 4kn
2 + 2kn
3 ) ⌧2
+ ( 3kn
1 + 4kn
2 kn
3 ) ⌧ + kn
1 . (5)
= 0, 1/2, and 1, we recover kn
1 , kn
2 , and kn
3 , respectively. Extrapolated
ard to ⌧ = 1, 3/2, and 2 (t = tn, tn + 1/2 t, and tn + t), we ﬁnd:
˜f(1) = kn
3
˜f
✓
3
2
◆
= kn
1 3kn
2 + 3kn
3
˜f(2) = 3kn
1 8kn
2 + 6kn
3 . (6)
he Method
l is to solve the following system of equations for the particle posi-
d velocities x and v given the force F:
˙x = v
˙v = F(t, x). (1)
standard, fourth-order Runge-Kutta method applied to this system
r the special case of a force that does not depend on v:
xn+1
= xn
+ vn
t +
1
6
(k1 + 2k2) t2
vn+1
= vn
+
1
6
(kn
1 + 4k2 + k3) t, (2)
k1 = F (tn
, xn
)
k2 = F
✓
tn
+
1
2
t, xn
+
1
2
vn
t +
1
8
k1 t2
◆
k3 = F
✓
tn
+ t, xn
+ vn
t +
1
2
k2 t2
◆
. (3)
the sequential nature of this algorithm; k1 must be computed before
h must be computed before k3. An alternative is to extrapolate the
om the previous time step. For example, the forces at the stages
nding to times tn t, tn 1
2 t, and tn were kn
1 , kn
2 , and kn
3 .
⌧ =
t (tn t)
t
, (4)
der interpolating polynomial that passes through the required points
All the right hand sides for Poisson can be computed at once for a
subset of particles that will fit in cache. Still 4th order accurate. Real
issue is time step.
3 - 4 x AI
High-order time integrators, w/ fewer bottlenecks -
Example: Extrapolating RK4
Total for going to high order: ~ 10 x

Andrew Myers, LBNL
10
Another important question: Will these techniques
actually give better results?
• Need to evaluate the utility of these methods on realistic problems
• For plasma PIC, yes. Convergence theory and empirical evidence from Wang, Miller,
and Colella 2011. Need remapping and high particle counts (100-1000 particles per
cell).
1e-06
1e-05
0 5 10 15 20 25 30
t
hx=L/64, hv=vmax/128
(a)
-1
0 5 10 15 20 25 30
co
t
(b)
Fig. 10. Error and convergence rate plots for the two-stream instability without remapping.
We set rh = 1/2. Scales (hx, hv) denote the particle grid mesh spacing at the base level. (a) The
L∞ norm of the electric field errors on three different resolutions. (b) The convergence rates for
the errors on plot (a). Second-order convergence rates are lost around t = 20.
1e-06
1e-05
0.0001
0.001
0.01
0.1
0 5 10 15 20 25 30
error
t
(a)
-1
0
1
2
3
0 5 10 15 20 25 30
convergencerate
t
(b)
Fig. 11. Error and convergence rate plots for the two-stream instability with remapping. We
set rh = 1/2. Scales (hx, hv) denote the particle grid mesh spacing at the base level. (a) The L∞
norm of the electric field errors on three different resolutions. (b) The convergence rates for the
Wang+2011

Andrew Myers, LBNL
11
For Cosmology, Evidence Suggests Not...
Romain TeyssierComputational Astrophysics 2009
High-resolution with constant force softening
Particle discreteness effects show up
quite dramatically in Warm Dark
Matter simulations (from Wang &
White, MNRAS, 2007)
Very slow convergence N^(-1/3)
These effects can be neglected if
d: local inter-particular spacing
Adaptive force softening ?
Splinter, R.J., Melott, A.L., Shandarin, S.F., Suto, Y., “ Fundamental Discreteness
Limitations of Cosmological N-Body Clustering Simulations”, ApJ, 497, 38, (1998)
Romeo, A.B., Agertz, O., Moore, B., Stadel, J., “Discreteness Effects in ΛCDM
Simulations: A Wavelet-Statistical View”, ApJ, 686, 1, (2008)
maintain the strict planar symmetry of the pancake collapse that
is apparently the root cause of the problem. To reiterate, the 2563
and 5123
PM runs roughly span the force resolutions used for
the other codes, and since only the 5123
run shows a very mild
failure of convergence, force resolution alone cannot be the source
of the difficulty.
Our results provide a different and more optimistic interpreta-
tion of the findings of Melott et al. (1997; see also Binney 2004).
While high-resolution codes when run with small smoothing
lengths (or several refinement levels in the case of AMR) are
not able to pass the pancake test after the formation of several
caustics, the main culprit appears to be an inability to maintain
the planar symmetry of the problem and not direct collisionality
(at least at the force resolutions relevant for this paper), which
Fig. 3.—Failure of convergence near the midplane for the pancake test: MC2
results, 643
particles with four grid sizes at z ¼ 0. Convergence fails at the final
resolution reduction step (going from a 2563
mesh to a 5123
mesh). See the text
for a discussion of these results.
F COSMOLOGICAL SIMULATIONS. I. 33
Wang + White 2007
Heitmann+2005
Also: “Demonstrating Discreteness and Collision Error in
Cosmological N-body Simulations of Dark Matter
Gravitational Clustering” - Melott + 1997
Need to be addressed before
benefiting from high order

– Regularization
Andrew Myers, LBNL
12
Talk outline

• Important point: all cosmology simulations are run with singular
initial conditions:
@f
@t
=
v
a
·
@f
@x
+
✓
˙a
a
◆
v +
1
a
r ·
@f
@v
Andrew Myers, LBNL
13
Vlasov-Poisson for Cosmology Simulations
f(x, v, tini) = ⇢(x, tini) (v ¯v)
• Low particle counts.
• Done for sound physical reasons, numerically problematic.
• We use PIC to solve this. Run a “Zel’dovich Pancake” setup.
• First, we do a 1D problem.

Andrew Myers, LBNL
14
The Zel’dovich Pancake - 1D Convergence Results
• Convergence is bad
after particle
trajectories cross
• Poor convergence
rates in 1D hint at
more serious
problems in higher
dimensions...

Andrew Myers, LBNL
15
The Zel’dovich Pancake - 2D, Tilted Results
• Spurious
fragmentation
regardless of the
number of particles
per Poisson cell
• Does not respect the
initial symmetry of
the problem setup
• Suggestive of Wang
+White 2007
1/4
1
256

– Regularization
Andrew Myers, LBNL
16
Talk outline

• Remove the singularity in the initial data
• Natural approach is to regularize the initial conditions via a finite, artificial
initial velocity dispersion, , for which we choose a Gaussian form:i
(v ¯v) !
✓
1
2⇡ i
2
◆D/2
exp

(v ¯v, tini))2
2 i
2
Andrew Myers, LBNL
17
Regularized Initial Conditions
• Makes things look more like plasma case. Many particles per cell.
• Analogy with shock-capturing schemes in gas dynamics is instructive.

lim
i!1
✓
1
2⇡ i
2
◆D/2
exp

(v ¯v, tini))2
2 i
2
= (v ¯v)
Andrew Myers, LBNL
18
i = 0.2i = 0.4i = 0.8

• For finite , we
do obtain the
expected order of
accuracy.
i
Andrew Myers, LBNL
19
Regularized Convergence Results - 1D

• This approach gives us a way to obtain
solutions to the original, cold problem
• For a given , increase resolution until a
converged solution is obtained.
• Then, look to see how the solutions
behave as .
• Inspired by a similar technique in vortex
methods
i
i ! 0
Andrew Myers, LBNL
20
The Double Limit
tion asmay be seenby comparison with the S = 400 solution in the iast panel
3~It is therefore presumed that the curves in Figs 2 and 3 arc essentially The
n of the 6 equations (l), (2) for the two particular values of 6 chosen, over
me intervat 0 d t 6 4. Comparable accuracy can be obtained at later times by
smaller 3I and larger N.
effect of decreasing 6 at a fixed time (I = 1) greater than the vortex sheet’s
time (TV= 0.375) is shown in Fig. 5 which plots the interpolating curve fc:-
al values of 6 between 0.2 and 0.05. These calculations used N = 406 and
b x 1
FIG. 5. Solution of the 6 equations (1 1. (2) at I = 1.0 using 6 = 0.2. 0.15, 0.1. 0.05
Krasny, 1986

Andrew Myers, LBNL
21
The Double Limit
• The converged, regularized solutions approach a well-defined curve.
• Artificial smooths out structures smaller than some length scale
• In practice, pick a length scale below which you won’t believe the results
i

Andrew Myers, LBNL
22
Regularized Results - 2D
• Regularization works in 1D. However, the problem with fragmentation in
2D persists...

• The failure of basic PIC for Cosmology applications
– Regularization
Andrew Myers, LBNL
23
Talk outline

eE
(x, t) / exp (at)
Andrew Myers, LBNL
24
Particle Remapping
High-order deposition
Positivity not guaranteed, need
mass distribution
• In plasma convergence theory, error for
field contains exponential term:
Before remap After remap
• Periodically restart problem with new particles
Wang+2011
Particles with tiny
masses are discarded
Requires regularization

• Wrinkle: In comoving
coordinates, velocities
shrink with time.
• shrinks as box
expands, must as well
• Solution: remap with AMR
• Resolves with same #
of particles throughout
• Example, 4 levels
v
Andrew Myers, LBNL
25
Particle Remapping, with AMR

Andrew Myers, LBNL
26
Remapping preserves order of method in 1D...
• Once this is
done, still get
2nd order in 1D

Andrew Myers, LBNL
27
And greatly improves artificial fragmentation issue
RemappedNot remapped
a = 0.013 levels,

• The failure of basic PIC for Cosmology applications
– Regularization
Andrew Myers, LBNL
28
Talk outline

Andrew Myers, LBNL
29
Conclusions and Future Research
• We know how to make PIC converge on Cosmology problems at the
stated order of accuracy. Can now benefit from high-order PIC.
Interpolation kernels, etc. for doing so are there.
• The necessary scheme looks a lot like PIC for electrostatic plasmas:
with particle remapping and high particle counts.
• We can exploit this information for designing high-AI methods. Example
- extrapolating RK4.
• Results on the convergence of PIC schemes for cosmology have been
submitted to ApJ, paper and code available here:
https://bitbucket.org/atmyers/cosmologicalpic

Thank you for listening!
Controlling Numerical Error in Particle-in-Cell
Simulations of Collisionless Dark Matter
Andrew Myers
atmyers@lbl.gov
Applied Numerical Algorithms Group, Computational Research Division
with Phillip Colella, Brian Van Straalen
SIAM-CSE Meeting
March 17th, 2015 Extreme Resilient Discretizations
Submitted to ApJ

• VP equation is a non-linear advection equation in phase space
• Can be solved using Eulerian methods in phase space on up to
128^6 domains (Yoshikawa + 2013)
• Expense of working in high-dimensional spaces is significant, both in
terms of memory requirements and the number of operations
involved.
• Large range of scales involved implies that adaptivity is usually
required.
Andrew Myers, LBNL
31
Eulerian Methods

f(x, v, tini) ⇡
X
p2P
mp x xi
p v vi
p
P
dmp
dt
= 0
dxp
dt
=
1
a
vp
dvp
dt
=
˙a
a
vp +
1
a
gp
(xp(t), vp(t))
vp
gp
Andrew Myers, LBNL
32
Particle Methods
• Discretize system with set of Lagrangian interpolating points,
• Reduces problem to system of ODEs for particle trajectories:
• Can reconstruct distribution at later times from
xp(t)

f(x, v, tini) ⇡
X
p2P
mp x xi
p v vi
p
P
dmp
dt
= 0
dxp
dt
=
1
a
vp
dvp
dt
=
˙a
a
vp +
1
a
gp
(xp(t), vp(t))
vp
gp
Andrew Myers, LBNL
33
Particle Methods
• Discretize system with set of Lagrangian interpolating points,
• Reduces problem to system of ODEs for particle trajectories:
“Viscous drag”
term associated
with comoving
coordinate
system
• Can reconstruct distribution at later times from
xp(t)

• Naturally adaptive
• Do not require keeping track of full, phase-space distribution function
• Basically all of the workhorse Dark Matter codes take this approach (e.g.
Enzo, Flash, Nyz, RAMSES, Gadget, ART, CHARM)
• Differ mainly in the way they compute gp
Andrew Myers, LBNL
34
Particle Methods

Andrew Myers, LBNL
35
2nd order PIC for Cosmology
Start
Initialize Particles
EndTime to stop?
Particle Kick
Particle Drift
Particle Deposition
Poisson Solve
Force Interpolation
Particle Kick
yes
no
• Deposition / Interpolation handled by CIC
• Poisson’s equation solved w/ 2nd order FD
• Kick-Drift-Kick scheme (Miniati+Colella 2007)
vn+1/2
p =
an
an+1/2
vn
p +
1
an+1/2
gn
p
t
2
.
xn+1
p = xn
p +
1
an+1/2
vn+1/2
p t.
vn+1
p =
an+1/2
an+1
vn+1/2
p +
1
an+1
gn+1
p
t
2
.
Kick
Kick
Drift
All these pieces should be 2nd order.

Andrew Myers, LBNL
36
The Zel’dovich Pancake
• Collapse of a single, sinusoidal perturbation in an expanding background
• A common test case for cosmological dark matter codes
• Analytic solution exists prior to the “first caustic” - the time at which the
first matter parcels cross
• “Single-mode” analysis of cosmological structure formation

• Usually, a uniform, zero-temperature fluid is discretized with evenly-
spaced, equal mass particles.
• These particles are then perturbed from the initial positions using the
Zel’dovich approximation.
• Each point in space has only one particle, no velocity dispersion
Andrew Myers, LBNL
37

• These initial conditions represent an initial distribution function that is
singular in velocity space:
f(x, v, tini) = ⇢(x, tini) (v ¯v)
Andrew Myers, LBNL
38
• This approximation is made for good physical reasons.
• However, singular initial data can pose problems for numerical solution
methods. Problem may be ill-posed.
• When we look at the Richardson-extrapolated order as a function of time:

• Sample the regularized distribution on a Cartesian grid in phase space,
discarding those with tiny masses.
• = initial particle spacing in physical, velocity space.
Andrew Myers, LBNL
39
(hx, hv)

Myers_SIAMCSE15

More Related Content

What's hot (20)

Viewers also liked (14)

Similar to Myers_SIAMCSE15 (20)

More from Karen Pao (8)

Recently uploaded (20)

Myers_SIAMCSE15