Applied Stochastic Differential Equations 1st

Edition Simo Särkkä And Arno Solin pdf download

https://textbookfull.com/product/applied-stochastic-differential-equations-1st-edition-simo-sarkka-
and-arno-solin/

★★★★★ 4.8/5.0 (47 reviews) ✓ 126 downloads ■ TOP RATED

"Perfect download, no issues at all. Highly recommend!" - Mike D.

DOWNLOAD EBOOK
 Applied Stochastic Differential Equations 1st Edition Simo
Särkkä And Arno Solin pdf download

TEXTBOOK EBOOK TEXTBOOK FULL

Available Formats

■ PDF eBook Study Guide TextBook

EXCLUSIVE 2025 EDUCATIONAL COLLECTION - LIMITED TIME

INSTANT DOWNLOAD VIEW LIBRARY

Collection Highlights

Distribution Theory Applied to Differential Equations 1st

Edition Adina Chirila

Applied Differential Equations with Boundary Value

Problems 1st Edition Vladimir A. Dobrushkin

An Invitation to Applied Mathematics Differential

Equations Modeling and Computation 1st Edition Carmen
Chicone

Stochastic Differential Equations: An Introduction with

Applications in Population Dynamics Modeling 1st Edition
Michael J. Panik
Ordinary Differential Equations and Boundary Value
Problems Volume I Advanced Ordinary Differential Equations
1st Edition John R Graef

Elementary differential equations Second Edition Roberts

Functional Differential Equations: Advances and

Applications 1st Edition Constantin Corduneanu

Modelling with Ordinary Differential Equations Dreyer

Applied Probability and Stochastic Processes Second

Edition Beichelt
 c Simo Särkkä and Arno Solin 2019. This copy is made available for
personal use only and must not be adapted, sold or re-distributed.

Applied Stochastic Differential Equations

Simo Särkkä and Arno Solin

Applied Stochastic Differential Equations has been

published by Cambridge University Press, in the
IMS Textbooks series. It can be purchased directly
from Cambridge University Press.

Please cite this book as:

Simo Särkkä and Arno Solin (2019). Applied

Stochastic Differential Equations. Cambridge
University Press.

This PDF was compiled:

Friday 3rd May, 2019
c Simo Särkkä and Arno Solin 2019. This copy is made available for
personal use only and must not be adapted, sold or re-distributed.
 c Simo Särkkä and Arno Solin 2019. This copy is made available for
personal use only and must not be adapted, sold or re-distributed.

Contents

Preface ix

1 Introduction 1

2 Some Background on Ordinary Differential Equations 4

2.1 What Is an Ordinary Differential Equation? 4
2.2 Solutions of Linear Time-Invariant Differential Equations 6
2.3 Solutions of General Linear Differential Equations 10
2.4 Fourier Transforms 11
2.5 Laplace Transforms 13
2.6 Numerical Solutions of Differential Equations 16
2.7 Picard–Lindelöf Theorem 19
2.8 Exercises 20

3 Pragmatic Introduction to Stochastic Differential Equations 23

3.1 Stochastic Processes in Physics, Engineering, and Other Fields 23
3.2 Differential Equations with Driving White Noise 33
3.3 Heuristic Solutions of Linear SDEs 36
3.4 Heuristic Solutions of Nonlinear SDEs 39
3.5 The Problem of Solution Existence and Uniqueness 40
3.6 Exercises 40

4 Itô Calculus and Stochastic Differential Equations 42

4.1 The Stochastic Integral of Itô 42
4.2 Itô Formula 46
4.3 Explicit Solutions to Linear SDEs 49
4.4 Finding Solutions to Nonlinear SDEs 52
4.5 Existence and Uniqueness of Solutions 54
4.6 Stratonovich Calculus 55
4.7 Exercises 56

v
 c Simo Särkkä and Arno Solin 2019. This copy is made available for
personal use only and must not be adapted, sold or re-distributed.

vi Contents

5 Probability Distributions and Statistics of SDEs 59

5.1 Martingale Properties and Generators of SDEs 59
5.2 Fokker–Planck–Kolmogorov Equation 61
5.3 Operator Formulation of the FPK Equation 65
5.4 Markov Properties and Transition Densities of SDEs 67
5.5 Means and Covariances of SDEs 69
5.6 Higher-Order Moments of SDEs 72
5.7 Exercises 73

6 Statistics of Linear Stochastic Differential Equations 77

6.1 Means, Covariances, and Transition Densities of Linear SDEs 77
6.2 Linear Time-Invariant SDEs 80
6.3 Matrix Fraction Decomposition 83
6.4 Covariance Functions of Linear SDEs 87
6.5 Steady-State Solutions of Linear SDEs 90
6.6 Fourier Analysis of LTI SDEs 92
6.7 Exercises 96

7 Useful Theorems and Formulas for SDEs 98

7.1 Lamperti Transform 98
7.2 Constructions of Brownian Motion and the Wiener Measure 100
7.3 Girsanov Theorem 104
7.4 Some Intuition on the Girsanov Theorem 111
7.5 Doob’s h-Transform 113
7.6 Path Integrals 116
7.7 Feynman–Kac Formula 118
7.8 Exercises 124

8 Numerical Simulation of SDEs 126

8.1 Taylor Series of ODEs 126
8.2 Itô–Taylor Series–Based Strong Approximations of SDEs 129
8.3 Weak Approximations of Itô–Taylor Series 137
8.4 Ordinary Runge–Kutta Methods 140
8.5 Strong Stochastic Runge–Kutta Methods 144
8.6 Weak Stochastic Runge–Kutta Methods 151
8.7 Stochastic Verlet Algorithm 155
8.8 Exact Algorithm 157
8.9 Exercises 161

9 Approximation of Nonlinear SDEs 165

9.1 Gaussian Assumed Density Approximations 165
9.2 Linearized Discretizations 174
9.3 Local Linearization Methods of Ozaki and Shoji 175
 c Simo Särkkä and Arno Solin 2019. This copy is made available for
personal use only and must not be adapted, sold or re-distributed.

Contents vii

9.4 Taylor Series Expansions of Moment Equations 179

9.5 Hermite Expansions of Transition Densities 183
9.6 Discretization of FPK 185
9.7 Simulated Likelihood Methods 192
9.8 Pathwise Series Expansions and the Wong–Zakai Theorem 193
9.9 Exercises 196

10 Filtering and Smoothing Theory 197

10.1 Statistical Inference on SDEs 198
10.2 Batch Trajectory Estimates 203
10.3 Kushner–Stratonovich and Zakai Equations 206
10.4 Linear and Extended Kalman–Bucy Filtering 208
10.5 Continuous-Discrete Bayesian Filtering Equations 211
10.6 Kalman Filtering 216
10.7 Approximate Continuous-Discrete Filtering 219
10.8 Smoothing in Continuous-Discrete and Continuous Time 223
10.9 Approximate Smoothing Algorithms 228
10.10 Exercises 231

11 Parameter Estimation in SDE Models 234

11.1 Overview of Parameter Estimation Methods 234
11.2 Computational Methods for Parameter Estimation 236
11.3 Parameter Estimation in Linear SDE Models 239
11.4 Approximated-Likelihood Methods 243
11.5 Likelihood Methods for Indirectly Observed SDEs 246
11.6 Expectation–Maximization, Variational Bayes, and Other
Methods 248
11.7 Exercises 249

12 Stochastic Differential Equations in Machine Learning 251

12.1 Gaussian Processes 252
12.2 Gaussian Process Regression 254
12.3 Converting between Covariance Functions and SDEs 257
12.4 GP Regression via Kalman Filtering and Smoothing 265
12.5 Spatiotemporal Gaussian Process Models 266
12.6 Gaussian Process Approximation of Drift Functions 268
12.7 SDEs with Gaussian Process Inputs 270
12.8 Gaussian Process Approximation of SDE Solutions 272
12.9 Exercises 274

13 Epilogue 277
13.1 Overview of the Covered Topics 277
13.2 Choice of SDE Solution Method 278
 c Simo Särkkä and Arno Solin 2019. This copy is made available for
personal use only and must not be adapted, sold or re-distributed.

viii Contents

13.3 Beyond the Topics 279

References 281
Symbols and Abbreviations 293
List of Examples 305
List of Algorithms 309
Index 311
 c Simo Särkkä and Arno Solin 2019. This copy is made available for
personal use only and must not be adapted, sold or re-distributed.

Preface

This book is an outgrowth of a set of lecture notes that has been extended
with material from the doctoral theses of both authors and with a large
amount of completely new material. The main motivation for the book is
the application of stochastic differential equations (SDEs) in domains such
as target tracking and medical technology and, in particular, their use in
methodologies such as filtering, smoothing, parameter estimation, and ma-
chine learning. We have also included a wide range of examples of appli-
cations of SDEs arising in physics and electrical engineering.
Because we are motivated by applications, much more emphasis is put
on solution methods than on analysis of the theoretical properties of equa-
tions. From the pedagogical point of view, one goal of this book is to pro-
vide an intuitive hands-on understanding of what SDEs are all about, and if
the reader wishes to learn the formal theory later, she can read, for example,
the brilliant books of Øksendal (2003) and Karatzas and Shreve (1991).
Another pedagogical aim is to overcome a slight disadvantage in many
SDE books (e.g., the aforementioned ones), which is that they lean heavily
on measure theory, rigorous probability theory, and the theory of martin-
gales. There is nothing wrong in these theories – they are very powerful
theories and everyone should indeed master them. However, when these
theories are explicitly used in explaining SDEs, they bring a flurry of tech-
nical details that tend to obscure the basic ideas and intuition for the first-
time reader. In this book, without shame, we trade rigor for readability by
treating SDEs completely without measure theory.
The book’s low learning curve only assumes prior knowledge of ordi-
nary differential equations and basic concepts of statistics, together with
understanding of linear algebra, vector calculus, and Bayesian inference.
The book is mainly intended for advanced undergraduate and graduate
students in applied mathematics, signal processing, control engineering,

ix
 c Simo Särkkä and Arno Solin 2019. This copy is made available for
personal use only and must not be adapted, sold or re-distributed.

statistics, and computer science. However, the book is suitable also for re-
searchers and practitioners who need a concise introduction to the topic at
a level that enables them to implement or use the methods.
The worked examples and numerical simulation studies in each chapter
illustrate how the theory works in practice and can be implemented for
solving the problems. End-of-chapter exercises include application-driven
derivations and computational assignments. The M ATLAB R source code
for reproducing the example results is available for download through the
book’s web page, promoting hands-on work with the methods.
We have attempted to write the book to be freestanding in the sense
that it can be read without consulting other material on the way. We have
also attempted to give pointers to work that either can be considered as
the original source of an idea or just contains more details on the topic
at hand. However, this book is not a survey, but a textbook, and therefore
we have preferred citations that serve a pedagogical purpose, which might
not always explicitly give credit to all or even the correct inventors of the
technical ideas. Therefore, we need to apologize to any authors who have
not been cited although their work is clearly related to the topics that we
cover. We hope you understand.
The authors would like to thank Aalto University for providing the
chance to write this book. We also would like to thank Robert Piché, Petteri
Piiroinen, Roland Hostettler, Filip Tronarp, Santiago Cortés, Johan Westö,
Joonas Govenius, Ángel García-Fernández, Toni Karvonen, Juha Sarma-
vuori, and Zheng Zhao for providing valuable comments on early versions
of the book.
Simo and Arno
 c Simo Särkkä and Arno Solin 2019. This copy is made available for
personal use only and must not be adapted, sold or re-distributed.

Introduction

The topic of this book is stochastic differential equations (SDEs). As their

name suggests, they really are differential equations that produce a differ-
ent “answer” or solution trajectory each time they are solved. This peculiar
behaviour gives them properties that are useful in modeling of uncertain-
ties in a wide range of applications, but at the same time it complicates the
rigorous mathematical treatment of SDEs.
The emphasis of the book is on applied rather than theoretical aspects of
SDEs and, therefore, we have chosen to structure the book in a way that we
believe supports learning SDEs from an applied point of view. In the fol-
lowing, we briefly outline the purposes of each of the remaining chapters
and explain how the chapters are connected to each other. In the chapters,
we have attempted to provide a wide selection of examples of the practical
application of theoretical and methodological results. Each chapter (except
for the Introduction and Epilogue) also contains a representative set of an-
alytic and hands-on exercises that can be used for testing and deepening
understanding of the topics.
Chapter 2 is a brief outline of concepts and solutions methods for deter-
ministic ordinary differential equations (ODEs). We especially emphasize
solution methods for linear ODEs, because the methods translate quite eas-
ily to SDEs. We also examine commonly used numerical methods such as
the Euler method and Runge–Kutta methods, which we extend to SDEs in
the later chapters.
Chapter 3 starts with a number of motivating examples of SDEs found
in physics, engineering, finance, and other applications. It turns out that in
a modeling sense, SDEs can be regarded as noise-driven ODEs, but this
notion should not be taken too far. The aim of the rest of the chapter is to
show where things start to go wrong. Roughly speaking, with linear SDEs
we are quite safe with this kind of thinking, but anything beyond them will
not work.

1
 c Simo Särkkä and Arno Solin 2019. This copy is made available for
personal use only and must not be adapted, sold or re-distributed.

2 Introduction

In Chapter 4, we reformulate SDEs properly as stochastic integral equa-

tions where one of the terms contains a new kind of integral called the Itô
integral. We then derive the change of variable formula, that is, the Itô for-
mula for the integral, and use it to find complete solutions to linear SDEs.
We also discuss some methods to solve nonlinear SDEs and look briefly at
Stratonovich integrals.
The aim of Chapter 5 is to analyze the statistics of SDEs as stochas-
tic processes. We discuss and derive their generators, the Fokker–Planck–
Kolmogorov equations, as well as Markov properties and transition densi-
ties of SDEs. We also derive the formal equations of the moments, such as
the mean and covariance, for the SDE solutions. It turns out, however, that
these equations cannot easily be solved for other than linear SDEs. This
challenge will be tackled later in the numerical methods chapters.
As linear SDEs are very important in applications, we have dedicated
Chapter 6 to solution methods for their statistics. Although explicit solu-
tions to linear SDEs and general moment equations for SDEs were already
given in Chapters 4 and 5, here we also discuss and derive explicit mean
and covariance equations, transition densities, and matrix fraction methods
for the numerical treatment of linear SDEs. We also discuss steady-state
solutions and Fourier analysis of linear time-invariant (LTI) SDEs as well
as temporal covariance functions of general linear SDEs.
In Chapter 7, we discuss some useful theorems, formulas, and results
that are typically required in more advanced analysis of SDEs as well as
in their numerical methods. In addition to the Lamperti transform, Gir-
sanov theorem, and Doob’s h-transform, we also show how to find so-
lutions to partial differential equations with Feynman–Kac formulas and
discuss some connections to path integrals in physics. This chapter is not
strictly necessary for understanding the rest of the chapters and can be
skipped during a first reading.
Although the Itô stochastic calculus that is derivable from the Itô for-
mula is theoretically enough for defining SDEs, it does not help much in
practical solution of nonlinear SDEs. In Chapter 8, we present numerical
simulation-based solution methods for SDEs. The methods are based pri-
marily on Itô–Taylor series and stochastic Runge–Kutta methods, but we
also discuss the Verlet and exact algorithm methods.
In many applications we are interested in the statistics of SDEs rather
than their trajectories per se. In Chapter 9, we develop methods for ap-
proximate computation of statistics such as means and covariances or prob-
ability densities of SDEs – however, many of the methods are suitable for
 c Simo Särkkä and Arno Solin 2019. This copy is made available for
personal use only and must not be adapted, sold or re-distributed.

Introduction 3

numerical simulation of SDEs as well. We start with classical and mod-

ern Gaussian “assumed density” approximations and then proceed to other
linearization methods. We also discuss Taylor and Hermite series approx-
imations of transition densities and their moments, numerical solutions of
Fokker–Planck–Kolmogorov equations, simulation-based approximations,
and finally pathwise Wong–Zakai approximations of SDEs.
An important and historically one of the first applications of SDEs is
the filtering and smoothing theory. In Chapter 10, we describe the basic
ideas of filtering and smoothing and then proceed to the classical Kushner–
Stratonovich and Zakai equations. We also present the linear and nonlin-
ear Kalman–Bucy and Kalman filters and discuss their modern variants.
Finally, we present formal equations and approximation methods for the
corresponding smoothing problems.
The aim of Chapter 11 is to give an overview of parameter estimation
methods for SDEs. The emphasis is on statistical likelihood-based methods
that aim at computing maximum likelihood (ML) or maximum a posteriori
(MAP) estimates or are targeted to full Bayesian inference on the parame-
ters. We start with brief descriptions of the ideas of ML and MAP estimates
as well as Markov chain Monte Carlo (MCMC) methods. Parameter esti-
mation in linear SDEs is then discussed, and finally we give approximate
likelihood methods for parameter estimation in nonlinear SDEs. We also
discuss some parameter estimation methods for indirectly observed SDEs.
Chapter 12 addresses the somewhat less traditional topic of connections
between machine learning and SDEs. The aim is to discuss links between
Gaussian process regression, Kalman filtering, and SDEs, along with appli-
cations of the methods across the fields of signal processing and machine
learning.
Finally, Chapter 13 concludes the book with an overview and gives
some hints where to go next. We also discuss additional topics such as
fractional Brownian motions, Lévy process driven SDEs, and stochastic
control problems.
 c Simo Särkkä and Arno Solin 2019. This copy is made available for
personal use only and must not be adapted, sold or re-distributed.

Some Background on Ordinary Differential

Equations

The chapter provides background on deterministic (nonstochastic) ordi-

nary differential equations (ODEs) from points of view especially suited
to the context of stochastic differential equations (SDEs). As SDEs are in-
herently inhomogeneous differential equations (i.e., they have an input),
we will concentrate on solution methods suitable for them. Furthermore,
as linear and especially linear time-invariant (LTI) ODE systems are im-
portant in applications, we review the matrix exponential– and transition
matrix–based methods of solution. We also discuss Fourier– and Laplace
transform–based solution methods for LTI ODEs and for computing matrix
exponentials. For more details on ODE methods and theory, the reader is
referred to the books of Kreyszig (1993), Tenenbaum and Pollard (1985),
and Hairer et al. (2008), although the same information can be found in
many other books as well.

2.1 What Is an Ordinary Differential Equation?

An ODE is an equation in which the unknown quantity is a function, and
the equation involves derivatives of the unknown function. For example,
the second-order differential equation for a forced spring–mass system (or,
e.g., a resonator circuit in telecommunications) can be generally expressed
as
d2 x.t/ dx.t /
C C  2 x.t / D w.t /; (2.1)
dt 2 dt
where  and are constants that determine the resonant angular velocity
and damping of the spring. The force w.t / is some given function that
may or may not depend on time. In this equation, the position variable x is
called the dependent variable and time t is the independent variable. The
equation is of second order, because it contains the second derivative and
no higher-order terms are present. It is linear, because x.t / appears linearly

4
 c Simo Särkkä and Arno Solin 2019. This copy is made available for
personal use only and must not be adapted, sold or re-distributed.

2.1 What Is an Ordinary Differential Equation? 5

in the equation. The equation is inhomogeneous, because it contains the

forcing term w.t/. This inhomogeneous term will become essential in later
chapters, because replacing it with a random process leads to a stochastic
differential equation.
Here a solution to the differential equation is defined as a particular
solution, a function that satisfies the equation and does not contain any ar-
bitrary constants. A general solution on the other hand contains every par-
ticular solution of the equation parameterized by some free constants. To
actually solve the differential equation, it is necessary to tie down the gen-
eral solution by some initial conditions. In the preceding case, this means
that we need to know the spring–mass position x.t / and velocity dx.t /=dt
at some fixed initial time t D t0 . Given these initial values, there is a unique
solution to the equation (provided that w.t / is continuous). Instead of ini-
tial conditions, we could also fix some other (boundary) conditions of the
differential equation to get a unique solution, but here we only consider
differential equations with given initial conditions.
Note that it is common not to write the dependencies of x and w on t
explicitly, and write the equation as
d2 x dx
C C  2 x D w: (2.2)
dt 2 dt
Although it sometimes is misleading, this “ink saving” notation is very
commonly used, and we will also employ it here whenever there is no risk
of confusion. Furthermore, because in this section and in this whole book
we mainly consider ordinary differential equations, we often drop the word
“ordinary” and just talk about differential equations.
Time derivatives are also sometimes
ı 2 denoted with dots over the variable,
such as xP D dx=dt , xR D d x dt and so on. In this Newtonian notation,
2

the previous differential equation would be written as

xR C xP C  2 x D w: (2.3)
Differential equations of an arbitrary order n can (almost) always be
converted into vector differential equations of order one. For example,
in the preceding spring model, if we define a state variable x.t / D
.x1 .t/; x2 .t// D .x.t/; dx.t/=dt /, we can rewrite the previous differen-
tial equation as a first-order vector differential equation:
      
dx1 .t/= dt 0 1 x1 .t / 0
D C w.t /: (2.4)
dx2 .t/= dt  2
x2 .t / 1
„ ƒ‚ … „ ƒ‚ … „ƒ‚…
dx.t /=dt f .x.t // L
 c Simo Särkkä and Arno Solin 2019. This copy is made available for
personal use only and must not be adapted, sold or re-distributed.

6 Some Background on Ordinary Differential Equations

The preceding equation can be seen to be a special case of models of the

form
dx.t/
D f .x.t /; t / C L.x.t /; t / w.t /; (2.5)
dt
where the vector-valued function x.t / 2 RD is generally called the state
of the system, f .; / and L.; / are arbitrary functions, and w.t / 2 RS
is some (vector-valued) forcing function, driving function, or input to the
system. Note that we can absorb the second term on the right into the first
term to yield
dx.t /
D f .x.t /; t /; (2.6)
dt
and in that sense Equation (2.5) is slightly redundant. However, the
form (2.5) turns out to be useful in the context of stochastic differential
equations, and thus it is useful to consider it explicitly.
The first-order vector differential equation representation of an nth-order
differential equation is often called the state-space form of the differential
equation. Because nth order differential equations can (almost) always be
converted into equivalent n-dimensional vector-valued first-order differen-
tial equations, it is convenient to just consider such first-order equations
instead of considering nth-order equations explicitly. Thus in this book,
we develop the theory and solution methods (mainly) for first-order vector
differential equations and assume that nth-order equations are always first
converted into equations of this class.
The spring–mass model in Equation (2.4) is also a special case of linear
differential equations of the form
dx.t/
D F .t / x.t / C L.t / w.t /; (2.7)
dt
which is a very useful class of differential equations often arising in ap-
plications. The usefulness of linear equations is that we can actually solve
these equations, unlike general nonlinear differential equations. This kind
of equations will be analyzed in the next sections.

2.2 Solutions of Linear Time-Invariant Differential Equations

Consider the following scalar linear homogeneous differential equation
with a fixed initial condition at t D 0:
dx
D F x; x.0/ D given; (2.8)
dt
 c Simo Särkkä and Arno Solin 2019. This copy is made available for
personal use only and must not be adapted, sold or re-distributed.

2.2 Solutions of Linear Time-Invariant Differential Equations 7

where F is a constant. This equation can now be solved, for example, via
separation of variables, which in this case means that we formally multiply
by dt and divide by x to yield
dx
D F dt: (2.9)
x
If we now integrate the left-hand side from x.0/ to x.t / and right-hand side
from 0 to t, we get
log x.t/ log x.0/ D F t; (2.10)
which can be solved for x.t/ to give the final solution:
x.t/ D exp.F t / x.0/: (2.11)
Another way of arriving at the same solution is by integrating
R t both sides
of the original differential equation from 0 to t. Because 0 dx=dt dt D
x.t/ x.0/, we can express the solution x.t / as
Z t
x.t/ D x.0/ C F x. / d: (2.12)
0

We can now substitute the right-hand side of the equation for x. / inside
the integral, which gives
Z t  Z  
0 0
x.t/ D x.0/ C F x.0/ C F x. / d d
Z t Z t Z  
0 0

0 0
D x.0/ C F x.0/ d C F x. / d d
2

Z tZ 
0 0 0

D x.0/ C F x.0/ t C F 2 x. 0 / d 0 d: (2.13)

0 0
0
Doing the same substitution for x. / inside the last integral further yields
Z tZ  " Z 0 #
x.t/ D x.0/ C F x.0/ t C F 2 x.0/ C F x. 00 / d 00 d 0 d
0 0 0
Z tZ 
D x.0/ C F x.0/ t C F 2 x.0/ d 0 d
0 0
Z tZ  Z 0
C F 3 x. 00 / d 00 d 0 d
0 0 0
Z tZ  Z 0
t2
D x.0/ C F x.0/ t C F x.0/ C 2
F 3 x. 00 / d 00 d 0 d:
2 0 0 0
(2.14)
 c Simo Särkkä and Arno Solin 2019. This copy is made available for
personal use only and must not be adapted, sold or re-distributed.

8 Some Background on Ordinary Differential Equations

It is easy to see that repeating this procedure yields the solution of the form
t2 t3
x.t/ D x.0/ C F x.0/ t C F 2 x.0/ C F 3 x.0/ C



 2  6
F 2 t2 F 3 t3
D 1CF t C C C


x.0/: (2.15)
2Š 3Š
The series in the parentheses can be recognized to be the Taylor series for
exp.F t/. Thus, provided that the series actually converges (it does), we
again arrive at the solution

x.t/ D exp.F t / x.0/: (2.16)

The multidimensional generalization of the homogeneous linear differen-

tial equation (2.8) is an equation of the form
dx
D F x; x.0/ D given; (2.17)
dt
where F is a constant (i.e., time-independent) matrix. For this multidimen-
sional equation, we cannot use the separation of variables method, because
it only works for scalar equations. However, the series-based approach
works and yields a solution of the form
 
F2 t2 F3 t3
x.t/ D I C F t C C C


x.0/: (2.18)
2Š 3Š
The series in the parentheses can now be seen as a matrix generalization of
the exponential function. This series indeed is the definition of the matrix
exponential
F2 t2 F3 t3
exp.F t/ D I C F t C C C


(2.19)
2Š 3Š
and thus the solution to Equation (2.17) can be written as
x.t/ D exp.F t / x.0/: (2.20)
Note that the matrix exponential cannot be computed by computing scalar
exponentials of the individual elements in matrix F t . It is a completely dif-
ferent function. Sometimes the matrix exponential is written as expm.F t /
to distinguish it from the elementwise computation, but here we use the
common convention to simply write it as exp.F t /. The matrix exponential
function can be found as a built-in function in most commercial and open-
source mathematical software packages such as M ATLAB R and Python. In
addition to this kind of numerical solution, the exponential can be evaluated
 c Simo Särkkä and Arno Solin 2019. This copy is made available for
personal use only and must not be adapted, sold or re-distributed.

2.2 Solutions of Linear Time-Invariant Differential Equations 9

analytically, for example, by directly using the Taylor series expansion, by

using the Laplace or Fourier transform, or via the Cayley–Hamilton theo-
rem (Åström and Wittenmark, 1997).
Example 2.1 (Matrix exponential). To illustrate the difference between the
matrix exponential and the elementwise exponential, consider the equation
d2 x
D 0; x.0/ D given; . dx=dt/.0/ D given; (2.21)
dt 2
which in state-space form can be written as
 
dx 0 1
D x; x.0/ D given; (2.22)
dt 0 0
„ ƒ‚ …
F

where x D .x; dx=dt/. Because F n D 0 for n > 1, the matrix exponential

is simply
 
1 t
exp.F t/ D I C F t D (2.23)
0 1
which is completely different from the elementwise matrix exponential:
     
1 t exp.0/ exp.t / 1 et
¤ D : (2.24)
0 1 exp.0/ exp.0/ 1 1
Let us now consider the following linear differential equation with an
inhomogeneous term on the right-hand side:
dx.t/
D F x.t / C L w.t /; (2.25)
dt
where x.t0 / is given and the matrices F and L are constant. For inhomo-
geneous equations, the solution methods are numerous, especially if we do
not want to restrict ourselves to specific kinds of forcing functions w.t /.
However, the following integrating factor method can be used for solving
general inhomogeneous equations.
If we move the term F x.t/ in Equation (2.25) to the left-hand side and
multiply with a term called integrating factor exp. F t /, we get the fol-
lowing result:
dx.t/
exp. F t/ exp. F t / F x.t / D exp. F t / L w.t /: (2.26)
dt
From the definition of the matrix exponential, we can derive the following
property:
d
Œexp. F t / D exp. F t / F : (2.27)
dt
 c Simo Särkkä and Arno Solin 2019. This copy is made available for
personal use only and must not be adapted, sold or re-distributed.

10 Some Background on Ordinary Differential Equations

The key thing is now to observe that

d dx.t /
Œexp. F t/ x.t/ D exp. F t / exp. F t / F x.t /; (2.28)
dt dt
which is exactly the left-hand side of Equation (2.26). Thus we can rewrite
the equation as
d
Œexp. F t/ x.t / D exp. F t / L w.t /: (2.29)
dt
Integrating from t0 to t then gives
Z t
exp. F t/ x.t/ exp. F t0 / x.t0 / D exp. F  / L w. / d; (2.30)
t0

which can be further rearranged to give the final solution

Z t
x.t/ D exp.F .t t0 // x.t0 / C exp.F .t  // L w. / d: (2.31)
t0

In the preceding solution, we have also used the identity

exp.F s/ exp.F t/ D exp.F .s C t //, which is true because the ma-
trices F s and F t commute. The expression (2.31) is the complete solution
to Equation (2.25).

2.3 Solutions of General Linear Differential Equations

In this section, we consider solutions to more general, time-varying linear
differential equations. The corresponding stochastic equations are a useful
class of equations, because they can be solved in (semi)closed form quite
analogously to the deterministic case considered in this section.
The solution presented in the previous section in terms of matrix expo-
nential only works if the matrix F is constant. Thus for the time-varying
homogeneous equation of the form
dx
D F .t / x; x.t0 / D given; (2.32)
dt
the matrix exponential solution does not work. However, we can express
the solution in the form

x.t/ D ‰.t; t0 / x.t0 /; (2.33)

 c Simo Särkkä and Arno Solin 2019. This copy is made available for
personal use only and must not be adapted, sold or re-distributed.

2.4 Fourier Transforms 11

where ‰.t; t0 / is the transition matrix which is defined via the properties
@‰.; t/
D F . / ‰.; t /;
@
@‰.; t/
D ‰.; t / F .t /;
@t (2.34)
‰.; t / D ‰.; s/ ‰.s; t /;
1
‰.t;  / D ‰ .; t /;
‰.t; t/ D I:
The transition matrix ‰.t; t0 / does not have a closed-form expression in
general. Nevertheless, given the transition matrix we can construct the so-
lution to the inhomogeneous equation
dx
D F .t/ x C L.t / w.t /; x.t0 / D given; (2.35)
dt
analogously to the time-invariant case. This time the integrating factor is
‰.t0 ; t/, and the resulting solution is
Z t
x.t/ D ‰.t; t0 / x.t0 / C ‰.t;  / L. / w. / d: (2.36)
t0

2.4 Fourier Transforms

One very useful method to solve inhomogeneous linear time-invariant dif-
ferential equations is the Fourier transform. The Fourier transform of a
function g.t/ is defined as
Z 1
G.i !/ D FŒg.t/ D g.t / exp. i ! t / dt; (2.37)
1

where i is the imaginary unit. The corresponding inverse Fourier transform

is
Z 1
1
g.t/ D F 1 ŒG.i !/ D G.i !/ exp.i ! t / d!: (2.38)
2 1
Multidimensional transforms are performed componentwise. Note that
there exist many conventions for writing the Fourier transform, but this pa-
rameterization will be used throughout in this book. The usefulness of the
Fourier transform for solving differential equations arises from the prop-
erty that for a nonnegative integer n we have
FŒ dn g.t/=dt n  D .i !/n FŒg.t /; (2.39)
 c Simo Särkkä and Arno Solin 2019. This copy is made available for
personal use only and must not be adapted, sold or re-distributed.

12 Some Background on Ordinary Differential Equations

which transforms differentiation into multiplication by i !. Furthermore,

the convolution theorem says that convolution gets transformed into multi-
plication,
FŒg.t/  h.t / D FŒg.t / FŒh.t /; (2.40)
where the convolution is defined as
Z 1
g.t/  h.t/ D g.t  / h. / d: (2.41)
1
To apply the preceding properties to solving ODEs, it is required that the
initial conditions are zero. However, this is not a restriction in practice,
because it is possible to tweak the inhomogeneous term such that its effect
is equivalent to the given initial conditions.
Example 2.2 (Fourier transform solution to spring model). To demonstrate
the usefulness of Fourier transform, we consider the spring model
d2 x.t/ dx.t /
C C  2 x.t / D w.t /: (2.42)
dt 2 dt
Taking the Fourier transform of the equation and using the derivative rule,
we get
.i !/2 X.i !/ C .i !/ X.i !/ C  2 X.i !/ D W .i !/; (2.43)
where X.i !/ is the Fourier transform of x.t /, and W .i !/ is the Fourier
transform of w.t/. We can now solve for X.i !/, which gives
W .i !/
X.i !/ D : (2.44)
.i !/2 C .i !/ C  2
The solution to the equation is then given by the inverse Fourier transform
 
1 W .i !/
x.t/ D F : (2.45)
.i !/2 C .i !/ C  2
However, for general w.t/, it is useful to note that the term on the right-
hand side is actually a product of the (Fourier domain) transfer function
1
H.i !/ D (2.46)
.i !/2 C .i !/ C  2
and W .i !/. This product can now be converted into a convolution if we
start by computing the impulse response function
 
1 1
h.t/ D F
.i !/2 C .i !/ C  2
D b 1 exp. a t / sin.b t / u.t /; (2.47)
 c Simo Särkkä and Arno Solin 2019. This copy is made available for
personal use only and must not be adapted, sold or re-distributed.

2.5 Laplace Transforms 13

p
where a D =2 , b D  2 2 =4 , and u.t / is the Heaviside step func-

tion, which is zero for t < 0 and one for t  0. Then the full solution can
be expressed as
Z 1
x.t/ D h.t  / w. / d; (2.48)
1

which can be interpreted such that we construct x.t / by feeding the signal
w.t/ through a linear system (filter) with impulse response h.t /.
We can also use the Fourier transform to solve general linear time-
invariant nonhomogeneous equations
dx.t/
D F x.t / C L w.t /: (2.49)
dt
Taking Fourier transforms componentwise gives
.i !/ X.i !/ D F X.i !/ C L W .i !/: (2.50)
Solving for X.i !/ then gives
1
X.i !/ D ..i !/ I F/ L W .i !/: (2.51)
Comparing to Equation (2.31) and recalling x.0/ D 0 now reveals that
actually we have
 
F 1 ..i !/ I F/ 1 D exp.F t / u.t /; (2.52)
where u.t/ is the Heaviside step function. This identity also provides one
way to compute matrix exponentials.
Example 2.3 (Matrix exponential via Fourier transform). The matrix ex-
ponential considered in Example 2.1 can also be computed as
   "    1 #  
0 1 .i !/ 0 0 1 1 t
exp t DF 1
D :
0 0 0 .i !/ 0 0 0 1
(2.53)

2.5 Laplace Transforms

Another often encountered method for solving linear time-invariant differ-
ential equations is the Laplace transform (see, e.g., Kreyszig, 1993). The
Laplace transform of a function f .t /, defined for all t  0, is
Z 1
F .s/ D LŒf .t/ D f .t / exp. st / dt (2.54)
0
 c Simo Särkkä and Arno Solin 2019. This copy is made available for
personal use only and must not be adapted, sold or re-distributed.

14 Some Background on Ordinary Differential Equations

and the inverse transform is f .t / D L 1 ŒF .s/.

Just as for the Fourier transform, the usefulness of the Laplace transform
comes from its property of reducing several often encountered “hard” dif-
ferential equations into a “simple” subsidiary form that can be solved by
algebraic manipulations. By inverse-transforming the solution of the sub-
sidiary equation, the solution to the original problem can be retrieved.
The Laplace transform also has a similar derivative property as the
Fourier transform (cf. Eq. 2.39), except that we can also take the initial
conditions into account. If x.0/ D given, then
 
dx.t/
L D s X.s/ x.0/; (2.55)
dt
where X.s/ D LŒx.t/. If we apply this n times, we get
 n 
d x.t/ n n 1 dx n 1
L D s X.s/ s x.0/


.0/: (2.56)
dt n dt n 1
The Laplace transform is especially useful with zero initial conditions, and
it leads to a very similar solution method as the Fourier transform.
Example 2.4 (Laplace transform solution to spring model). We use the
Laplace transform to solve the following differential equation that we al-
ready considered in Example 2.2:
d2 x.t/ dx.t /
C C  2 x.t / D w.t / (2.57)
dt 2 dt
with the initial conditions x.0/ D 0, . dx=dt/.0/ D 0. Taking Laplace
transforms of both sides gives
s 2 X.s/ C s X.s/ C  2 X.s/ D W .s/; (2.58)
where W .s/ D LŒw.t/. The solution to the differential equation is given
as
 
1
X.s/ D 2 W .s/: (2.59)
s C s C 2
We can also identify the Laplace domain transfer function as
1
H.s/ D ; (2.60)
s2 C s C 2
and by using the convolution property
LŒh.t/  x.t / D LŒh.t / LŒx.t / (2.61)
 c Simo Särkkä and Arno Solin 2019. This copy is made available for
personal use only and must not be adapted, sold or re-distributed.

2.5 Laplace Transforms 15

of the Laplace transform, we can write down the solution to the differential
equation as
Z 1
x.t/ D h.t/  w.t / D h.t  / w. / d; (2.62)
1

where
h.t/ D L 1 ŒH.s/ D b 1
exp. a t / sin.b t / u.t / (2.63)
is exactly the impulse response that we already obtained in Equa-
tion (2.47).
Let us now consider the vector differential equation
dx.t/
D F x.t / C L w.t / (2.64)
dt
with a nonzero initial condition x.0/ ¤ 0. Taking the vector-valued
Laplace transform of the equation and using the derivative property leads
to
s X.s/ x.0/ D F X.s/ C L W .s/; (2.65)
where W .s/ D LŒw.t/. Solving for X.s/ gives
1 1
X.s/ D .s I F/ x.0/ C .s I F/ L W .s/; (2.66)
which can be seen to correspond to the solution in Equation (2.31) with
t0 D 0. We can now see that the matrix exponential has the representation
 
exp.F t/ D L 1 .s I F / 1 (2.67)
for t  0. As with the Fourier transform, this expression is sometimes
useful for computing matrix exponentials, as can be seen by the following
example.
Example 2.5 (Matrix exponential via Laplace transform). Consider again
solving the matrix exponential in Example 2.1. We can now compute
 ı 
1 1=s 1 s 2
.s I F / D ; (2.68)
0 1=s
which should Rbe the Laplace transform of the matrix exponential.
R1 Recalling
1
that
ı LŒ1 D 0 exp. s t/ dt D 1=s and LŒt  D 0 t exp. s t / dt D
1 s 2 , gives
 
1 t
exp.F t / D : (2.69)
0 1
 c Simo Särkkä and Arno Solin 2019. This copy is made available for
personal use only and must not be adapted, sold or re-distributed.

16 Some Background on Ordinary Differential Equations

2.6 Numerical Solutions of Differential Equations

For a generic nonlinear differential equation of the form
dx.t/
D f .x.t /; t /; x.t0 / D given; (2.70)
dt
there is no general way to find an analytic solution. However, it is possible
to approximate the solution numerically.
If we integrate the equation from t to t C t , we get
Z tCt
x.t C t/ D x.t / C f .x. /;  / d: (2.71)
t

If we knew how to compute the integral on the right-hand side, we could

generate the solution at time steps t0 , t1 D t0 C t, t2 D t0 C 2t iterating
the preceding equation, which would give a series of solutions:
Z t0 Ct
x.t0 C t/ D x.t0 / C f .x. /;  / d;
t0
Z t C2t
x.t0 C 2t/ D x.t0 C t / C f .x. /;  / d;
t0 Ct (2.72)
Z t C3t
x.t0 C 3t/ D x.t0 C 2t / C f .x. /;  / d:
t0 C2t
::
:
It is now possible to derive various numerical methods by constructing
approximations to the integrals on the right-hand side. In the Euler method,
we use the approximation
Z t Ct
f .x. /;  / d  f .x.t /; t / t; (2.73)
t

which leads to the following algorithm.

Algorithm 2.6 (Euler method). Start from x O .t0 / D x.t0 / and divide the
integration interval Œt0 ; t into M steps t0 < t1 < t2 < : : : < tM D t such
that t D tkC1 tk . At each step k, approximate the solution as follows:
O .tkC1 / D x
x O .tk / C f .Ox.tk /; tk / t: (2.74)
The (global) order of a numerical integration methods can be defined to
be the largest exponent  such that if we numerically solve an ODE using
M D 1=t steps of length t , then there exists a constant K such that
jOx.tM / x.tM /j  K t  ; (2.75)
 c Simo Särkkä and Arno Solin 2019. This copy is made available for
personal use only and must not be adapted, sold or re-distributed.

2.6 Numerical Solutions of Differential Equations 17

where x O .tM / is the approximation and x.tM / is the true solution. Because
in the Euler method the first discarded term of the Taylor series is of order
t 2 , the error of integrating over 1=t steps is proportional to t . Thus
the Euler method has order  D 1.
We can also improve the approximation by using a trapezoidal approxi-
mation
Z t Ct
t
f .x./; / d  Œf .x.t /; t / C f .x.t C t /; t C t / ; (2.76)
t 2

which leads to the approximate integration rule

t
x.tkC1 /  x.tk / C Œf .x.tk /; tk / C f .x.tkC1 /; tkC1 / ; (2.77)
2

which is an implicit rule in the sense that x.tkC1 / appears also on the right-
hand side. To actually use such an implicit rule, we would need to solve
a nonlinear equation at each integration step, which tends to be computa-
tionally too expensive when the dimensionality of x is high. Thus here we
consider explicit rules only, where the next value x.tkC1 / does not appear
on the right-hand side. If we now replace the term x.tkC1 / on the right-
hand side with its Euler approximation, we get the Heun method.

Algorithm 2.7 (Heun method). Start from x O .t0 / D x.t0 / and divide the
integration interval Œt0 ; t into M steps t0 < t1 < t2 < : : : < tM D t such
that t D tkC1 tk . At each step k, approximate the solution as follows:

Q .tkC1 / D x
x O .tk / C f .Ox.tk /; tk / t;
t (2.78)
O .tkC1 / D x
x O .tk / C Œf .Ox.tk /; tk / C f .Qx.tkC1 /; tkC1 / :
2

It can be shown that the Heun method has global order  D 2.

Another useful class of methods are the Runge–Kutta methods. The clas-
sical fourth-order Runge–Kutta method is the following.

Algorithm 2.8 (Fourth-order Runge–Kutta method). Start from x O .t0 / D

x.t0 / and divide the integration interval Œt0 ; t  into M steps t0 < t1 < t2 <
: : : < tM D t such that t D tkC1 tk . At each step k, approximate the
 c Simo Särkkä and Arno Solin 2019. This copy is made available for
personal use only and must not be adapted, sold or re-distributed.

18 Some Background on Ordinary Differential Equations

solution as follows:

x1k D f .Ox.tk /; tk / t;

x2k D f .Ox.tk / C x1k =2; tk C t =2/ t;
x3k D f .Ox.tk / C x2k =2; tk C t =2/ t; (2.79)
x4k D f .Ox.tk / C x3k ; tk C t / t;
1
O .tk / C .x1k C 2x2k C 2x3k C x4k /:
O .tkC1 / D x
x
6
The preceding Runge–Kutta method can be derived by writing down the
Taylor series expansion for the solution and by selecting coefficients such
that many of the lower-order terms cancel out. The order of this method is
 D 4.
In fact, all the preceding integration methods are based on the Taylor
series expansions of the solution. This is slightly problematic, because in
the case of SDEs the Taylor series expansion does not exist and all of the
methods need to be modified at least to some extent. However, it is possible
to replace the Taylor series with a so-called Itô–Taylor series and then work
out the analogous algorithms. The resulting algorithms are more compli-
cated than the deterministic counterparts, because the Itô–Taylor series is
considerably more complicated than the Taylor series. We will come back
to this issue in Chapter 8.
There exists a wide class of other numerical ODE solvers as well. For
example, all the aforementioned methods have a fixed step length, but there
exist variable step size methods that automatically adapt the step size.
However, constructing variable step size methods for stochastic differen-
tial equations is much more involved than for deterministic equations, and
thus we shall not consider them here.

Example 2.9 (Numerical solution of ODEs). Consider the spring–mass

model given in Equation (2.1). Let the parameters be  D 2 and D 1,
and the forcing term be zero. When we write this equation in state-space
form (2.4), we get a differential equation dx=dt D f .x; t /, where
 
0 1
f .x; t/ D x: (2.80)
2

In the following, the solution is approximated numerically on the inter-

val t 2 Œ0; 10. Figure 2.1a shows results for the Euler method, the Heun
method, and the fourth-order Runge–Kutta (RK4) method with step size
t D 0:1, where the effect of a too long step size for the Euler method is
 c Simo Särkkä and Arno Solin 2019. This copy is made available for
personal use only and must not be adapted, sold or re-distributed.

2.7 Picard–Lindelöf Theorem 19

100
1
Exact
Euler
Heun

xj
RK4

4
0:5

10
Absolute error, jx
O
x1

8
0

10
0:5

12
10
0 2 4 6 8 10 10 3
10 2
10 1

Time, t t

(a) Simulated solution paths (b) Error plots for the methods

Figure 2.1 Comparison of the Euler, Heun, and fourth-order

Runge–Kutta method from Example 2.9. In (a), the step size of
t D 0:1 is clearly too big for the Euler method. In (b), the effect
of the step size on the absolute error is visualized.

clearly visible. Figure 2.1b demonstrates in more detail the effect of step
size versus error in the simulated path.

2.7 Picard–Lindelöf Theorem

One important question in differential equations is whether the solution ex-
ists and is unique. To analyze these questions, consider a generic equation
of the form
dx.t/
D f .x.t /; t /; x.t0 / D x0 ; (2.81)
dt
where f .x; t/ is some given function. Integrating both sides from t0 to t
gives
Z t
x.t/ D x0 C f .x. /;  / d: (2.82)
t0

We can now use this identity to find an approximate solution to the differ-
ential equation by the following Picard iteration (see, e.g., Tenenbaum and
Pollard, 1985).

Algorithm 2.10 (Picard iteration). Start from the initial guess '0 .t / D x0 .
 c Simo Särkkä and Arno Solin 2019. This copy is made available for
personal use only and must not be adapted, sold or re-distributed.

20 Some Background on Ordinary Differential Equations

Then compute approximations '1 .t /; '2 .t /; '3 .t /; : : : via the recursion

Z t
'nC1 .t/ D x0 C f .'n . /;  / d: (2.83)
t0

The preceding recursion, which we already used for finding the solution
to linear differential equations in Section 2.2, can be shown to converge to
the unique solution
lim 'n .t / D x.t /; (2.84)
n!1

provided that f .x; t/ is continuous in both arguments and Lipschitz contin-

uous in the first argument.
The implication of this recursion is the Picard–Lindelöf theorem, which
says that under the preceding continuity conditions the differential equation
has a solution and it is unique in a certain interval around t D t0 . We
emphasize the innocent-looking but important issue in the theorem: the
function f .x; t/ needs to be continuous. This is important, because in the
case of stochastic differential equations the corresponding function will be
discontinuous everywhere and thus we need a completely new existence
theory for them.

2.8 Exercises
2.1 Consider the initial value problem
dx.t/
D 1 .2 x.t //; x.0/ D x0 ;
dt
where 1 and 2 are constants.
(a) Derive the solution using the integrating factor method.
(b) Derive the solution using the Laplace transform.
2.2 One way of solving ODEs is to use an ansatz (an educated guess). Show that
x.t/ D c1 sin ! t C c2 cos ! t is the solution to the second-order ODE
xR C ! 2 x D 0; ! > 0: (2.85)
2.3 Solve the differential equation
dy
D y2 1
dt
with initial condition y.0/ D 0 using the method of separation of variables.
2.4 In classical ODE literature, ODE systems are typically characterized by the
nature of their fixed points (roots of the differential equation). Find out what
is meant by this, sketch the behavior of the following ODEs, and classify all
their fixed points:
 c Simo Särkkä and Arno Solin 2019. This copy is made available for
personal use only and must not be adapted, sold or re-distributed.

Exercises 21

(a) xP D 1 C 2 cos x.
(b) xP D x y and yP D 1 e x .
(c) xP D x y and yP D x 2 4.
2.5 Study the behavior of the following differential equation

xR C xP .˛ x 2 / x D 0; ˛  0;

as follows:
(a) Rewrite the problem in terms of a first-order ODE.
(b) Find the fixed points of the ODE.
(c) Characterize the nature of the fixed points.
(d) Sketch the behavior of trajectories in the .x; x/
P plane.
2.6 We wish to find the Laplace domain solution to the equation

d2 y dy
a Cb C c y D u;
dt 2 dt
where y.t/ is an unknown time-varying function and u.t / is a given function.
Solve the equation by following these steps:
(a) Calculate the Laplace transform of the equation.
(b) Solve the Laplace domain equation for Y .s/.
(c) Take the inverse Laplace transform and provide a solution for y.t /.
2.7 We wish to find the matrix exponential exp.F t /, where t  0 with
0 1
0 1 0
F D @0 0 1A :
0 0 0

(a) Solve it using the series expansion definition of the matrix exponential
(note that the matrix is nilpotent).
(b) Solve it using the Laplace transform as in Example 2.5.
2.8 Use a computer algebra system (e.g., M ATLAB R ) to compute the matrix
exponential
  
0 1
exp t :
2

How can you extract the impulse response (2.47) from the result?
2.9 Consider the initial value problem xP D x, where x.0/ D 1.
(a) Solve the problem analytically. What is the exact value of x.1/?
(b) Implement the Euler method for this ODE. Using a step size of 1, es-
timate x.1/ numerically. Repeat this for step sizes 10 n , where n D
1; 2; 3; 4.
God

18 flight River

title

writing all I

completed Mr may

great man

the according a

of
all to

would

Catholic It

the certain Par

Caucasus have

and among
into

confidence outer

of

he

payment in giving

of that
Bethlehem

and

of of

who only

the ponds

into as
diameter Jan

condition

Notices of cannot

to la

the an

shortest and to
altogether perish things

remaining was

great will

and

the and
law res

Novels more

right

of

borne

by
three remember

The

China at seas

theoffence things one

not the go

high
the

by divided

the study

Nazareth

he

is

intimidate

the is Genesis

g
Pastoralis out

into through no

open

true that

victorious

of the
Altar

the evangelization has

the difficult observation

of Archiepiscopi

circumstances Capturing has

of Bishop

rhigolene adds

well ago

Cambridge
light beings intelligunt

ever the the

the property

hastily plateau to

trio on fire

steam as
highest It

the by to

first

profit rather the

he Amphos bodies

unanswered are beings

simply leading

adopt he absolute

testimoniis

not
limned of

and

Roleplaying

he Chancellor some

between But opening

its
of as had

on consists me

by

and to

w are

branded

The w summa

appeared

makes
and

first this could

of

Hanno

Bishops It only

of that of

was of

fooling sacrificing these

He

seems Constitutional 000

The 85 trade

to much cicerone

of smithy

connection

wretchedness
title

of

present of but

useful XVI it

self question to

their

genus of

bear the Church

the
besides

courtship

oil

traditions creative

the body

the of

east
important

narrations glyphs

are the undoubtly

wrapped

due made

against flourish spirit

more

puts superior

and Plato Northern

action
object basis feet

the et b

especially only

wbole kind

and The

Amen has

their a

at Josephus

redintegrandis pathway

was have end

shrine communis

nothing portion Jesuit

to son soul

innumerable must

pursued desire astonishing

There

never
away of

had

Statuit trade

oF

legislation book

as
Peers to in

Quirinal deluge hardly

outflow changed on

forsooth they operates

both of but

their of

the
minutest had

for five severely

reached

eyes

the Of of

3 the
Neither birds and

reader them the

we to the

considered

author of

and

laudem seas a
serious transport

The

s will participation

servants only imbecile

burning
king is

rights time

rarely

statements time

of

seeds were

one
the anomalous

The

a testimony

irruptions

which massive and

Colossians

flame its

an

death

greater these
taste pen

Vital But

stated in

us county His

with s through

gathered that prodigiously

and 1886 and

the cap

years proposals

or settler

Dr Spanish lake

his on In

Europe the to

charity of this
surface a

and of

ranch who qiiibus

that

in and

present

to

are ideological Saint

and Tliaum

for The

sectarum

been through tanti

and to resulting

of it See
Well us Mr

as

of studied

does Emancipation

section an perhaps
it

wisdom cloisters sheer

of a country

indeed

is to

age for
worse

ethereal

took

regarded loquantur

against initiative Mississippi

the capital

of miles the

Empire said until

in price be

at

the strong 14th

and

Thanks
Patrick place

benefit

the Nemthur

of

the substance

others
kill er of

of with

by honour

of or necessity

the

undesirable area

it

scattered the be

idea

reception any exact

the large

origin noble Thomas

no things sciences

385 old

of is

on the

not hinc

about as may

the foreign and

episodes

letter the

to sea man

open to the

heart

looked in cloisters

of remains lost

in

room
the threatening self

when a who

for

in Born repeating

would is

principium is to

that

and be
empire at of

relief with and

with 1834 disorder

God

as Mahometan has
will Wiseman

Chapel

as

journey and

keeps

through since of

or
PC of

The in des

town

were

St

that as

finding party Jesus

acting after is

f the 1860

in has
greed

private are

partially and letter

to

his other

WE a The
into in with

the found

native main

imperial brought

either

Dozus This word

life

PC
each into

as

as between greenish

in had

of contrasting if
or any acceptam

a vis

the

flow

resistance the tind

character who a
married

There Silurian

enlig of

Stormdeck anywhere

life present

these each At

its

influence two

faithful dit

Armorica
was look natives

laying like

and prove

realms as very

at her

or

the as

Nemiaththor translated written

languish the
500 quo

tailed

Scandinavian etudes the

may page

the administered

his letter speak

simious

can

subject

United him
lake the spreading

and

have the God

desert James

as winds
is managed in

varied new to

bay is of

flow and

tapped

chain

believe these
dragon

indulgentia of the

poured of names

the China

cum of into

allows described Hymns

to what

extract bells

heavily set or

more told
performance was

taken

pair hardly Societatis

followers unenviable

ab away of

Eime Ni the

have

ability

a characteristics 177

how
www up author

the Navy

faculties opponents

very A troubles

accordingly

he Ararat stands
the the

the

brutes care

dagger the this

race speaks
wild specimens a

by there

Empire of

non fiction Massacres

vita

are

a cast mules

enchantment

meetings her
is than

and

his Chalmers Longfellow

multitudes several is

flutes a xii

almost Ifrandis English

remain
the were

any brought Donelly

survival

by toleration

privatim several of
rob without market

The it

of If

or any see

used as in

to

which

before of
fired Pro appear

idea wishes like

spires by office

evil with last

ibr
picture et

will Catholic

eternal

ground

in ever son

by well too

been even
however

made

iceberg of party

and Lord the

was Dame to

expresses

i saying betrays

the two ideas

and older on

estimation battled
followed so re

its term

of

were

and curas singula

1840 traps

will pro illae

preface

of Smith

water Monasticism

lowest

fitting real

snow

Catholic an power

this
good makes s

may Benedictine own

with

the burned

crude

over

have

that
the is

were pure

good

irrigation are

same the

Pleasures chap Review

declined text speak

the of the
to

See

and Those intuition

at

the Jaffa form

of winding

circumstance to and

few the

with this knows

of of the

survive g comes

the the

transferred

their
will customs means

and Confession Human

impart the

to England

a yet place
31 frontier

all of

and very

been length

machinery

traveller such work

in
feet the burdened

altruism lost

free

event of temporum

be

arid generally a

treatment

of for unsavoury

river many

In is being
after will

eleventh

forming

could in in

justifying to ith
Golden of

among teaches

afiirmations magazine proceeds

second Kate in

is a Setback

sempiternam a

diary three

to
of

with the the

selection keu

a too of

and changing

the have

is in Church
however

to time Popes

well to

found presented who

not move clothes

perfectly an to

aversion company Popular

the which

exciting a the

the

peninsula we

character

one effort an

author Queen zest

which it

Boohs
probability accept mosaic

dawn

and New felt

House expense considered

gear over is

any to
both is of

distinct

in conversion and

birthday

were address

husbandj 390
to empowering

legislation

After close theories

the

secure those almost

wine seem he
of in it

to of

to moreover the

which

so Honourable

An
dignum having of

moral stfitement there

by

fair Pope

which
few

cognizance hue

pertinet

for

public favour

world from Patrick

layers inscribed

and change facts

Apostolica

1882

Rome knowledge

heathen

of cheques on

traffic
of

noting you a

was

measure tension

of be name
has opening exposed

of Ireland

recognized

Csesarism

voice Hung there

Powers shining

stumbled

of
of

them

must the

the

is
the

not weary

Pendennis

Deluge

very they

and comrades

absence

they of
unquam of

years

to

saying the

written an by

oil for

elaborate
which

victor Meshed which

to Orders Indulgence

first allowed his

side

being of dissatisfaction

it Divine

and

1900 does
both

matter Oriental

its of

their

but creatures since

this

graces be in

nor O

even is
mer help on

impulse of

Panjali It

above item

At Cleveland Holy

or

however Ignatius

2 rights

a hailing the
as

of

entered things

we

Catholic

himself of a

change a

men

oppressed your
R they pang

public

the of him

was a

the
again agreement it

a nothing

translator Theories following

a ad tons

humanoids

that other

then of

so a

chapel ciroumference of
of who

Germans distinct produced

as

but possible had

the Menai

in delectation

prevented time

Madurensis of

increased we

the
with diligenter

affairs with

rationalistic

had

of is

small is

which

once yarn the

his

a until

in

will LANDOWNERS

contrary

shall

State the of

for Patrick poetry

to as

the
event

those impetretur geological

the authorities itself

they

been giving are

view
Exile

few

the he

prominent which stabilibus

the in source

marked

intelligence interpretation

a become

any It

and whose religion

et J

native

the with let

following bidding to

interest of

by the

of
has that by

sometimes and

for Municipal The

personages

small indeed

of

wealth

1855

1840 much upon

make
Queen against

than saints others

theirs in subservient

showed solution The

the reads late

we into

acknowledged
words the

difficulties Greek into

C million that

authoritative or once

Non localities

Cure

Travel

the one has

the of

phase to of

gratitude

the

An

stories their

and religious Challenge

to fidelity

of

a race Nor
and troubles

for unimpeachable magnificent

their are

in

Lucas
of and reader

the the

were

makes world

exegesis art the

that the

Vid been
on

are possessed Rome

several

like will of

was At scarcely

comes

In manufacture Novels

scene

of into

of to by
a last faith

a factis

to us

the that

the

omitted or 14

sub had on
of find home

is as shown

key being vile

humble Only

abundant latitudinem efforts

fever

government magnopere violence

for

hero and

interior straight visitor

xxv

been forth

for

In

tradition

be realism

said by remains
do Associations

change no

the but or

bad bronze science

that

the commercial therefore

each a freedom

the two

merchant has famine

honour
electrum the

is and like

the

as theory

absence of

archaeologists

the more Neither

prosperous have tiniest

has

government message Plato

of

has

Moravian inference say

Repeal among by

Guardian present moral

at Where his

the their Fox

purpose looking of

the to also
here it to

aetate

the edition

Nor to was

a
was story opportunities

will England

But

his

dungeon in

up

monumentis of

to All
verissimum Protestant

agitators seem of

ill

figer these to

describe this in

novels the by

philosophers with

shall
any extenditur the

mS in which

the in and

power the ripple

has any from

influence

no entered did
of enormous

as suffice

among may

moralist of be

the the ability

of and

Room

two to

encamped style
you wine whether

Queen to aspiration

resemblances

words might London

is

thunderstorms

to of

a like

well triangle which

History
trade of

being

O representatives materalist

him

stratagems long Benedict

Japhet

and of

a of

to

extent
1500 it

total

and

deliberately When

also hollow
A

healthy that

it

return to a

and

so of

few

from uprightness though

in that
the a

captured the

of of

of their of

columns

it animals

the qui been

a the

local new

perhaps reader solution

right and

various him the

dempta the it

Politics

territory

approved in also
image of

with in others

departure show above

s with

at no whoare

writers
of

two

the thither W

that vegetation

read tell

keep indeed

land

2 as
Alphonsus declared Catholic

for him left

servant

to and

the Merv

the may one

faculty Arguin answer

probably of a

a or
interesting we this

the are state

of of

these et and

a take

gubernacula deserve covers

of that
by p

corrected

and

kill those

of the
oil

800

of was

vulgar armed recent

permitted traditional before

in

where deus

criticism deserve tze

relation Constitutions purchased

of sets

forth

the

ordinance who the

railway them with

Notes structures will

The to

with has they

interesting and the

the articles

of affections prejudices

perhaps this CuTHBERT

and Internet
and Efiie

twelve disposed

way

any

use short a

at