![1 Introduction
Optical flow is defined as the apparent motion of the brightness patterns in
image. Computing it leads to many applications, including video compression
and motion estimation. It is also the first step of many video inpainting
algorithms.
Even though the first attempts of computing the optical flow has been
made in 1981 by [3], the subject is the object of many recent studies leading
to many improvements.
In this introduction to the optical flow computation, we will start by
giving a formal definition of the problem. The existing methods will be then
overviewed and classified. Finally, we will specify a roadmap for this study.
2 Definition of the optical flow
In the following, we will only handle the 2D case with 1D values (i.e. gray
images).
Let I(x, y, t) be the value of the (x, y) pixel at frame t and (u(x, y, t), v(x, y, t))
the flow at the (x, y) pixel at frame t.
The main idea of most algorithms for computing optical flow is to use the
Brightness Constancy : from a frame to an other, the brightness of a pixel
does not change according to the flow. This can be written as :
I(x, y, t) = I(x + u, y + v, t + 1) (1)
Equation 1 can be approximate with Taylor series to
∂I ∂I ∂I
I(x, y, t) = I(x, y, t) + u +v + (2)
∂x ∂y ∂t
∂I ∂I ∂I
Introducing the abbreviations Ex = ∂x
, Ey = ∂y
and Et = ∂t
, the following
holds :
uEx + vEy + Et = 0 (3)
Equation 3 is often denoted as the Optical Flow Contraint. We have a single
equation with two unknowns. This ill-posed problem is often called the
Apperture Problem illustrated by figure 1.
2](https://image.slidesharecdn.com/introduction-120522081844-phpapp01/85/Introduction-to-Optial-Flow-2-320.jpg)
![(a) First frame (b) Second frame
Figure 1: We can not tell in which way the line is moving with only those 2
frames.
To compute optical flow from equation 3, we must add additional con-
straints to obtain a system with more equations. Finding those constraints
is the main challenge adressed by articles about optical flow.
3 Overview of the existing methods
Methods for computing optical flow can be classified into two categories. The
first one is known as local methods whereas the other take a global approach
to the problem.
3.1 Local methods
Local methods are based on the approximation that the flow does not vary
inside small regions. The first method using this assumption is [3].
Following this simple idea, we can obtain, given a window of size n × n
3](https://image.slidesharecdn.com/introduction-120522081844-phpapp01/85/Introduction-to-Optial-Flow-3-320.jpg)
![and by denoting pn the nth pixel inside :
Ex (p1 )u + Ey (p1 )v + Et (p1 ) = 0
Ex (p2 )u + Ey (p2 )v + Et (p2 ) = 0
...
Ex (pn2 )u + Ey (pn2 )v + Et (pn2 ) = 0
Thus, this assumption allows us to have n2 equations for 2 unknowns,
giving a simple system to solve.
Further developpements of this method includes a coarse-to-fine imple-
mentation by pyramidal representation of the image. This have two advan-
tages :
• We can take into account larger displacements.
• It can speed-up computations.
This coarse-to-fine implementations seem to be the most used optical flow
computation algorithm into video inpainting solutions. On the opposite, it
has not been the object of many improvements during the recent years for
optical flow algorithms. Indeed, the global methods seems to have more
potential.
3.2 Global methods
Global methods generally works by favoring small first-order derivates of the
flow field. The first method to follow this idea is [2] which considers the opti-
cal flow computation by defining the measure of departure from smoothness
in the flow using an L2 norm :
2 2 2 2
2 ∂u ∂u ∂v ∂v
Es = + + + (4)
∂x ∂y ∂x ∂y
Using equations 3 (denoted Ec = uEx + vEy + Et ) and 4 and introducing
2
α to balance both terms, we can define the total error to be minimized :
Err2 = (α2 Es + Ec ) dx dy
2 2
(5)
4](https://image.slidesharecdn.com/introduction-120522081844-phpapp01/85/Introduction-to-Optial-Flow-4-320.jpg)
![We can note that having a high value for α2 will lead to smoother flows. This
factor is application dependant.
Other approaches include weightening Es along with the gradients (| I|)
to give less importance to areas having a high value for | I|. Indeed, it
is more likely to find flow discontinuities at an edge than elsewhere. The
equation to be minimized become :
Err2 = (α2 w(| I|)Es + Ec ) dx dy
2 2
(6)
Equation 6 can be improved by adopting an anisotropic approach such as in
[5]. Some methods also look at the second-order derivate ([4]).
A method which might have a lot of potential is [1]. It considers several
energies including ones given by high level descriptors such as SIFT. We will
see in section 4 an idea to improve it.
4 Further ideas
Understanding existing methods implies implementing a certain amount of
them. After doing that, some idea must be considered :
• We can make a better use (than in [1]) of high level descriptors. Whereas
in [1] the descriptor is used only fot the computation of the optical flow
where it is defined, we want to use it to guide our computation. Indeed,
those descriptors gives much more robust information than the optical
flow we could compute gives.
• We can use the fact that the optical flow is temporally smooth. In
section 3.2, we have seen how to use the fact that the optical flow
is smooth inside a frame. We could apply this idea to many frames
and trying to find the smoothest flow at each point. This idea could
be combined with the traditionals global methods in order to improve
their results.
Trying those ideas will be the main challenge of this semester.
5](https://image.slidesharecdn.com/introduction-120522081844-phpapp01/85/Introduction-to-Optial-Flow-5-320.jpg)
![References
[1] Thomas Brox and Jitendra Malik. Large displacement optical flow: De-
scriptor matching in variational motion estimation. IEEE Trans. Pattern
Anal. Mach. Intell., 33(3):500–513, March 2011.
[2] Berthold K. P. Horn and Brian G. Schunck. Determining optical flow.
ARTIFICAL INTELLIGENCE, 17:185–203, 1981.
[3] Bruce D. Lucas and Takeo Kanade. An iterative image registration tech-
nique with an application to stereo vision. In Proceedings of the 7th
international joint conference on Artificial intelligence - Volume 2, pages
674–679, San Francisco, CA, USA, 1981. Morgan Kaufmann Publishers
Inc.
[4] Werner Trobin and Daniel Cremers. An unbiased second-order prior for
high-accuracy motion estimation ?, 2008.
[5] Manuel Werlberger, Werner Trobin, Thomas Pock, Andreas Wedel,
Daniel Cremers, and Horst Bischof. Anisotropic huber-l1 optical flow.
In Proceedings of the British Machine Vision Conference (BMVC), Lon-
don, UK, September 2009. to appear.
6](https://image.slidesharecdn.com/introduction-120522081844-phpapp01/85/Introduction-to-Optial-Flow-6-320.jpg)
Introduction to Optial Flow
- 1. An Introduction to the Optical Flow Sylvain LOBRY Contents 1 Introduction 2 2 Definition of the optical flow 2 3 Overview of the existing methods 3 3.1 Local methods . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 3.2 Global methods . . . . . . . . . . . . . . . . . . . . . . . . . . 4 4 Further ideas 5 1
- 2. 1 Introduction Optical flow is defined as the apparent motion of the brightness patterns in image. Computing it leads to many applications, including video compression and motion estimation. It is also the first step of many video inpainting algorithms. Even though the first attempts of computing the optical flow has been made in 1981 by [3], the subject is the object of many recent studies leading to many improvements. In this introduction to the optical flow computation, we will start by giving a formal definition of the problem. The existing methods will be then overviewed and classified. Finally, we will specify a roadmap for this study. 2 Definition of the optical flow In the following, we will only handle the 2D case with 1D values (i.e. gray images). Let I(x, y, t) be the value of the (x, y) pixel at frame t and (u(x, y, t), v(x, y, t)) the flow at the (x, y) pixel at frame t. The main idea of most algorithms for computing optical flow is to use the Brightness Constancy : from a frame to an other, the brightness of a pixel does not change according to the flow. This can be written as : I(x, y, t) = I(x + u, y + v, t + 1) (1) Equation 1 can be approximate with Taylor series to ∂I ∂I ∂I I(x, y, t) = I(x, y, t) + u +v + (2) ∂x ∂y ∂t ∂I ∂I ∂I Introducing the abbreviations Ex = ∂x , Ey = ∂y and Et = ∂t , the following holds : uEx + vEy + Et = 0 (3) Equation 3 is often denoted as the Optical Flow Contraint. We have a single equation with two unknowns. This ill-posed problem is often called the Apperture Problem illustrated by figure 1. 2
- 3. (a) First frame (b) Second frame Figure 1: We can not tell in which way the line is moving with only those 2 frames. To compute optical flow from equation 3, we must add additional con- straints to obtain a system with more equations. Finding those constraints is the main challenge adressed by articles about optical flow. 3 Overview of the existing methods Methods for computing optical flow can be classified into two categories. The first one is known as local methods whereas the other take a global approach to the problem. 3.1 Local methods Local methods are based on the approximation that the flow does not vary inside small regions. The first method using this assumption is [3]. Following this simple idea, we can obtain, given a window of size n × n 3
- 4. and by denoting pn the nth pixel inside : Ex (p1 )u + Ey (p1 )v + Et (p1 ) = 0 Ex (p2 )u + Ey (p2 )v + Et (p2 ) = 0 ... Ex (pn2 )u + Ey (pn2 )v + Et (pn2 ) = 0 Thus, this assumption allows us to have n2 equations for 2 unknowns, giving a simple system to solve. Further developpements of this method includes a coarse-to-fine imple- mentation by pyramidal representation of the image. This have two advan- tages : • We can take into account larger displacements. • It can speed-up computations. This coarse-to-fine implementations seem to be the most used optical flow computation algorithm into video inpainting solutions. On the opposite, it has not been the object of many improvements during the recent years for optical flow algorithms. Indeed, the global methods seems to have more potential. 3.2 Global methods Global methods generally works by favoring small first-order derivates of the flow field. The first method to follow this idea is [2] which considers the opti- cal flow computation by defining the measure of departure from smoothness in the flow using an L2 norm : 2 2 2 2 2 ∂u ∂u ∂v ∂v Es = + + + (4) ∂x ∂y ∂x ∂y Using equations 3 (denoted Ec = uEx + vEy + Et ) and 4 and introducing 2 α to balance both terms, we can define the total error to be minimized : Err2 = (α2 Es + Ec ) dx dy 2 2 (5) 4
- 5. We can note that having a high value for α2 will lead to smoother flows. This factor is application dependant. Other approaches include weightening Es along with the gradients (| I|) to give less importance to areas having a high value for | I|. Indeed, it is more likely to find flow discontinuities at an edge than elsewhere. The equation to be minimized become : Err2 = (α2 w(| I|)Es + Ec ) dx dy 2 2 (6) Equation 6 can be improved by adopting an anisotropic approach such as in [5]. Some methods also look at the second-order derivate ([4]). A method which might have a lot of potential is [1]. It considers several energies including ones given by high level descriptors such as SIFT. We will see in section 4 an idea to improve it. 4 Further ideas Understanding existing methods implies implementing a certain amount of them. After doing that, some idea must be considered : • We can make a better use (than in [1]) of high level descriptors. Whereas in [1] the descriptor is used only fot the computation of the optical flow where it is defined, we want to use it to guide our computation. Indeed, those descriptors gives much more robust information than the optical flow we could compute gives. • We can use the fact that the optical flow is temporally smooth. In section 3.2, we have seen how to use the fact that the optical flow is smooth inside a frame. We could apply this idea to many frames and trying to find the smoothest flow at each point. This idea could be combined with the traditionals global methods in order to improve their results. Trying those ideas will be the main challenge of this semester. 5
- 6. References [1] Thomas Brox and Jitendra Malik. Large displacement optical flow: De- scriptor matching in variational motion estimation. IEEE Trans. Pattern Anal. Mach. Intell., 33(3):500–513, March 2011. [2] Berthold K. P. Horn and Brian G. Schunck. Determining optical flow. ARTIFICAL INTELLIGENCE, 17:185–203, 1981. [3] Bruce D. Lucas and Takeo Kanade. An iterative image registration tech- nique with an application to stereo vision. In Proceedings of the 7th international joint conference on Artificial intelligence - Volume 2, pages 674–679, San Francisco, CA, USA, 1981. Morgan Kaufmann Publishers Inc. [4] Werner Trobin and Daniel Cremers. An unbiased second-order prior for high-accuracy motion estimation ?, 2008. [5] Manuel Werlberger, Werner Trobin, Thomas Pock, Andreas Wedel, Daniel Cremers, and Horst Bischof. Anisotropic huber-l1 optical flow. In Proceedings of the British Machine Vision Conference (BMVC), Lon- don, UK, September 2009. to appear. 6