Differential methods for optical flow computation

The most common algorithms to compute optical flow are the differential methods. These methods model the temporal change of intensity due to the underlying optical flow present in the image sequence. The approach is based on the well known brightness conservation equationthat is the starting point of all differential methods Barron [1994].

Let Y(p, t) : R2 ×R → R denote the image brightness at pixel position p ∈ R2 and timet∈R. The common assumption about image brightness is that the objects composing the scene are made of Lambertian materials. That is, the light scattered by the material is invariant to the viewer’s view angle. Given this assumption, it is possible to say that the value of image brightness is constant over time. In other words, the total rate of change of image brightness over time is zero. Mathematically, one has

dY

dt = 0 (2.1)

Since brightness is a function of independent variablespandt, one can decompose Equa- tion (2.1) in terms of its partial derivatives as

dY dt = ∂Y ∂p dp dt + ∂Y ∂t = 0 (2.2)

where ∂Y_{∂p ∈} R2 is the image gradient vector and d_dp_t ∈ R2 is the relative change in pixel position, that is, the optical flow atp. Equation (2.2) is thebrightness conservation equation

modeling the relationship between temporal brightness change and optical flow, and is the common starting point of all differential techniques.

4_{https://durian.blender.org/} 5_{https://www.blender.org/}

§2.2 Background 13

Equation (2.2) imposes only one constraint to the two-dimensional optical flow vector d_dp_t. That is, only the component of d_dp_t parallel to the image gradient ∂Y_∂p is relevant to (2.2). This is known as theaperture problemin the literature. Algorithms need to use some form of data integration or regularization to recover the full two-degrees of optical flow.

In differential methods, there are two families of algorithms to find the optical flow from a sequence of images. The first family, local-based methods, consists in algorithms that use a local support window around each image pixel to find the underlying optical flow. The second family,global-based methods, impose global constraints over the whole image that the resulting optical flow field must satisfy.

Local-based methods use a finite support window around each image pixel to find optical flow. Within this window, flow is assumed to be constant and all image data within the window is used to recover one single flow vector. The most popular algorithm of this family was proposed by Lucas and Kanade [1981]. LetΩpdenote a support window around pixelp. The

basic formulation of Lucas-Kanade method is to minimize cost function εp,Φ =

X

q∈Ωp

k∂pYqΦ+∂tYqk2 (2.3)

for the unknown optical flowΦ:= d_dp_t atp. Here∂pYqdenotes the image gradient vector at

pixelqand∂tYqis the image temporal derivative atq.

A popular implementation of Lucas-Kanade method is the one developed by Bouguet [2001] and available in OpenCV6. To support large optical flow fields, this implementation uses a pyramidal approach to subdivide the problem in different image scales. At coarse scale, the algorithm computes a coarse estimation of flow from the low resolution data. This coarse estimate is refined using higher resolution data from the next scale level, until the original resolution level is reached.

Local based methods, in general, are subject to noise in the data within the support window. In regions of the image where there is no gradient information, that is, in textureless regions of the image, it is not possible to recover optical flow. Moreover, the estimated optical flow suffers from the aperture problem, where only the flow in the direction of the image gradient can be computed.

Global-based methodsimpose a prior constraint over the optical flow at each pixel. Typi- cally, this prior has the form of a smoothness constraint where the optical flow field is assumed to be smooth across the image. One example of global based methods is the seminal work by Horn and Schunck [1981]. In their work, the authors formulate the optical flow problem as a variational problem where the brightness conservation equation is complemented with a flow smoothness term to regularize the computation of flow in image regions with poor texture data. The goal is to minimize cost functional in Equation (2.4) with respect toΦ.

ε2=

Z Z

k∂pYΦ+∂tYk2+α2(∂xφx)2+ (∂yφx)2+ (∂xφy)2+ (∂yφy)2dxdy (2.4) Equation (2.4) can be solved using calculus of variations, expressing the integral in terms of its underlying Euler-Lagrange system of partial differential equations. The resulting system

couples the solution of each pixel and its neighbors, and requires an iterative method to find the solution (for example, Gauss-Jordan method).

The key advantage of global-based methods over local-based ones is that estimations at each pixel are well defined thanks to the smoothness term. In regions of the image where image gradient information is poor, the smoothness term in Equation (2.4) dominates over the image term. Consequently, the flow in those regions will be filled with flow coming from regions with high texture content such as image edges.

The work of Horn and Schunck opened a new research field in computer vision for the computation of optical flow using variational methods. One of such algorithms is proposed by Broxet. al. Brox et al. [2004] which is used as reference in the experimental evaluation in Section 2.6. Global-based methods typically outperform local-based methods in terms of accuracy in benchmarks such as Middlebury, Kitti and Sintel. However, local based methods perform better in terms of runtime, as it will be illustrated in the next section.