2.4 Filter Architecture
2.4.2 State propagation
Consider the optical flow field Φ(p, t) : R2 → R2 in continuous pixel coordinate p ∈ R2 at time t. Let p(0) be the initial pixel position and Φ(p(0),0) be the optical flow at p at time zero. For the derivation of the propagation equations modeling the temporal evolution of brightness and optical flow, it is assumed that the optical flow field Φ(p, t) is constant over time. That is8
dΦ(p(t), t)
dt ≈0 (2.8)
Under this assumption, the evolution of pixel positionp(t)is given by the ordinary differ- ential equation
dp
dt =Φ(p(t), t) (2.9)
That is, the pixel velocity at time tis equal to the optical flow at that pixel location. Given the initial conditionsp(0)and the optical flowΦ(p(0),0), one can directly compute the pixel position at future time by solving ODE (2.9).
For the derivation of the optical flow transport equation, it is convenient to think about the optical flow field as a vector attached to pixel p(t). That is, to assume that the optical flow moves along with pixelp(t). At any point in time, the optical flow associated top(t)is the
8
While the transport equations are correct under the assumption of constant flow and pixel motion in the image plane, they do not explicitly model the underlying kinematics of points in the scene projected onto the image plane. The correct formulation of the propagation of optic flow requires the concept of structure flow (Chapter 4), and is one of the principal contributions of the present thesis. The present assumption, however, provides a reasonable approximation and works well in practice, where the error in the propagation is dominated by the correction provided by the innovation.
same and is fully characterized by the initial conditions of the problem.
Φ(p(t), t)≈Φ(p(0),0) (2.10) Computing ddt of (2.10) one obtains the constant flow assumption made in Equation (2.8) and these two equations are equivalent. Since optical flow is a vector field depending on pixel position and time, its evolution model is given by decomposing Equation (2.8) in terms of its partial derivatives. Mathematically, one has
dΦ(p(t), t) dt = ∂Φ ∂p dp dt + ∂Φ ∂t ≈0 (2.11)
where ∂∂Φp ∈ R2×2is the Jacobian matrix ofΦat positionp. Replacing the pixel velocity ddpt by the optical flow value atp, one obtains Partial Differential Equation(2.12) modeling the transport of the optical flow field by the optical flow field itself9.
dΦ(p(t), t) dt = ∂Φ ∂pΦ+ ∂Φ ∂t ≈0 (2.12)
Equation (2.12) belongs to the family of non-linear hyperbolic partial differential equations used to model several types of transport processes. See the book of LeVeque [2002] for several examples of such transport phenomena.
Following a similar approach, it is possible to model the transport of an arbitrary scalar fieldc(p, t) :R2 →Rby the optical flow. The basic assumption is the conservation ofc(p, t) in time. That is
dc
dt = 0 (2.13)
Sincecis a function of both position and time, one can decompose Equation (2.13) in terms of its partial derivatives as
dc dt = ∂c ∂pΦ+ ∂c ∂t = 0 (2.14)
where ∂∂cp ∈ R2 is the gradient vector ofc atp. Equation (2.14) has the same formulation as thebrightness conservation equation(2.2) derived in Section 2.2 and can be used to create predictions of a fieldcsuch as image brightness.
Propagation equations for the filter algorithm
Equations (2.12) and (2.14) are the fundamental PDEs that model the propagation of the filter state variables in the algorithm.
For top level h = H, the propagation of the low resolution optical flow state HΦk is 9
An example in which Equation (2.12) approximately holds to zero is when the camera moves along the focal axis direction, thus inducing a divergent optical flow field such as that in Figure 1.1b. In such case, the optical flow vectors magnitude grows as it travels from the focus of expansion to the image sides. Thus, PDE (2.12) would require an extra term to inject (or remove) energy from the optical flow based on the camera and the environment kinematics. However, in practice this energy is small and can be injected to the estimated flow field at the update stage of the algorithm
§2.4 Filter Architecture 21
governed by an instance of Equation (2.12) acting on it. That is ∂HΦ
∂p Φ
H +∂ Φ
H
∂t = 0 (2.15)
Provided with initial conditions HΦ(0) := HΦk at timet = 0, one is interested in finding a
solutionHΦ(1) =:HΦk+at timet= 1corresponding to the propagated flow field for next time
step.
For lower levelsh= 1, . . . , H−1, the propagation stage models the transport of stateh∆Φk
and brightness constant parameter hYˆk by the optical flow field hΦk computed in Equation
(2.6), which in turn is being transported by itself. This is modeled by the system of PDEs ∂h∆Φ ∂p Φ h +∂ ∆Φ h ∂t = 0 (2.16) ∂hYˆ ∂p Φ h +∂ Yˆ h ∂t = 0 (2.17) ∂hΦ ∂p Φ h + ∂ Φ h ∂t = 0 (2.18)
The initial conditions of the system at timet = 0are set to h∆Φ(0) := h∆Φk, hYˆ(0) := ˆhYk
and hΦ(0) := hΦk. The PDE system is solved for time t = 1 for h∆Φ(1) =: h∆Φk+ and
ˆ
Y
h (1) =: ˆhYk+. Notice that, while there is a solutionhΦ(1)for the optical flow, this solution
never abandons the propagation block of the filter (Figure 2.3), and it can be regarded as an internal state in the propagation.
The numerical details for the solution of these equations are provided in Section 2.5