Optical Flow Estimation

We tried several optical flow methods, ranging from a highly accurate, regularised subpixel- accurate dense algorithm to cell level motion estimation using an MPEG-4 based block coding algorithm. This section provides details.

6.3.1 Regularised Flow Method

The first set of experiments were done with the Otago implementation [Galvin et al. 1998] of the Proesmans et al. [1994] multi-scale nonlinear diffusion based algorithm. This gives high- quality, sub-pixel dense motion estimates, but is computationally expensive taking around 15 seconds per frame. It results in accurate but somewhat over regularised (for this application) flow orientation estimates, typically smoothing the motion boundaries that are critical for hu- man detection. As an example, Fig. 6.4(a,b) shows pairs of input images and Fig. 6.4(c) shows the estimated flow fields. The estimates are globally accurate, correctly estimating the transla- tion in the background, but the motion arising from the person’s movement is also smoothed significantly relative to the optimal spatial derivative scale of 1–2 pixels.

6.3.2 Unregularised Multi-Scale Flow Method

The smooth motion estimates produced by the Otago method turn out to be over-regularised for our purpose, and they are also slow to compute. As an alternative we implemented a simple but

6.3 Optical Flow Estimation 79

(a) (b)

(c) (d)

Fig. 6.4.A pair of consecutive images and the estimated regularised and unregularised flow fields. (a,b) Consecutive images from a video sequence. The scene involves human motion and translation of the camera to the left. (c) Flow fields estimated using Proesmans’ flow algorithm [Proesmans et al. 1994]. The method gives accurate but somewhat over-regularised flows that blur the sharp motion boundaries, e.g. on the moving woman. (d) Flow fields estimated us- ing our unregularised method described in Sect. 6.3.2. The method estimates flows over 5×5 windows for each pixel and propagates the flow fields from coarse to fine scales, but does not perform any regularisation. The sharp motion boundaries on the moving woman are better preserved.

fast flow method based on the constant brightness assumption [Horn and Schunck 1981]. Flow is found top-down in a multi-scale approach, with initial flow estimates made at a coarse scale propagated downwards and refined in fine scale steps. The infinitesimal motion w is estimated independently at each pixel by solving a damped Linear Least Squares equation

w= −(A>A+ βI)−1A>b (6.1)

based on the constant brightness equation∂

I

/∂t+w.∇

I

=0 over a small N×N neighbour- hood, where b is an N2_{column vector encoding the temporal image differences}

_I

∆t=

I

t+1−

I

t,

Ais an N2×2 matrix of spatial gradients [

I

x,

I

y] of image

I

t, andβis a damping factor intro- duced to control numerical problems in cases where A>Abecomes singular. The model does not include any explicit spatial regularisation or smoothing and its flow estimates are visibly less smooth than the Otago ones. However the multi-scale estimation framework allows it to estimate rapid movements such as leg motions rather well. Figure 6.4(d) shows a flow field estimated using the algorithm.

Figure 6.5 describes the complete flow algorithm. Our experiments show that flow estimates computed using N=5, a scale refinement step of 1.3, and no smoothing (simple [−1,0,1] mask) while computing gradients gives the best detection performance. Hence we set these values as

80 6 Oriented Histograms of Flow and Appearance for Detecting People in Videos

Input:A pair of consecutive images

I

tand

I

t+1

Output:The estimated dense optical flow fields

– Using a scale step of 1.3, compute the number of steps in a scale-space pyramid such that the image at the top of the pyramid is at least 16×16 pixels

– Starting from the top of the pyramid, for every scale (a) Resize the images

I

tand

I

t+1to the current scale

(b) Propagate and resample the 2-D flow fields

I

w_{from the previous scale to the current}

one; if the current scale corresponds to top of the pyramid, initialise the flow fields to zero

(c) Use the flow field to warp the image

I

t. Denote the warped image by

I

0 t

(d) Compute

I

xand

I

y, the x and y gradients of

I

t0, and the difference image

I

∆t=

I

t+1−

I

t0 (e) Compute the squared gradient images

I

2

x,

I

x

I

y,

I

y2,

I

x

I

∆t, and

I

y

I

∆t

(f) If the input images are in colour, for each pixel sum the three colour channels for all of the above gradient images

(g) Compute integral arrays for each of all the gradient images (h) For each pixel,

(1) Compute (A>A+ βI)−1=  ∑i, j

I

y2+ β ∑i, j−

I

x

I

y ∑i, j−

I

x

I

y ∑i, j

I

x2+ β  (6.2) and A>b= ∑i, j

I

x

I

∆t ∑i, j

I

y

I

∆t > (6.3) using the integral array representation, where ∑i, j denotes the sum over N×N window centred on the pixel

(2) Estimate the flow vector w0using (6.1)–(6.3) (3) Update the flow vector w←w+w0

(i) If at the finest level, return the estimated flow fields

I

w

Fig. 6.5.Our multi-scale dense optical flow computation method. The steps are optimised for rapid flow computation.

default. To speed up the implementation, we use integral arrays [Viola and Jones 2001] over the gradient images

I

2

x,

I

x

I

y, and

I

y2 (used to compute A>Asee eq. (6.2)), and

I

x

I

∆t,

I

y

I

∆t(used to compute A>b, see eq. (6.3)). This allows the matrices A>Aand the vectors A>bto be rapidly

computed with only four additions per element for any window size N×N. The complete

multi-scale method is much faster than the Otago one, running in 1 second on DVD resolution 720×405 images5_.

6.3.3 MPEG-4 Block Matching

We also tested motion descriptors based on a MPEG-4 block matcher taken from the www.

xvid.orgcodec. No attempt was made to enforce motion continuity between blocks. Even

though the matching estimates were visually good, the detection results were not competitive. We think that there are several reasons for this. Firstly, block matching6_{provides only one vote} 5_{Usually feature films on movie DVDs are captured at an aspect ratio of 16:9 during production, but}

stored at a resolution of 720×576 pixels with an aspect ratio of 5:4. Typical movie players resample (zoom in or out) the images to the correct aspect ratio of 16:9 before displaying them. We resample all images to 720×405 pixels.

6.5 Experiments and Performance Comparison 81

for each cell, whereas with dense optical flow each pixel provides a separate vote into the his- togram. Secondly, the block matching flow estimates do not have deep sub-pixel accuracy (usu- ally algorithms in MPEG-4 block matching provide motion estimates up to quarter-pixel accu- racy). Experiments on rounding the flow values from the Otago code showed that even 1/10th of a pixel of rounding causes the performance to drop significantly. The need for accurate ori- entation voting is one reason for this. Thirdly, the 8×8 MPEG blocks are too large for optimal results.

In document Finding People in Images and Videos (Page 92-95)