The goal of a tracking system is to estimate the locations and sizes of the targets in a video sequence. In order to accomplish this task, the trackers have to know the appearance of the targets. A template, or an exemplar, provides the information about the appearance of the targets, and thus plays an important role in the tracking system. Unfortunately, due to the fact that the targets may be non-rigid objects and the viewpoint of the camera may change in the video, the appearance of the targets may not remain the same during tracking. Therefore, in order to reliably track the targets throughout the video sequence, a template updating algorithm is required to adapt the template to the newly observed appearance of the targets.
Many template updating algorithms have been developed recently. The most na¨ıve approach uses the previous observation as the template for the tracker to find the most probable location of the target in the next frame. Though simple, this approach has problems because the estimation of the target’s location inevitably has errors so that the bounding box may include the background or other objects.
If we take the previous observation as the template in the next frame, the errors will accumulate and finally lead to loss of targets in the future [36]. When the target is rigid, an alternative is to first use the previous observation as the template to obtain
a rough estimation of the target’s location. Then, we can conduct a local search utilizing the reliable first template, and start the search from the rough estimated location in order to correct the rough estimation [36]. However, this technique does not work when the targets are deformable such as hockey players.
Toyama and Blake introduced the exemplar tracker [57]. They learned the representation and the transition of exemplars offline. During tracking, the exemplar tracker infers both the position of the target and the exemplar used by the tracker.
The problem of the exemplar tracker is that the exemplars only encode a fixed num-ber of appearance variations of the targets. In order to introduce more appearance variations, Elgammal et al. modeled the distribution between the exemplars and the intermediate observation using a non-parametric distribution [12].
Instead of constraining the appearance of the targets to be similar to some fixed number of templates, Black et al. [4] constrained the target to lie on a learned eigen-subspace. Their EigenTracking algorithm simultaneously estimates the loca-tion of the targets and the coordinates of the target’s appearance in the subspace to minimize the distance between the target’s appearance and the subspace. However, since the EigenTracker learns the appearance model off-line, it cannot fully capture the appearance variations of the targets. Recently, Ross et al. [50] and Ho et al. [17]
proposed algorithms to efficiently update the subspace on-line.
Khan et al. [24] formulated the EigenTracking algorithm in a probabilistic framework, and applied it to track honey bees (Figure 2.1 (a)). Instead of using PCA, they used Probabilistic Principal Component Analysis (PPCA) [56], which is a generative model that generates the observed variable from the hidden vari-ables by factor analysis model with isotropic Gaussian noise. They projected the appearance of the target to a single latent subspace using Probabilistic Principal Component Analysis (PPCA) [56]. Figure 2.1 (b) shows the probabilistic graphical model of their tracking system. Since the hidden variables are continuous and have high dimensionality, they used a Rao-Blackwellized particle filter [9] to infer the
5. Experimental Results
We used the Rao-Blackwellized particle filter to track an unmarked honey bee in an observation hive. The auto-mated recording of trajectory data has significant appli-cations in the study of honey bee behavior and physi-ology [19]. This application presents substantial chal-lenges due to temporary occlusions that occur in the close proximity of other bees, complex variations in the appearance of the tracked bee, and the unpredictability of the tracked bee’s movements.
(a)
(b)
Figure 2: (a) Tracking honey bees in a hive presents sub-stantial challenges for a tracker due to temporary occlu-sions that occur in the close proximity of other bees and complex variations in the appearance of the tracked bee.
(b) Mean target image with the first 5 principal compo-nents, or eigen-bees, estimated from a training set
The test sequence was recorded at 15fps at 720x480 pixels and scaled to 360x240 pixels (see Figure 2a ).
We used 40 iterations of EM (see [20]) to learn the PPCA image model from a training image data set con-sisted of 172 color bee images, each measuring 14 by 31 pixels. The mean image and the first five principal components (eigen-bees) are shown in Figure 2b. The center of rotation of the rectangle was offset 15 pix-els from the top of the image. Because the motion of the bee is difficult to predict, we used a Gaussian mo-tion model where the target state is updated according tol|lt−1= A(θt−1)∆l + lt−1by sampling from a zero
Table 1: When using the RBPF with 500 hybrid parti-cles, increasing the complexity of the PPCA model by adding principal components both improved the quality of the trajectory and decreased the number of failures.
Particles Failures MSE
500 69 52.76±67.68
1500 44 42.37±63.19
3000 33 31.71±51.10
6000 20 21.11±45.03
Table 2: In a “plain” particle filter where the state vector contains 12 appearance coefficients which are sampled, instead of computed analytically, the number of particles required was increased substantially.
mean Gaussian∆l ∼ N(0, diag(σ2∆xσ2∆yσ∆θ2 )) with preset variances,σ∆x= 2, σ∆y= 3, and σ∆θ= 0.3 for position and orientation, whereA is matrix that rotates a locationl according to the angle θt−1. The varianceQij
in the coefficients was set to and7Iqand10Iqwhen the position of the target changed by more than 2 pixels.
Using increasingly rich subspace models by adding additional PPCA components both decreased the num-ber of failures and improved track quality. In Table 1 we show the results of using the RBPF with 500 hybrid particles. A failure was recorded when the position re-ported tracker deviated more than half the width of the target, 7 pixels in the image, from the ground truth tra-jectory. When a failure occurred, the tracker was reini-tialized at the ground truth position and allowed to re-sume tracking. We measured the quality of the trajectory by computing the mean squared error (MSE), or squared distance, of the position reported by tracker from the ground truth. The zero principal component model is equivalent to using a simple Gaussian image model with meanµ and variance σ2Id, which results in poor track-ing. With q=12 components in the PPCA model we re-port no failures and track quality markedly improved.
The same results are qualitatively illustrated in Fig-ure 3, where we have shown the corresponding 2D tra-jectories for theq = 0 and q = 12 component models.
Through the correlations between pixel color captured by the principal components the tracker is able to more
Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’04) 1063-6919/04 $20.00 © 2004 IEEE
(a)
over the appearanceat−1[16]:
P (lt−1, at−1|Zt−1) = P (lt−1|Zt−1)P (at−1|lt−1, Zt−1)
conditioned on particlei and the measurements Zt−1: α(i)t−1(at−1)= P (a∆ t−1|l(i)t−1, Zt−1) (6) Substituting the hybrid approximation (5) into the ex-pression for the marginal filter (4), we obtain after some manipulation the following Monte-Carlo approximation to the exact marginal Bayes filter:
P (lt|Zt) ≈ k
Figure 1: Dynamic Bayes network for the Rao-Blackwellized particle filter. The target state contains both the locationltand the subspace coefficientsat
It might seem that we could now implement the same importance sampling scheme as with the particle filter.
However, the complicated form of the approximation (7) makes that intractable in general. In theory, it is possible to directly sample from the approximation (and hence do away with the importance weights), but this is both com-putationally and analytically difficult in all but the sim-plest cases. Hence, to obtain a practical algorithm we make one additional assumption, namely that the mo-tion model for the locamo-tionltdoes not depend on the appearanceat−1at timet − 1:
P (lt|lt−1(i), at−1) = P (lt|lt−1(i)) (8)
The dynamic Bayes network for this model is shown in Figure 1. We can now move the motion model out of the integral in (7), obtaining
Now we can do importance sampling in the usual way, using the empirical predictive density
iwt−1(i)P (lt|l(i)t−1) as the proposal density. The final algorithm is summarized below.
3.2. RB Particle Filter Summary
For each time stept, starting from the posterior approx-imation{l(i)t−1, w(i)t−1, α(i)t−1(at−1)}Ni=1, repeat forj ∈ 1..N:
1. Randomly select a particlel(i)t−1from the previous time step according to the weightsw(i)t−1. 2. Sample a new particle ˆlt(j)from the chosen model:
ˆl(j)t ∼ P (lt|lt−1(i))
3. Extra step: calculate the posterior densityα(j)t (at) on the subspace coefficientsat:
α(j)t (at) = k(j)t P (Zt|ˆlt(j), at)×
at−1
P (at|ˆl(j)t , l(i)t−1, at−1)α(i)t−1(at−1) (10)
Note that this is where the measurementZtis in-tegrated, conditioned on the chosen location ˆlt(j). Also, the integral yields a predictive density onat
given a move from locationl(i)t−1to locationl(j)t . 4. Calculate the importance weightw(j)t as the
inte-gral over the unnormalizedα(j)t (at), i.e., wt(j)= 1/k(j)t
wherekt(j)is the normalization constant in (10).
4. RB EigenTracking
4.1. Probabilistic PCA
The integrals in the general Rao-Blackwellized particle filter framework become analytically tractable when we
Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’04) 1063-6919/04 $20.00 © 2004 IEEE
(b)
Figure 2.1: A Rao-Blackwellized Particle Filter for EigenTracking: (a) Khan et al. [24] presented the Rao-Blackwellized Particle Filter for EigenTracking to track honey bees. (b) The probabilistic graphical model of the Rao-Blackwellized Particle Filter for EigenTracking. lt represents the location of the bee at time t, at
represents the appearance of the bee at time t, and Ztis the observation at time t.
These figures are taken from [24].
probability of the hidden variables for greater efficiency.
Instead of using a single eigen-subspace to model the appearance of the tar-get, researchers also tried to model the appearance by using multiple subspaces.
For example, Lee et al. [28] presented a system that models the faces by a set of eigen-subspaces. In the learning phase, they divided the training data into groups according to the identities of the faces, and learned a single eigen-subspace for each group. During the runtime, they first recognized the identity of the face based on the history of the tracking results, and chose the eigen-subspace belonging to the recognized person. Then, they utilized the chosen eigen-subspace to generate a tem-plate for the tracker, and tracking could be performed by searching for the location whose image patch is most similar to the template.
Recently, Lim et al. [29] have introduced an appearance updating technique that first transforms the observation to a latent space by Locally Linear Embedding (LLE) [52], and then uses the Caratheodory-Fejer (CF) approach to predict the next position of the appearance in the latent space, and finally inversely transform the point from the latent space to the observation space by a set of Radial Basis
244 J. Giebel, D.M. Gavrila, and C. Schn¨orr
Fig. 2. Learning Dynamic Point Distribution Models
clustering. Instead the clustering is based on a similarity measure derived from the registration procedure. To be specific, the average distance between corre-sponding points after alignment. Only if this distance is lower than a user defined threshold, the shapes fall into the same cluster and the registration is assumed valid. For details, the reader is referred to [8].
Linear subspace decomposition: A principal component analysis is ap-plied in each cluster of registered shapes to obtain compact shape parameteri-zations known as “Point Distribution Models” (PDMs) [3]. From the Ncshape vectors of cluster c given by their u- and v-coordinates
sci= (uci,1, vci,1, uci,2vi,2c , ..., uci,nc, vci,nc), i ∈ {1, 2, ..., Nc} (1) the mean shape ¯scand covariance matrix Kcis derived. Solving the eigensystem Kcecj = λcjecj one obtains the 2nc orthonormal eigenvectors, corresponding to the “modes of variation”. The kcmost significant “variation vectors” Ec= (ec1, ec2, ..., eckc), the ones with the highest eigenvalues λcj, are chosen to cover a user specified proportion of total variance contained in the cluster. Shapes can then be generated from the mean shape plus a weighted combination of the variation vectors
sc= ¯sc+ Ecb. (2)
To ensure that the generated shapes remain similar to the training set, the weight vector b is constrained to lie in a hyperellipsoid about the subspace origin. Therefore b is scaled so that the weighted distance from the origin is less than a user-supplied threshold Mmax
kc
Markov transition matrix: To capture the temporal sequence of PDMs a discrete Markov model stores the transition probabilities Ti,jfrom cluster i to j. They are automatically derived from the transition frequencies found in
Figure 2.2: Learning Dynamic Point Distribution Model: A principal com-ponent analysis (PCA) is applied to in each cluster of registered shapes to obtain compact shape parameterization known as “Point Distribution Model” [7]. The transition probability between clusters is also learned. This figure is taken from [16].
functions (RBFs). The experimental results show that their approach can accurately predict the appearance of the target even under occlusions. Urtasun et al. [59] and Moon et al. [38] also transform the observation to a latent space and then predict the next appearance. Instead of using LLE and CF, they use Gaussian Process Dynamic Models (GPDM) [62] to predict and update the template of the tracker.
The work most similar to ours is Giebel et al. [16]. They presented a sys-tem that can track and detect pedestrians using a camera mounted on a moving car. Their tracker combines texture, shape, and depth information in their ob-servation likelihood. The texture is encoded by the color histogram, the shape is represented by a Point Distribution Model (PDM) [7], and the depth information is provided by the stereo system. In order to capture more variations of the shape, they constructed multiple eigen-subspaces from the training data, and the transition probability between subspaces were also learned (Figure 2.2). During runtime, they used a Particle Filter to estimate the probability of the hidden variables of the track-ing. To reduce the number of particles, they also used a smart proposal distribution based on the detection results. Our tracker shares the same merits. However, in the template updating part, we infer the probability of the hidden variables using the Rao-Blackwellized Particle Filter to increase the speed. In multi-target tracking, we
(a) Two consecutive frames Figure 2.3: Decomposed Optical Flow: (a) Two consecutive frames. (b) The Decomposed Optical Flow constructed by decomposing the optical flow of (a) into four channels (FX+, FX−, FY+, FY−), where FX+, FX−, FY+, and FY− represent the optical flow along the X+, X−, Y+, and Y− directions, respectively.
use the Boosted Particle Filter that incorporates the cascaded Adaboost detector to obtain fast and reliable detections.