Bundle Merge Pools

Thenth andhth bundles are deemed similar if the metricς_n,his greater than a thresholdς_τ = 0.8. This means that 80% of the measurements along the trajectory overlap duration satisfy the error similarity constraint. All pairs of bundles are tested in this way.

The nth bundle may have several merge candidate bundles that satisfy ςn,. > ςτ. This requires a method for deciding which candidates should be a part of the final merged bundle. There are situations that commonly arise where not all the candidate are suitable for the final bundle. For example, if a bundle is partially similar to two bundles that are absolutely different, caution must be taken in the merging of these bundles. A decision must be taken about how to merge these bundles while keeping the bundles with no similarity separate. A robust solution to this problem is to make the candidate bundles vote for each other. A candidate bundle h1

votes for bundle h2 ifςh1,h2 > ςτ.

Fig. 4.15 shows examples of the similarity metrics ςn,hfor 5 trajectory bundles. Sinceςn,h=

ς_h,n only the metricsς_n,h are shown, where n < h. In this example we are attempting to merge bundles 2, 3 and 4 to bundle 1. Hence bundles 2, 3 and 4 are defined as the candidate bundles

4.2. Initialization: Motion Model Estimation and Merging 75

Figure 4.14: The merging of the trajectory bundles at frame 12 in theCalendar and Mobile

sequence. Top left: The bundles after Mean Shift clustering, this is the input to the merging step. Clockwise from top right: The merging result using the thresholds on the elements of the similarity matrix sτ = 0.5,0.3,0.1 respectively.

for bundle 1, since ς_1,2 > ς_τ, ς_1,3 > ς_τ and ς_1,4 > ς_τ. These metrics are highlighted in green in fig. 4.15. Note that bundle 5 is not a candidate for bundle 1 since ς1,5 < ςτ. All metrics where

ς_n,h< ς_τ are highlighted in red.

The candidate bundles 2 and 4 both vote for each other since ς2,4 = ς4,2 > ςτ. However bundle 3 does not receive a vote from either bundle 2 or 4 becauseς2,3 < ςτ andς2,4 < ςτ. Hence only candidate bundles 2 and 4 are merged with bundle 1.

The final merged bundles along with the unmerged bundle at the current iteration are then considered for merging at the next iteration. When there are no more mergers the bundle merging step is complete.

ς_1,2 = 0.91 ς_1,3 = 0.82 ς_1,4= 0.95 ς_1,5 = 0.10

ς_2,3 = 0.50 ς_2,4= 0.93 ς_2,5 = 0.00

ς_3,4= 0.47 ς_3,5 = 0.01

ς_4,5 = 0.20

Figure 4.15: An example of the similarity metrics ς_n,h among 5 bundles. Since ς_n,h=ς_h,n only the metrics ςn,h are shown, where n < h. The metrics for which ςn,h> ςτ are coloured green, otherwise they are colouredred.

4.3

Summary

This chapter introduced the general framework for oursparse trajectory segmentation technique. The segmentation task is broken down into two stages; aninitialization and arefinement stage. Theinitialization stage is used to obtain a rough estimate of the trajectory segmentation, which is subsequently improved in the refinement stage. Therefinement stage introduces spatial and temporal constraints on the trajectory bundles. This chapter discuss theinitialization stage in detail, while the details of the refinement stage are exposed in chapter 5.

In theinitialization stage the trajectory data is expressed in a new feature space that allows trajectories with similar motion to be clustered. We perform trajectory clustering in this space using the Mean shift [29] algorithm. To reduce over segmentation introduced from the clustering step, we then proceed to merge the trajectory bundles. We model the 2D motion of the trajectory bundles using Affine transformations, and merge bundles that have similar motion models.

5

Sparse Trajectory Segmentation:

Model Refinement with Spatial Smoothness

The label field obtained from the initialization stage discussed in the previous chapter provides a reasonable guess of trajectory segmentation. However, in this initial segmentation there are no spatial constraints on the trajectories in each bundle. Hence fig. 5.1 shows an example from the Calendar and Mobile sequence where the lack of these spatial constraints prevent the trajectories in the bundles from being confined to specific image regions. For example, the area occupied by the calendar is an image region where it is desired to have only green trajectories. In this region there are unwanted background (red) and train (yellow) trajectories. Also we have unwanted green trajectories in the background. Therefinement stage discussed in this chapter, introduces spatial constraints to make the bundles represent specific image regions.

The first step in the refinement stage involves relabelling the trajectories according to a posterior defined on the label field. This posterior distribution uses a MRF prior for enforcing spatial smoothness on the trajectory labels. The refined label field is then obtained as the MAP estimate generated using theα-expansion Graphcut optimization algorithm [135].

AModel relaxation step follows the GC optimization. In this step another attempt is made to merge the trajectory bundles, as their motions are more coherent. The framework for merging is as outlined in section 4.2, with a change in the similarity metric. The new similarity metric here is based on the separation of the trajectories in the bundles. This metric unlike the one defined in section 4.2, facilitates for the merging of bundles generated by the motion of non-rigid objects.

Figure 5.1: The effect of no spatial constraints in theinitializationstage are demonstrated with frames 22 (left) and 25 (right) of the Calendar and Mobilesequence. Top row: The white boxes highlight trajectories that lie outside the desired image regions of their corresponding bundles. Here all trajectories with the same colour belong to a particular bundle. Bottom row: The illustration in the top row displayed on a white background for clarity.

The MRF prior included in the posterior distribution design mentioned previously encourages neighbouring trajectories to have the same label. The illustration on the left of fig. 5.2 shows a set of trajectories that all belong to the same bundle. The MRF constraint correctly allows these trajectories to have the same label in spatially separated groups. Each group is outlined with a closed contour. Note that the groups are separated spatially by trajectories belonging to other bundles. That is, there are trajectories belonging to other bundles that exist in the gaps between these groups.

Even though having all these spatially separated groups of trajectories belonging to the same bundle does not violate the MRF smoothness constraint, we require that every group be a separate bundle. By doing this, we can fulfill the requirement that a bundle represents a single

5.1. Graphcut Optimization 79

Figure 5.2: The local region constraint algorithm uses Delaunay triangulations to find net- works of connected trajectories in the bundle on the left. The bundles in the center and right are the two sets of connected trajectories obtained from this analysis. These two set of trajectories are the new bundles generated by the algorithm.

image region.

We use a novel algorithm (local region constraint algorithm) that performs Delaunay trian- gulations at every frame to identify each group of connected trajectories. The algorithm then forms a new bundle from each group identified with more than three trajectories. The middle and right illustrations in fig. 5.2 show the two groups of trajectories that are used to form new bundles from the initial bundle on the left.

The merging of bundles and the formation of new bundles for new image regions in theModel relaxation step updates the knowledge of the motion models. Therefore these models are then reestimated and we iterate the GC optimization and Model relaxation steps until there are no changes in the label field.

5.1

Graphcut Optimization

As outlined in the previous section, the GC optimization step relabels the trajectories by gener- ating the MAP estimate for a distribution defined on the label field. Given there areN labels in the initial label field, corresponding to the number of trajectory bundles. Thetth trajectory has a label _Lt ∈ {1 : N} according to the bundle it belongs to. The posterior distribution p(Lt|.) for the trajectory labels is given below.

p(_Lt|X,L∼t)∝px(Xt|Lt)ps(Lt|L∼t) (5.1) Where_L∼tare the neighbouring labels of trajectoryXtthat has the labelLt,px(Xt|.) andps(Lt|.) are the likelihood and MRF prior distributions respectively. The likelihood distributions for the

N possible labels are dependent on the member trajectories in the bundles. The design of the likelihood is motivated by observations about the behaviour of trajectories in a bundle. Firstly, trajectories tend to follow the average motion in a bundle in some sense. Secondly, the spread of the trajectories in the image plane tends to be consistent from frame to frame. This point about the consistent spread of the trajectories in each bundle is illustrated in fig. 5.3 using four trajectory bundles from theCalendar and Mobile sequence. It may be observed that the texture

Figure 5.3: Illustrated in each row are the trajectory bundles for the four independently moving objects in the Calendar and Mobile sequence. Each row shows the endpoints (dots) of the trajectories in the bundles for frames 1,5,10,15 and 20. For every bundle, the spread of the trajectories in the image plane tends to be consistent from frame to frame.

of the objects in the scene determines the spatial separation of the trajectories. And since the texture of the objects changes slowly from frame to frame the spacing between the trajectories in each bundle is approximately constant. Thetop row shows the trajectories for the ball. Even though the ball is rotating the relative separation between a pair of these trajectories over their paths is roughly constant.

As before, Affine transformation modelsAn_f,f+1for the bundles are used to generate a predic- tion error for the spatial locations of the trajectories. Here the bundle label with the minimum prediction error for a trajectory provides the maximum motion likelihood.