3.2 Motion Analysis and Object Extraction Component
3.2.2 Object detection
Once the pixels have been distinguished as belonging to either background and foreground, another step must be provided in order to segment objects. This step consists of grouping foreground pixels using a technique named connected compo- nents.
Each pixel of every frame has a label assigned. The label is zero when the pixel has been classified as background, and an integer otherwise. Connected components consists of scanning an image’s labels and grouping its pixels into components based on pixel connectivity. Regarding the presented approach, a two-pass connected component analysis algorithm is applied assuming an 8-neighbour connection. In fact, the connected components labelling operator scans the image by moving along a row until it comes to a point where the pixel has been classified as foreground by the adaptive background subtraction (refer to Section 3.2.1). Then, its four neighbours that have been already scanned are examined, and two difference circumstances may take place:
• All the previously scanned neighbour’s label is zero, and therefore, belong to the background. In such case, a new label must be assigned to the current pixel.
• At least one of the neighbours belongs to the foreground, then its same label is assigned to the current pixel.
Figure 3.2: Performance example of the proposed object segmentation and tracking technique on surveillance videos
Finally, in order to ensure a more robust object segmentation, the background / foreground mask is scanned a second time. Each label has an equivalent class, an assigned unique identification number. This number replaces its equivalent label during the second scan. As a result, foreground objects are segmented as shown in Figure 3.2, where the image on the right represents the background/foreground classification of all the pixels of the image while the image on the left shows the resulting object segmentation.
3.2.3
Object tracking
Once objects have been spatially segmented, temporal segmentation must be addressed. In our approach, establishing correspondence of the spatially segmented objects between consecutive frames is accomplished using a linearly predictive mul- tiple hypothesis tracking algorithm based on a set of Kalman filters.
Object tracking involves two consecutive actions. On one hand, the prediction of the tracks related to each frame (refer to Track Calculation Technique) and, on the other, the assignment between the available tracks and the blobs detected in the frame (refer to Track Assignment Algorithm).
Track Calculation Technique
Each analysed frame has associated with it an available pool of tracks, imple- mented using Kalman filters, therefore, an available pool of Kalman models. Within that group, two different kinds of tracks can be distinguished, active tracks and ini- tial tracks. The former considers all the established long trajectories. While, the latter includes all the temporary and short trajectories. These categories are not rigid allowing a certain track to evolve over time and change to a different category.
In fact, an initial track can evolve and become an active track if a set of rules is satisfied (for example, if the trajectory is long enough); otherwise, the track can remain as an initial track or it can even be deleted. Moreover, initial tracks can also be seen as a new available pool of connected components, for instance, new measures to explain the tracks.
The track calculation technique consists on a three-stage procedure. First, if there are confirmed tracks, the current measurements are associated to these tracks using a 2D assignment algorithm (explained in Track assignment algorithm). Sec- ond, if there are still measurements left, they are associated to initial tracks using a two-point initialisation. Finally, if there are still measurements left, new initial tracks will be created using those measures.
The association of current measurements to confirmed tracks involves compar- ing each existing measurement in the current frame against the available pool of confirmed tracks. Several situations might take place:
• If a match takes place, it is used to update the model (measurements). When a model (measurement) is matched its likeliness to be used in the following frame increases.
• If no match is found for that model, the model still propagates, but its likeliness to be used later decreases. Moreover, if the object reappears in a predictable region of uncertainty shortly after being lost, the model will regain the object. The last step of the Track Calculation Algorithm consists of the generation of new tracks from the current frame’s unmatched models. The unmatched models from the current frame and the previous two frames are used to hypothesise new models. In fact, a new model is hypothesised using pairs of unmatched connected components from the previous two frames. If the current frame contains a match with enough likeliness, the updated model is added to the existing models. Moreover, in order to avoid an exponential increase of existing models in noisy situations, the maximum number of existing models is limited and the list of existing models is refreshed by removing the least probable models when excessive models exist. In fact, in noisy situations or situations with low-light conditions, short tracks are removed to avoid an exponential increase. However, it may result in random correspondences.
Kalman filters for tracking An active track has its state and observation pro- cesses modelled as a Kalman filter, where Xtis the state variable at time t and Ztis
the measurement or observation of the variable at time t for a Kalman filter where Xt represents the x and y coordinates of the foreground blob at time t. Moreover,
affected not only by the applied Kalman filter equations to estimate the motion, but also by the process and measurement noise (represented by independent normal probability distributions).
The Kalman filter estimates the tracks, calculating the process state at some time and then the noisy measurements which provide some feedback about the process state. Furthermore, the Kalman filter uses two kinds of equations, time update and measurement update. The former, time update equations are responsible of projecting forward in time the current state and error covariance in order to obtain the a-priory results for the next time step. While, the latter, measurements update equations, are responsible for incorporating the new measurement into the a-priory estimation in order to obtain an improved a-posteriori estimated measure. To sum up, the time update equations can be explained as predictor equations, while the measurement update equations can be considered as corrector equations3.
Track Assignment Algorithm
The association between measurements (blobs detected in the current frame) and the tracks explained formerly is a one-to-one matching problem. The aim is to maximise the corresponding total matching benefit. The assignment uses an auction algorithm based on the Naive auction algorithm [23].
The auction algorithm’s aim is to find equilibrium between the assignment and the total benefit of the matching. The algorithm iteratively proceeds, generating a sequence of assignments and therefore price vectors (benefits vectors). Moreover, the algorithm terminates when all the measurements are matched, and the statement is checked at the beginning of each iteration. Otherwise, another iteration of the algorithm is performed.
Each iteration of the auction algorithm performs two phases, bidding phase and assignment phase. During the bidding phase, each measurement is matched with all the available tracks and its price vector is calculated. In fact, each comparison between the measurement and each available track generates a price vector. Finally, the tracks associated to the best and second best price vectors are stored as well as the difference between them, named bidding value. Subsequently, the assign- ment phase is performed, assigning the track with the best bidding value to the measurement.
The algorithm continues performing iterations until all blobs (measurements) have a track assigned.