Classification-based Fusion Methods

Considering the exponential growth of the types and amount of media, the new object classification approaches try to convey information captured by different means in order to achieve a higher robustness and accuracy in the classification pro- cess. Several classification-based multimodal fusion methods have been proposed in the literature in an attempt to categorise the multimodal input data into one of the pre-defined classes associated to the application under analysis, however, the most popular multimodal fusion techniques are: (i) Support Vector Machines, (ii) Bayesian Inference, (iii) Dempster-Shafer Theory, (iv) Dynamic Bayesian Networks, (v) Neural Networks and (vi) Maximum Entropy Model. Each technique provides a set of advantages and disadvantages preselecting its suitable specific applications and scenarios. In the remainder of this section, each technique is detailed and described along with some highlighted approaches and surveillance applications.

Support Vector Machines (SVMs)

SVMs acquired great popularity for data classification, especially, in the domain of multimedia, where different approaches have used this technique for different ap- plications such as face detection, object classification, modality fusion, etc. SVM is a supervised learning method, which assuming a set of input data vectors, provides an optimal binary classification, partitioning the input data into the two training classes. Typically, SVMs are used for multimodal fusion, assuming the set of in- puts represents the scores given by individual classifiers. Multimodal fusion and classification using SVMs partitions the input data, applying different kernel func- tions which allow non-linear classification. SVM formulation is further detailed in Appendix 9.4.

Many existing literature approaches use the SVM-based fusion scheme. Nirmala et al. [151] proposed a multimodal image fusion technique using Shift Invariant Discrete Wavelet Transform (SIDWT) for surveillance applications. This approach addressed the fusion of visual and infrared images, extracting their SIDWT and using SVM to fuse the transforms at feature-level. The proposed multimodal im- age fusion technique combined two information sources, but enabled its extension providing a scalable approach. To compute the SIDWT, images were divided into non-overlapping blocks of fixed size and three features including energy, entropy and standard deviation, were computed for each block. The SVM was trained based on the extracted features for each block and determined whether the wavelet coefficient block from the visual or infrared image was to be used. Finally, the fused image was obtained by performing inverse SIDWT on the selected coefficients. Arsic et al. [10] targeted the automatic detection of certain passenger’s behaviour in an airplane

situation, i.e. aggressive, nervous, tired, etc, using an SVM during the classification stage. A set of low-level features based on difference imaging were extracted from different parts of the image such as skin colour regions, face or the entire image. The proposed low-level features were based on the global motion, representing its movement or mean deviation. Finally, a vector containing all features was created and classified using a SVM based on the polynomial kernel.

Bayesian inference

Bayesian inference combines multimodal information by applying rules of pro- bability theory [128]. Multimodal information sources provide either features or decisions from individual classifiers which are combined to derive the inference of the joint probability of an observation or decision [164]. Bayesian networks allow the use of prior knowledge about the likelihood of the hypothesis to be utilized in the inference process. New observations or decisions can be used to update the a-priori probability in order to compute the posterior probability of the hypothesis. Finally, the Bayesian inference fusion method allows for uncertainty modelling.

The Bayesian inference method has been used in the literature to combine mul- timodal information due to its possibility to adapt as the information evolves as well as its capability to apply subjective or estimated probabilities when empirical data is absent. Due to these advantages, Bayesian inference has been used for different tasks, such as speech recognition or video analysis. In surveillance, Bayesian in- ference has been applied for combining classification results for various applications [16, 15]. Atrey et al. [13] fused multimodal information using the Bayesian inference fusion approach for event detection in surveillance scenarios, such as standing and talking, running and shouting, walking or standing and door knocking. Meuter et al. addressed vision-based traffic sign recognition in a hybrid classification method based on a decision tree and a Bayesian fusion algorithm [142]. The fusion module combined the classification results of the different classifiers over time and fused similar signs on both sides of the road, taking advantage of the redundancy existent in German roads, where identical signs are mounted on both sides of the road. Dempster-Shafer Theory

The Dempster-Shafer evidence theory allows the inclusion of belief and plausibil- ity values to represent evidences and their corresponding uncertainty in the fusion process, rather than representing the evidence using only uncertainty values [11]. According to the Dempster-Shafer theory, an hypothesis is characterised by belief and plausibility. Whilst the degree of belief implies a lower bound of the confidence, the plausibility represents the upper bound, delimiting the confidence interval or

the possibility of the hypothesis to be true [12]. After the assignment of a proba- bility to every hypothesis, the decision regarding the hypothesis is measured by a confidence interval. Multimodal fusion using the Dempster-Shafer theory applies evidence combination rules to fuse multimodal information.

Multimodal fusion techniques have acquired great relevance in recent years, the inclusion of higher levels of freedom within the fusion process has been used in dif- ferent applications. For instance, vehicle classification based on Dempster-Shafer theory was addressed by Klausner et al. [110]. The proposed approach fused single- source classifier’s results into a matrix of uncertainty intervals. The authors applied the SVM distance mass function and the Dempster-Shafer belief function to classify objects into three categories, including large trucks, small trucks and cars accord- ing to a set of visual and acoustic features. Moreover, Dempster-Shafer theory of evidence was applied on other surveillance applications such as gender profiling. In [129], authors proposed a multimodal fusion technique based on the Dempster- Shafer theory to combine the partial decisions provided by different gender profiling techniques to overcome existing limitations such as the face occlusion or body shape alteration. The provided experiments exhibited an improvement versus single pro- filing or classic fusion results.

Dynamic Bayesian Networks (DBNs)

Multimodal fusion considering the temporal axis requires specific models to de- scribe the evolution of the observed data. For the analysis and fusion of this type of information, Bayesian inference fusion methods can be extended to Dynamic Bayesian Networks (DBN), also called probabilistic generative model or a graphical model [12]. DBNs have been applied in a diverse range of multimedia applications where the time-series data affected the analysis due to its two main advantages, (i) its ability to model multi-node dependency and (ii) to integrate the temporal dependency of the multimodal data. Despite, the variety of DBN systems proposed, the most popular and simplest form of a DBN is the Hidden Markov Model (HMM). HMMs have been used for diverse applications from recognising tennis strokes to gait-based human identification. Additionally, their ability to exploit the spatio- temporal patterns has driven human activity recognition research. A comprehensive review of modelling, recognition and analysis of human activities and interactions was presented by Turaga et al. [205]. Amongst the existing human activity recog- nition techniques based on DBNs, techniques could be categorised according to the number of agents involved in the activity and/or the amount of information sources. Oliver et al. [152] proposed a system for detection of two person interactions using coupled hidden Markov models (CHMMs). The CHMMs was a variant of HMMs

which integrated two or more sources of information to model and recognise human behaviour. Liu and Chua [123] proposed a technique to classify three agent activi- ties, including groups approaching, walking together or meeting and turning back, applying Observation Decomposed Hidden Markov Model (ODHMM). While, Due et al. [66] proposed to decompose an interaction into multiple interacting stochastic processes and proposed a coupled hierarchical durational-state dynamic Bayesian Network. Suk et al. [193] analysed human interactions based on their moving tra- jectories. Each human interaction was decomposed into elementary components or sub-interactions, which were modelled individually using HMMs, and finally assem- bled using a directed graph.

Despite DBNs great development in human activity recognition, multimodal fu- sion using DBNs was also applied in other surveillance applications such as vision- based traffic monitoring [96, 34], vehicle detection [52], scene description [106, 166] or action recognition based on contextual information [145].

Neural Networks

Neural Networks (NN) are another approach for multimodal fusion, providing a non-linear mapping between the input information sources and the output decisions. The NN method consists of a network including input, hidden and output nodes. The input nodes accept information from the different sources while the output nodes provide the results of combining the input information or decisions. The mapping between the input and output nodes, using the hidden nodes, defines the network architecture and therefore its behaviour. The architecture and the weights defining its topology can be adjusted during the training phase to obtain the optimal fusion results [28].

In recent years, multiple applications have used NNs as a multimodal fusion technique. In general scenarios, applications such as speaker tracking [236] or struc- tural damage detection [102] applied NNs to combine different information sources. In surveillance scenarios, NNs have been widely used in traffic control [153]. For example, in [141], traffic magnetic sensors captured the information to detect traffic incidents using NNs. Traffic flow prediction was tackled using a radial basis function neural network [222] or genetic-based NNs [211]. Furthermore, NNs and SVMs were compared for the prediction of traffic speed in [206]. Authors determined that SVM was a viable alternative to NNs for short-term prediction due to the high dependence of NNs performance to the training stage.

Despite NNs are suitable for high-dimensional problem spaces and generating high-order nonlinear mapping, NNs present several challenges, including (i) slow training and (ii) complexity to select an appropriate network architecture according

to the application under analysis. These challenges limited NNs impact on the multimedia analysis compared to other fusion methods [12].

Maximum Entropy Model

The maximum entropy model presents a statistical classifier which provides a probability of an observation belonging to a particular class based on the input information. The maximum entropy model is used in multimodal fusion, classifying fused multimedia observations, coming from different acquisition sources, into a set of predefined concepts. The maximum entropy model-based fusion method learns possible correlations between the extracted features and the selected concepts to build statistical models. Assuming a classification problem, where Oi and Oj are

two different types of input observations/features and X is the concept to classify, maximum entropy models calculate the probability of the observations belonging to the class X using an exponential function:

P (X|Oi, Oj) =

1 ne

F (Oi,Oj) _(6.1)

where n is the normalisation factor and F (Oi, Oj) is the combined observation vector.

The maximum entropy model has been applied for semantic multimedia indexing and annotation. In [133], text and image features were combined to index and retrieve images using the Maximum Entropy Model. The proposed approach was evaluated against the Naives Bayes classifier using the Reuters-21578, Corel Images and TRECVID 2005 datasets, exhibiting a precision improvement of, between a 5 - 25% achieved by the Maximum Entropy Model approach. In [9], a multimedia- content automatic annotation approach based on maximum entropy models was presented. Statistical models were calculated extracting colour, texture and shape features to represent each classifying concept. For each concept to be predicted, the set of relevant models were extracted to estimate the probability of the observation to be a particular concept.