Multiple Subject Identification

When more subjects are present in the scene, the skeleton extraction step returns multiple skeletons. This information is not propagated across video frames, thus making necessary the inclusion of a tracking algorithm, within the proposed pipeline, to reliably associate each skeleton to the linked subject. Examples of this step are shown in Fig. 3.11. The tracking algorithm proposed in this module extends the method reported in [204] allowing the tracking of multiple subjects. The extension is based on three main steps. First, for each frame of the video sequence a gray-scale image b ∈ Rw×h (where h and w are height and width, respectively) is wrapped around each skeleton using a bounding box defined by the head/feet joint positions. Second, each image b is then convoluted with a set z = {z_1,1, . . . , zw,h} of rectangular filters, at multiple scales, to capture possible scale variations of tracked subjects (e.g., due to the perspective). Each kernel zi,j assigns the value 1 to the pixel in position (x, y) when 0 ≤ i ≤ x and 0 ≤ j ≤ y, and 0 otherwise. The resulting convolutions are reshaped as columns and concatenated into a feature vector x_b= {x_b1, . . . , xbm},

with m = (wh)2. Finally, for compactness, the vector xb is projected into a lower- dimensional vector v_b = {v_b1, . . . , vbn} ∈ R

n _{(with n  m) by means of a sparse} random projector [204].

The low-dimensional representation v_b is, lastly, provided to a Naïve Bayes classifier [56] to estimate the correct label ϑ_bcorresponding to a person s ∈ {1, . . . , S} in the scene: ϑb = arg max s∈{1,...,S} p(Cs|vb) = arg max s∈{1,...,S} p(Cs)p(vb|Cs) (3.13)

where p(C_s) = _S1 and p(v_b|C_s) is modeled as a Gaussian distribution:

p(vb|Cs) = 1 p 2πσ2 s e (vb−µs)2 2σ2_{s (3.14)}

3.3 Action recognition module 37

Figure 3.11. Example of multi-person skeleton tracking.

with σ_s2 being the sample variance of class Csand µs the corresponding sample mean. For ϑ_b = Cs, the values for σs2 and µs are updated according to:

µs= λµs+ (1 − λ) ˆµs (3.15)

σ2_s =

q λσ2

s+ (1 − λ)ˆσs2+ λ(1 − λ)(µs+ ˆµs) , (3.16) where λ is the learning rate (empirically set to 0.85 in our experiments), ˆµs is the mean of the v_b values, and ˆσs is the corresponding variance. As shown in Fig 3.11, the output of this tracking step is a set of skeleton sequences, each of them with a different subject label.

3.3.6 3D-DenseNet

Extracting skeleton information from video sequences is not always possible, es- pecially for those composed by face close-ups. To improve the robustness of our approach, with the aim to support the two-branch stacked LSTM in such case, a 3D CNN-type network is introduced into the pipeline. This support network, other than helping to manage missing skeletal data, allows us to perform feature extraction from the scene with which the subject is interacting. In detail, the chosen support network architecture is the 3D-DenseNet, an alternative version of the standard DenseNet [63], where all the 2D convolution and pooling layers are replaced by their 3D versions. Although this type of network could be less accurate in analysing the body movements, it does not depend on the skeletal data correctness, therefore appearing suitable in several contexts. The strength of the DenseNet derives from the use of dense blocks. In these blocks, the layers are densely connected together, receiving in input all previous layers output feature maps. A single layer of a dense block typically consists of: a batch normalization, a ReLU activation function, and a convolution (i.e., 3D convolution). A 3D convolution can be easily obtained by adding the temporal dimension into the input volume, the kernel, and the output volume of a 2D convolution. The input is represented with three dimensions: height, width, and depth. The latter is the number of images sampled from the video clip. The 3D pooling operation involves just adding the temporal dimension to the 2D pooling operation. In Fig.3.12, the overall architecture of a 3D-DenseNet with 3 dense blocks is shown. Let yc be the output of the last 3D max pooling layer of our 3D-DenseNet, a new vector named ˙y is obtained from the concatenation between y_c

and ˆy, as shown in Fig.3.13. In case of partial occlusion (Sec.3.3.4), then ˙y = ˜yθ⊕ yc (e.g., if the lower part is occluded, then θ = U ). Notice that, in this combination

sampled frames Dense Block 1 Dense Block 2 Dense Block 3 yc

3D Max Pooling 3D Convolution

Figure 3.12. 3D-DenseNet architecture with 3 dense blocks.

Skeleton extraction 3D-DenseNet 2- Branch Stacked LSTMs

Dense Layer

Figure 3.13. Integration of 3D-DenseNet within the proposed architecture.

with the 3D-DenseNet, the values contained in ˜y_θ do not need to be normalized through a softmax function. Since ˆy and ˜yθ have different sizes, a simple zero padding operation will be applied to ˜y_θ before the concatenation. Next, the vector

˙

y is given as input into a dense level and, finally, a softmax function is applied to

normalize each class probability for the final classification.