Multi-Task Convolutional Neural Network: High Level Feature Ex-

In this section , we will firstly introduce the overall view of our CNN model and then describe how we select the architecture.

Overall of the network

The CNN models are constructed with several layers, where each layer has maps and parameters. As shown in Fig 7.3,the maps of each layer would be calculated according to this layer’s parameter and the maps of the previous layer, since its input maps are the output maps of previous layer.

Fig. 7.2 Examples of gait energy images (GEIs).

inMap

_layer

outMap

parameters

7.2 System Overview 115

Convolutional layer The purpose of convolutional layer is to extract the local patterns. It is parameterized by following factors: the number of maps, the kernel sizes and stride. In this approach, we denotes C(d, f , s) to represent a convolutional layer, where d is the number of feature maps, f is the kernel size and s is the stride. For instance, C(10, 5, 1) means the convolutional layer have 10 maps with vernal size 5 and the stride is 1. The size of output are determined by the size of input, the kernel size and the stride.

Non-linear activation function The convolutional layer works as a linear filter. In order to form a nonlinear complex model. The nonlinear activation function would be added with the convolutional layer. The common activation function includes tanh, sigmod, relu, so f plus, etc. Here is an example of activation functions shown in Fig 7.4.

−2.5 −2 −1.5 −1 −0.5 0 0.5 1 1.5 2 2.5 −1 −0.5 0 0.5 1 1.5 2 2.5 3 x y

Nonlinear activation functions

Sigmoid tanh Relu Softplus

Fig. 7.4 An example of non-linear activation function

Max-pooling layer The purpose of Pooling layer is to subsample the feature maps and reduce the resolution. The reason is that those features extracted from convolutional layer have precise positions. It maybe harmful for following procedure, e.g. classifying since different training instances with the same label have different precise positions. Inspired by [200, 209], the max-pooling method is used in this approach

Normalization The normalization operation is able to give the trained model a better generalization. Inspired by [200], the local response normalization (LRN) is applied for

generalization purpose. The equation of LRN is shown in eq 7.2 yi_m,n= x i m,n [k + α ∑min(N−1,i+l/2)_{j=max(0,i−l/2)} (xm,nj )2]β (7.2)

where the summation operates over a l “adjacent” kernel maps at the same spatial position m, n, and N is the total number of kernels in the layer. In this approach, we followed [200] use the empirical parameters: k = 2, n = 5, α = 10−4, β = 0.5.

Fullly connected layer The previous operations in convolution layer, nonlinear activa- tion, normalization and pooling layer could be defined as a large convolutional stage. The input has been converted to lots of low-resolution feature maps. In full link layer, these small size feature maps are concatenated in to a long vector, where such a vector plays a same role as those manually designed features. Then, the long vector is fed to a one-hidden-layer neural network, which works similar as linear regression. After that, an nonlinear activation function is applied to finally form the output feature map.

Output layer - Multiple layer perception (MLP) The final stage of the model is MLPs. Each MLP is corresponding to an attribute identification task, e.g. Subject, Probe Scene (the wearing conditions), Probe view, etc. A softmax function will be added to each MLP to finally predict the attributes.

The structure of the networks and the hyper-parameters were empirically initialised based on the training data and previous works using CNNs, then we setup cross-validation experiment to optimize the selection of network architecture in section 7.2.2.

Architecture Selection

The overall CNN architecture is shown in Fig.7.5. The network consists of three convolution stages followed by one fully connected layers. Each convolution stage includes convolutional layer, non-linear activation function, local normalization and pooling layer. The lower layers are finally shared by N split layers, each of which is fulled connected with the topmost shared layers. Each split layer is respected to one attribute identification task and contains several additional fully connected layers with size 128. The size of split layer is k, which equals to the class number in respected task.

In order to determine the parameters, we use the training data to do cross validation experiment. We firstly set default values and choose the best one according to the cross validation (CV) performance, which is shown in Fig 7.6.

7.2 System Overview 117

Fig. 7.5 The architecture of our convolutional neural network. (conv) stands for convolutional layer, (nonl) stands for nonlinear activation function, (norm) stands for normalization and (pool)stands for the max-pooling layer. The network contains three stages, each of which is consisted of convolution layer, non-linear activation layer, local response normalization layer, and max-pooling layer. Only convolution and max-pooling layers which change the data size during operating, are illustrated here.

Table 7.1 Architecture of our CNN

Layer Type Feature maps and Size Kernel

1 Input 1 map with 124×124 neurons

2 Convolution C1(24, 11, 1) 24 maps with 114×114 neurons 11×11

3 Max pooling 24 maps with 57×57 neurons 2×2

4 Convolution C2(32, 6, 1) 32 maps with 52×52 neurons 6×6

5 Max pooling 32 maps with 26×26 neurons 2×2

6 Convolution C3(40, 5, 1) 40 maps with 22×times22 neurons 5×5

7 Max pooling 40 maps with 11×times11 neurons 2×2

8 Fully connection 512 neurons

Task 1 Task 2 Task 3

9 Fully connection 128 neurons 128 neurons 128 neurons

(a) Cross validation to evaluate six group of map numbers

(b) Cross validation for six group of filter size

(c) Cross validation error of four activation functions

7.2 System Overview 119

Max-pooling layer The parameter of max-pooling layer is chosen empirically based on the input GEI image size and the conditional layer at same convolutional stage.

Kernel numbers in convolutional layer With respect to the width of the network. The pooling layer which follows the convolution layer, will decrease resolution of the feature map. To prevent the information from being lost too quickly, the filter size will generally increased. Here, we setup experiment to compare with six groups of filter number. The validation testing error with epoch are shown in Fig.7.6a. From the figure, the red line which stands the filter number group of 24-32-40, has the least testing error. Hence, we choose this group as our filter numbers in each convolution layer.

Kernel size in in convolutional layer According to learning theory, if the architecture has too much capacity, it tends to overfit the training data and has poor generalization. We first evaluate the filter size by comparing six empirically selected group of kernel size. The average result of cross validation among the six group of filter size is plotted in Fig.7.6b. We select the filter size of 11-6-5 for the network as its lowest CV error.

Activation Function Selection. After the convolutional layer parameters are determined, the ReLU has been reported faster convergence than others in next step CV test, where the result is shown in Fig.7.6c. From the figure, the network is not convergence using softplus or sigmoid activation function in a learning rate of 0.01 within the observed epoches due to the diminishing gradient flow [214]. While the ReLU makes the convergence faster and yield a better performance as reported in most recent CNN approaches.

After the cross validation experiment, the parameters chosen in our final architecture are listed in Table 7.1.