Artificial Neural Networks

Artificial neural networks (ANNs) are the computing systems inspired by the biological neural networks of the animal brains aiming to resolve perception and recognition problems. They are capable of approximating nonlinear relationships between inputs and outputs.

The basic elements of the ANNs are neurons, which receive input, change their internal state (activation) correspondingly, and produce output depending on the input and activation. ANNs are formed by connecting the output of certain neurons to the input of other neurons forming a directed, weighted graph. The adaptive weights along paths between neurons and the functions that compute the activation can be modified (adapted) by learning algorithms. However, unlike the biological neural networks, once an ANN is formed, the connections between artificial neurons are not usually added or removed.

One of the earliest and best known computational models for neural networks based on mathematics and algorithms was introduced by Warren McCulloch and Walter Pitts in 1943 [65] called threshold logic, however, the technology available at that time were insufficient for them to work on practical problems. In 1970s and 1980s, with the development of the computational resources, a number of new, more complex ANNs started to emerge.

Nowadays, deep learning neural networks (DLNNs) have gained a lot of popularity in both the academic circles and the general public [19], [66]. In fact, deep learning is the latest name of ANNs [66]. However, DLNNs have also gone beyond the original neuroscientific perspective, but appear to be a more general principle of learning multiple levels of composition, which can be applied in machine learning frameworks that are not necessarily naturally inspired [66]. Currently, the popular variants of the ANNs include, but not limited to 1) feedforward neural networks [4]; 2) deep convolutional neural networks (DCNNs) [21], [24]; 3) recurrent neural networks (long short-term memory) [67]; 4) deep belief networks [68] and 5) spiking neural networks [69], etc.

As the feedforward neural networks (NNs) and DCNNs are directly related to the main topic of this thesis, this section will focus on reviewing these two particular types.

2.2.2.1. Feedforward Neural Network

Feedforward NNs are a class of ANNs whose neurons form an acyclic graph where information moves only in one direction from input to output. They are extensively used in pattern recognition. The general architecture of a multilayer feedforward NN is depicted in Figure 3.

Figure 3. General architecture of a multilayer feedforward NN.

A typical multilayer feedforward neural network consists of three types of layers: input layer, one or more hidden layers and output layer. Each layer is a group of neurons receiving connections from the neurons of the previous layer. Neurons inside a layer are not connected to each other.

Input layer is the first layer of the network and it receives no connections from other layers, but instead, uses input vector as its activation. Input layer is fully connected to the first hidden layer. Each hidden layer is fully connected to the next hidden layer, and the last hidden layer is fully connected to output layer. The activation of output units is considered to be the output of the feedforward neural network. The output of the network is the result of the transformations of input data through neurons and layers in a form of distributed representation consisting of a huge number of weighted local representations across the entire networks.

Backpropagation procedure (backward propagation of errors) is the most widely used supervised learning algorithm for adapting connection weights of feedforward NNs [19]. Weights of the network are tuned to minimise square error between the system output and the target value:

𝜀 = ∑ (𝑦_𝑖− 𝑡_𝑖)2

where 𝜀 stands for overall square error; 𝑦𝑖 is the system output corresponding to the ithinput

and 𝑡_𝑖 is the respective target value.

Backpropagation is nothing more than a practical application of the chain rule for derivatives. The key insight is that the derivative of the objective function with respect to the input of a layer can be computed by working backwards from the gradient with respect to the output of that layer. The backpropagation equation can be applied repeatedly to propagate gradients through all modules, starting from the output layer all the way to the input layer. Once these gradients have been computed, it is straightforward to compute the gradients with respect to the weights of each module [19].

Although feedforward NNs are very powerful, they suffer from the following drawbacks:

1) The structure is complex and considered as a black box;

2) The structure identification requires a number of ad hoc decisions, i.e. number of neurons, number of layers;

3) The training process is computationally expensive and once it is finished, the parameters of the NNs cannot be updated and requires a full retraining if new data samples are given.

2.2.2.2. Deep Convolutional Neural Network

Convolutional neural network (CNN) was firstly introduced by Kunihiko Fukushima [70] almost 50 years ago and was significantly improved later [71].

Currently, DCNNs are the state-of-the-art approaches in the field of computer vision. A number of publications have demonstrated that DCNNs can produce highly accurate results in various image processing problems including, but not limited to, handwritten digits recognition [22], [24], [72]–[74], object recognition [21], [23], [75], [76], human action recognition [77], [78], remote sensing image classification [79]–[82], etc. Some publications suggest that the DCNNs can match the human performance on the handwritten digits recognition problems [22], [24], [73], [74].

Except the input and output layers, a typical CNN consists of a number of hidden layers, which can be a combination of the following four types:

1) Convolution layer; 2) Pooling layer;

3) Normalisation layer; 4) Fully connected layer.

The convolutional and pooling layers in DCNNs are directly inspired by the classic notions of simple cells and complex cells in visual neuroscience [83].

Convolutional layers apply a convolution operation to the input, passing the result to the next layer. The role of the convolutional layer is to detect local conjunctions of features from the previous layer.

Pooling layer is for merging semantically similar features into one. A typical max pooling unit (which is the most commonly used one) calculates the maximum of a local patch of units in each sub-region of the image. Neighbouring pooling units take input from patches that are shifted by more than one row or column resulting in the reduction of the dimensionality of the representation and the increase of robustness to small shifts and distortions.

Normalisation layer is useful when using neurons with unbounded activations (e.g. rectified linear neurons), because it permits the detection of high-frequency features with a big neuron response, while damping responses that are uniformly large in a local neighbourhood.

Fully connected layer connects every neuron in the previous layer to every neuron in it, which is in principle the same as the feedforward NN as described in subsection 2.2.2.1. However, one major difference between the fully connected layer in the DCNN and the one in the feedforward NN is that the fully connected layer of the DCNN only connects to a small region of the input volume, while the fully connected layer of feedforward NN connected to all the neurons of the previous layer.

Recent DCNN architectures have 10 to 20 layers of rectified linear neurons, hundreds of millions of weights, and billions of connections between units. Thanks to the very large progress in hardware, software and algorithm parallelisation, the training times can be only a few hours if enough computational resources are provided, which could be extremely expensive. The performance of DCNN-based vision systems has caused most major technology companies, including Google, Facebook, Microsoft, Baidu to initiate research and development projects and to deploy DCNN-based image understanding products and services [19].

Nonetheless, ANNs/DCNNs still have a number of deficiencies and shortcomings: 1) The computational burden of training using huge amount of data is still very heavy; 2) The training process is opaque, and the classifier has low or no human interpretability (black box type);

3) The training process is limited to offline and requires re-training for samples with feature properties different than the observed samples, as well as for samples from unseen classes;

4) Its internal structure identification involves a number of ad hoc decisions, i.e. number of layers, the order of the layers, the types of convolutional kernel, the type of pooling.