Artificial Neural Networks

Artificial Neural Networks (ANN) is one of the first MLAs that were developed, as shown in Figure 3-1. The basic functionality of the ANN was inspired in the architecture and the functionality of the brain (at a very basic level) (Hagan et al., 2014; Cohen, 2014; Bell, 2015; Marsland, 2015). As the “learning” process occurs within the brain, researchers thought that by mimicking the way the brain works it might be possible to create a MLA that could learn as well as the humans. Unfortunately, the processes happening within the brain are not completely known.

Interestingly, in the beginning, it was thought that ANN modelled exactly the functionality of a human brain. Later, it was found that it resembles more, in a very basic way, the functionality of an animal brain.

The brain is composed of more than 𝟏𝟎𝟏𝟏 _{cells (Hagan et al., 2014). These}

cells are called neurons and are the processing units of the brain (Hagan et al., 2014; Cohen, 2014; Marsland, 2015). Each neuron is interconnected with other neighbouring neurons (around 𝟏𝟎𝟒 _{connections per element) (Hagan et al., 2014).}

Each neuron is divided in three sections: the cell body, the axon and the dendrites. Each neuron is interconnected with other neurons’ axons through its dendrites. The connections between the neurons are called synapses.

A communication between two neurons happens due to changes in the transmitter chemicals within the brain. Consequently, the electrical potential inside a neuron changes. If the change of the electrical potential surpasses a certain threshold, this specific neuron will ‘fire’ an electrical pulse, which will travel down its axon to the other neurons connected to that axon. This electrical pulse, and all the other electrical pulses received by a neuron through its dendrites are then summed in the cell body. If the summation of the pulses surpasses the threshold of that specific neuron, a new pulse is sent from that neuron, through its axon, to all the neurons connected to that axon.

There are many neural structures and connections defined at birth. In addition, it has been found that throughout neural development, it is possible to modify “the strength of synaptic connections between neurons” (Marsland, 2015, p. 40) and create new connections (Hagan et al., 2014; Cohen, 2014; Marsland, 2015). This is called plasticity and it is believed to be the main process of learning (Marsland, 2015). Figure 3-2 shows the mathematical model of a neuron produced by McCulloch and Pitts (Marsland, 2015) and combines all the concepts that have been presented in this section. The 𝑿𝒏 represent the electrical pulse sent by other neurons through their

axons to the dendrites of a specific neuron. The cell body then sums all the electrical pulses and compares it to a pre-determined threshold. If the sum is higher than the threshold, the neuron will ‘fire’, otherwise it will not. The 𝑾_𝒏 are called the weights and it represents how strong is the connection between the two neurons. The

plasticity term is represented here. The weights will change depending on the learning process, i.e. the synaptic connection between two neurons will change. At the end of every “learning” iteration, the ANN adapts these weights and the threshold in order to ‘learn’ from a dataset.

Figure 3-2 Mathematical model of a neuron produced by McCulloch and Pitts in 1943 (Source: Marsland, 2015) Reprinted from “Machine Learning: An Algorithmic Perspective” by Stephen Marsland. Copyright © 2015 by Stephen Marsland. Used by permission of CRC Press, Florida, USA.

The mathematical model presented in Figure 3-2 can be adapted to create a MLA, i.e. the ANN. Figure 3-3 shows a single-input neuron algorithm, i.e. one neuron connected to another neuron. In an ANN, every input neuron and every output from a neuron is called a node. The input node p is multiplied by the weight w (strength of the synaptic connection). The input b is called the bias node. This bias node will allow an algorithm to adapt even if the neuron input is a zero value. The input node and the bias node are sum up and compare to a threshold function f. The output a is defined by the equation 𝒂 = 𝒇(𝒘 ∗ 𝒑 + 𝒃). The threshold function, also called transfer function, determines the output of the neuron. The most common transfer functions are hard limit, linear and sigmoid.

Once the input p and the bias b are summed and analysed by the transfer function, in a supervised learning approach, output a is compared to the expected (real) value. The difference is then be used to update the value of weight w, as shown in equation (5).

𝒘

← 𝒘

− 𝜼 𝒚

− 𝒂

∗ 𝒑

(5)

In equation (5), 𝒘_𝒏 is the weight that will be update, i.e. learning process; 𝒚_𝒏 is the expected (real) value and 𝒂𝒏 is the output of the neuron; 𝒑𝒏 is the input value;

𝜼 is a parameter called the learning rate. The learning rate parameter is a predefined value that will determine the rate of change of the weight value. If the learning rate is too high, the learning rate is faster but it might overshoot the best weight value. If the learning rate is too low, the learning rate is slow but it is more accurate in reaching the best weight value.

An ANN can have multiple input nodes (many neurons connected to one neuron) as shown in Figure 3-4. More complicated ANN can be designed with multiple inputs and output nodes within many layers (the layer in between the input nodes layer and the output nodes layer are called hidden layers). In some cases, connecting the output of one neuron back as input can generate feedback. If no feedback is present, the layer is called feed-forward. Independently of the number of input, output and hidden layers, the process is the same as explained for the single input ANN (Figure 3-3).

Figure 3-4 Multiple input neuron ANN design (Adapted from: Hagan et al., 2014)

3.4.2.1 ANN to predict sleep while driving

ANN has been used to predict the sate of sleep in drivers (Patel et al., 2011; Vuckovic et al., 2002). For example, Patel et al. (2011) used the heart rate variability of 12 truck drivers (mean age 47, SD=11) to determine different levels of sleepiness. Heart rate variability was used as it has been related to a decrease in mental workload and increase in driver fatigue, as presented in chapter 2. A single layer feed-forward ANN was used. The transfer function was a bipolar logistic function (Patel et al., 2011; Hagan et al., 2014). To train the ANN, heart rate variability of awake truck drivers and heart rate variability of sleepy truck drivers was input into the MLA. The heart rate variability was presented as a 30x30 pixel spectral image of the power spectral density analysis obtained through Fast Fourier Transformation of the electrocardiography data obtained from each participant, as seen in Figure 3-5. The ANN had 900 input nodes (30x30 pixels) and two output nodes (awake and sleep nodes). The activation function was a bipolar logistic function and a learning constant of 0.5 gave the best results. The NN was trained using five data sets (5 participants) and was tested using five data sets. The dataset was obtained from Lal and Craig (2002) and was classified into alert and sleep by subjective visual analysis (using a video image of the participants). The algorithm gave 90% accuracy when predicting sleepiness in drivers.

Figure 3-5 Power spectra density of the heart rate variability of “awake” drivers (a) and “sleepy” drivers (b) during a driving simulator study. (Source: Patel et al., 2011). Reprinted from “Applying neural network analysis on heart rate variability data to assess driver fatigue” by M. Patel et al. Copyright © 2011 by M. Patel et al. Used by permission of Elsevier.