This is a type of neural network architecture that brings inputs to a given region from another region located at an earlier stage along a particular processing pathway as shown in Figure 2.6. This neural network architecture was adopted for this study considering its ability to model the river water quality parameters efficiently. Through this connections, information flows in one direction along a connecting pathways, from the input layer via the hidden layers (in case of multilayer perceptron) to the final output layer, respectively.
In feed-forward neutral network architecture, the output of any layer does not affect that same or preceding layer. Varoonchotikul (2003) reported in his study that Feed-forward neural networks architecture (FFNNA) are found to perform best for one time-step forecasting.
61 Feed-forward neural networks architecture is further classified as single layer or multilayer perceptron.
2.12.7.1 Feed-Forward Single Layer Network Architecture
This is the simplest kind of neural network architecture, which consists of a single layer of output neurons; the inputs are fed directly to the outputs via a series of synaptic weights (Wk) as shown in Figure 2.7. In this way, it can be considered as the simplest kind of feed-forward network. The sum of the products of the synaptic weights and the inputs is calculated in each neuron, and if the net value is above the threshold, the neuron strikes and takes the activated value; otherwise it takes the deactivated value. The neurons with this kind of activation function are also called linear threshold units. In the literature the term perceptron often refers to networks consisting of just one of these neurons. Nevertheless, it was known that a multi-layer perceptron is capable of producing Figure 2.8: schematic diagram of Feed Forward Multilayer Neural Network Architecture.
(Source: Varoonchotikul, 2003) Input layer
Hidden layer
Output layer X1i
X2i
X3i
X4i
Wij
62 any possible Boolean function. Although a single layer perceptron is quite limited in its computational power, it has been shown that neural networks of parallel threshold units can approximate any continuous function from a compact interval of the real numbers into the interval of -1 to 1 (Auer et al., 2008).
Feed-Forward Multi-Layer Perceptron
This is a two-layer neural network that is capable of calculating XOR or any Boolean problems. A feed-forward multi-layer perceptron has a layer called hidden layer in between the input and output layers, which consist of computational units called neurons, usually interconnected in a feed-forward way. Each neuron in one layer is directly connected to the neurons of the subsequent layer as information is being propagated from the input layer through the hidden layer to the output layer. In many applications, the neurons of these networks use the sigmoid transfer function which is also known as the output function to model the inputs and outputs pattern. The sigmoidal output function has a continuous derivative function which can easily be computed and is very suitable with back propagation training algorithm.
A multi-layer neural network can compute a continuous output instead of a step function.
The universal approximation theorem for neural networks states that every continuous function that maps intervals of real numbers to some output interval of real numbers can be approximated arbitrarily closely by a multi-layer perceptron with hidden layer (Awu et al, 2017). This theory holds for a wide range of activation functions, such as sigmoidal functions, tan function etc. A feed-forward multilayer perceptron-type of ANN is the
63 most suitable type of ANN for learning the stimulus-responds relationship for a given set of measured data (Minns and Halls, 1996), and this was the type employed in this study.
In feed-forward multilayer perceptron, the choice for neuron arrangement for hidden layer(s) has been by trial and error method. This could be the reason for yet-to-be breakthrough in a neural network development that behaves exactly like the human counterpart. Therefore, this study sheds more light on the adoption of finite neuron pattern architectures to optimize neural network model.
2.12.7.2 Recurrent Neural Networks (RNN)
Recurrent networks interconnect neurons within a particular region that are considered to be at the same layer along the processing pathway. This network differs from feed forward network architectures in the sense that there is at least one feedback loop. RNN propagate data from ‟downstream‟ processing elements to earlier units. Thus RNN, have feedback connections between neurons of different layers or loop type self-connections (Ahmad and Ismail, 2004). This implies that the output of the network not only depends on the external inputs, but also on the state of the network in the previous time steps as shown in Figure 2.8. Thus, in these networks, for example, there could exist one layer with feedback connections. There could also be neurons with self-feedback links, that is, the output of a neuron is feeding into itself as input. There are several advantages of RNN, the first one being that RNNs have the capability to retain values from previous cycles of processing, which can be used in current computations. This advantage allows RNN to produce complex, time varying outputs in response to simple static inputs. Since a RNN can have connections between units of any of the layers, the
64 output of each unit has to be identified in terms of time steps, for example outputs at time step t−1 can be inputs at time step t-n.
(Source: Varoonchotikul, 2003)
2.13 Applications of Artificial Neural Network (ANN) In Water Quality Modeling