ANN Trainer

The ANN trainer is a module of PEPSO, which uses the prepared training set to train an ANN for modeling the WDS hydraulically. PEPSO uses Fast Artificial Neural Network (FANN) libraries of code to create and train ANNs. FANN is a widely used free and open source library that was initially developed in C language in 2003 (Nissen 2003). A .Net wrapper is used that let PEPSO call FANN functions directly from VB.NET environment. ANN trainer module of PEPSO has two parts. The first part creates an ANN structure and the second part trains it. By using FANN library, PEPSO can create a standard, shortcut or sparse structure for the ANN. In a standard structure, each layer has connections to the next layer, while in a shortcut structure, a neuron can be connected to neurons of all the later layers. A sparse structure allows neural networks that are not fully connected. Activation (transfer) functions of hidden and output layers of ANN can be selected from a list of functions that are introduced in Table 5.

Table 5- Available activation (transfer) functions of hidden and output layers of ANN

Name Description

Sigmoid Special case of logistic function with range of 0 to 1

Sigmoid Symmetric Hyperbolic tangent function with range of -1 to 1

Sigmoid Stepwise Stepwise linear approximation of sigmoid function

Linear Linear function

Linear Piece Bounded linear function

Sin Symmetric Periodic sine function

Cos Symmetric Periodic cosine function

Gaussian Gaussian curve function

Gaussian Symmetric Symmetric type of Gaussian function

Elliot Fast sigmoid-like function defined by David Elliott

Elliot Symmetric Fast sigmoid symmetric-like function defined by David Elliott

The FANN library provides two training stop functions: Mean Squared Error (MSE) and Bit. MSE is a common type of stop function of ANN training process for function fitting. The Bit stop function can be used for training of the binary classification ANNs. FANN also can calculate training error using two linear and hyperbolic tangent functions.

Tangent hyperbolic function aggressively selects outputs that differ considerably from target values.

By using the FANN library, PEPSO can train ANNs with standard and cascade methods. In the standard method, the number and size of ANN layers must be defined by the user. In the cascade method, the trainer automatically adds layers to the ANN structure one by one to reach to an optimum structure. ANN trainer module is also able to train the network with three different algorithms. The incremental algorithm is a standard backpropagation method where weights are updated after each training. This means the weights will be updated many times during a single epoch. The batch algorithm is similar to the incremental algorithm, but all weights will be updated at once during an epoch (at the end of calculating MSE of the entire training set). It is also possible to use the Rprop and Quickprop algorithm for training. Although these advanced batch training algorithms can be more efficient than the standard incremental and batch algorithm, they have more parameters that need to be adjusted.

The FANN library enables PEPSO to use these many different options for training the ANN. However, most of them are not familiar options for WDS operators, and designers and PEPSO do not rely on the user to select the ANN training options. Instead, if a user selects the ANN metamodel instead of EPANET hydraulic model, all the related options will be selected automatically by PEPSO.

By default, PEPSO uses the standard structure with one hidden layer for ANN training of WDSs. Using the sigmoid symmetric and linear activation functions for hidden and output layers are suggested in the case of training ANN for function fitting. For training ANNs for classification problem, the sigmoid and linear activation functions work better (Kriesel 2007). Training ANN for warning message simulation that provides binary output

(true and false: for existence or absence of warning message at a time step) is a classification problem, so the Sigmoid and linear functions are the default activation function of ANN trainer of PEPSO. However, training ANN for calculating final tank level, power demands of pumps, junction pressures, and velocities are function fitting type of problems. Accordingly, the sigmoid symmetric and linear activation functions have been used as default activation function of ANN trainer of PEPSO. For this ANN, the MSE functions are used as default stopping function of ANN trainer. The Bit function is used as the default stopping function of the ANN trainer for warning messages. By default, the batch algorithm with the back propagation training method is used for training all ANNs.

It is important to know that when PEPSO trains an ANN, it trains a metamodel that receives inputs as an initial condition of the system at the start of an optimization time step and provides outputs that are the hydraulic result of the system at the end of the optimization time step. So for each time step metamodel should be used to provide outputs, and this process should be repeated to model the WDS during the whole optimization period. For instance, for an optimization run during a 24 hour period with one hour time intervals, ANN should be used 24 times.