Convergence data from real world CFD simulations have been used to evaluate the developed model. The data set used is for a transonic wing design and has convergence histories for CLand CD. An example of a CLconvergence history can be seen in Fig. 2.6a and an example CDcnvergence history can be seen in Fig. 2.6b.
The data set is broken down into three angles of incidence (α = 1.35◦, 1.65◦ and 6.00◦) and each convergence history is from a viscous coupled Euler CFD method. Each data set has a total of 150 flow iterations, which is a typical level for practical use and would have taken approximately 40 minutes to run on one CPU. The data set has been provided by the Flight Physics Group at QinetiQ Ltd, Farnborough. Prediction accuracy is compared to the data at the 150th flow iteration and Table 3.1 provides what these target values are and Fig. 3.2 illustrates exactly where on the convergence history this is.
Table 3.1: Data Sets Data Set Target 1.35◦CD 0.01922 1.65◦CD 0.02176 6.00◦CD 0.10333 1.35◦CL 0.56842 1.65◦CL 0.60677 6.00◦CL 1.13679
The number of flow iterations needed for training and validation, as well as a visual inspection of predic- tion trajectories of each ensemble member were used as the performance indicator when determining the various parameters that would be used for the final test work. The mean prediction of all ensemble members for various CD and CL data sets were also used. In terms of the network structure, the number of memory layers and connections in the network were determined by systematically changing the default Shark settings. These settings changed the recurrent connections between neurons, as well as bias connections. A single memory layer was ultimately used and the bias terms were removed from the hidden and output neurons. The input neurons do not receive recurrent activations from any other neurons, whereas the hidden and output neurons receive activation from themselves and any other neurons. This means that by having recurrence and delayed inputs to the network they incorporate features from all types of RNNs discussed in Section 3.3 (i.e. Elman, Jordan and NARX).
Figure 3.4 is an example of a recurrent neural network with 3 input, 1 hidden and 1 output neuron. The solid arrows represent forward connections and the dotted arrows represent recurrent connections, which originate from the neurons at (t − 1).
Neuron State at (t-1) Neuron State at (t) 𝑦(𝑡 − 2) 𝑦(𝑡 − 1) 𝑦(𝑡) 𝑦(𝑡 + 1)
Figure 3.4: An example of a Recurrent Neural Network (3 input, 1 hidden and 1 output neuron)
Hidden layers of 1, 5 and 10 neurons were tested to investigate the impact of different network sizes. For reference the total number of connections in each type of network is summarised in Table 3.2.
Table 3.2: Number of Connections
Number of input neurons 1 hidden neuron 5 hidden neurons 10 hidden neurons
1 7 57 187
2 9 63 198
3 11 69 209
With regards to the other parameters in the methodology, Table 3.3 presents a summary of those investi- gated, including the range of values considered and the final values used. The warm-up-length is unique to the Shark library, but there is no indication what quantity of data should be used. However, it needs to be less than the amount of data in the training and validation data sets. Also, the validation data set needs to be kept small, as predictions are to be made using as few flow iterations as possible, which means both values were kept to a minimum.
The early-stopping method used to stop training initially just considered the validation error, with training being stopped if it went below a threshold. This was deemed not suitable as it became clear that a suitable value of error was difficult to determine. Therefore, the number of increases in the validation error during a set number of training epochs was investigated. This method requires two parameters to be determined, the “strip length” and the number of validation error “increases”. The strip length refers to the number of training epochs that are monitored and was previously discussed in Section 3.4 and illustrated in Fig. 3.1. The number of “increases” refers to the number of times in the strip length the error needs to increase before training is stopped.
The number of training epochs was increased from 500 to 10,000 to improve learning, however the performance from this parameter is dependent on the early-stopping values used, as these are designed to stop training. The maximum number of training epochs is only used to ultimately stop training if the early-stopping criterion is not met.
The variance range is used on the normalised data and a variety of ranges were investigated. The ranges tested were based on variance values of the final predictions whilst determining the other parameters.
Table 3.3: Setup Parameters
Parameter Values Investigated Value
Warm-up-length - 5
Validation data set size - 10
Strip length 5, 10, 20 20
Validation error increases 3, 5, 15 15
Maximum training epochs 500, 1,000, 10,000 10,000
Variance range 0.00003 - 0.00001, 0.0003 - 0.0001, 0.001 var
0.0005 - 0.0001, 0.001 - 0.0001, 0.1 - 0.01, 0.0001
The parameters are kept the same for each ensemble member and data set. Each ensemble setup was run ten times to account for the random initial weights and Section 3.6 presents the test results, including the mean prediction and standard deviation of these ten runs. The absolute error and percentage error, as well as the average number of training epochs needed to train the individual networks in the final ensemble are recorded, along with the average number of data points needed across the ten runs for training and validation.