There are a number of open questions that are not addressed in this thesis that would need to be answered before the full benefit of the convergence based prediction method could be realised for these industries. These, along with some specific ideas related to the developed models are discussed next.
The main open question associated with the long-term convergence prediction method is what is the best way to use the surrogate to benefit an aerodynamic optimisation task that is reliant on computationally expensive CFD simulations for evaluating performance?
Firstly, the computational saving could be exploited for each of the hundreds or thousands of individual CFD simulations required as part of an online optimisation study. Using the methodology in this way would reduce the overall computation time for an optimisation task. The complexities of integrating an optimiser with a CFD package is not a trivial task and model management would play an important
part in terms of where in an optimisation process the method could be used and which individuals it would evaluate. It is felt that if the convergence prediction methods were used at the beginning of an optimisation process, when accuracy is not as important, the search would still be guided to the region of the global optimum, due to the trends in the design space being maintained.
Secondly, promising areas of the design space could be identified using the methodology and then fully converged CFD simulations conducted in these areas to improve the overall optimisation search. Similarly, CFD simulations could be stopped in less promising areas of the design space based on either the predictions or the classification given to the convergence history. Furthermore, the insights into aerodynamic performance and the classification of convergence histories could be used to determine the number of flow iterations needed to make suitable predictions.
Alternatively, the methodology could be used to provide performance information for decision variables in a traditional parameter based surrogate. The model could be used to populate more points in a design space or the same number of points but in less time. It would need to be established if the variations in the design space were less than the noise of the predictions and if the level of accuracy achieved by the predictions were suitable for directing a design search.
Another open question is how the error and computation time of the convergence predictions compare to a CFD simulation with a coarser grid or less accurate numerical method? This question could only be answered by using an additional extensive set of CFD simulations, but is a worthwhile study as it would indicate whether it is more beneficial, in terms of computation time, to predict performance or use a lower fidelity simulation that takes less time to run. Additional CFD simulations that also included a greater number of decision variables, as part of each individual design, would also be useful to investigate a more comprehensive comparison between the convergence based surrogate and the parameter method.
As there are different classifications of convergence history, alternative methods of convergence pre- diction could be investigated, including different model types. For example, monotonic convergence histories were the easiest profiles to predict and are very similar to an inverted exponential curve of the form ys = as(1 − e−xs), where xsrepresents the flow iteration and ys the performance measure. By using an optimiser to solve this equation to find the parameter as, the horizontal asymptote of the monotonic convergence history could be found. General equations could be determined for the different classifications of convergence histories and then the specific parameters of these equations could be found. Once determined, the resulting formulas could be used to predict the performance measure after a number of flow iterations.
Finally, although the study has been exclusively conducted with CFD convergence data, simulation data from similarly converging engineering tasks (i.e. finite element analysis) should also be tested. The results presented indicate the modeling technique would also benefit other types of simulations that were also determined using an iterative numerical process.
tions seen in the transonic data sets could be investigated further. A common error may be why the trends are maintained and if this is understood then the prediction accuracy may improve.
Also, a combination of the different surrogate methods and the technique used to combine them (e.g. weighting), may yield interesting results, by introducing additional diversity into an ensemble. In addition, instead of feeding back each individual predictors values, perhaps the average of a number of predictors could be fed back, sharing information between individual learners. Alternatively, data from a single member that is deemed to be the best could be used by all networks, although a selection method to establish this best individual would need to be determined.
A number of changes could be made to the models presented, including the transfer functions used, as well as the learning algorithms. These could either be changed for all networks to investigate the performance difference or for some individuals to increase diversity. Changing the requirement for the networks to be able to form a square matrix during the H-MOEA should also be investigated. This requirement clearly restricted the allowable connections (i.e. no bias terms) and could potentially restrict the diversity of the networks. Also, additional chromosomes for each individual in the H-MOEA could be introduced to incorporate additional features. These could include the number of hidden neurons and selection of input data, which would allow for different network sizes to be investigated, as well as different values of Taken’s Theorem. The structure of a network and the corresponding weights from a prediction that has a similar convergence classification could also be used as the starting point for the learning of either the convergence based prediction methods or the parameter based method.
For the parameter based surrogate, a global optimiser similar to the H-MOEA could be adopted. Different data sampling techniques could also be investigated, but similar to above, this would be reliant on the availability of additional data sets. The majority of surrogate models used for aerodynamic optimisation have used parameter based prediction methods, which means there are a number of other model types that could be compared to the convergence based methods presented as well.
As mentioned, the large number of parameters that need to be determined for all methodologies in- vestigated has been difficult to manage, particularly for the global and local search mechanisms of the H-MOEA. Despite this limitation the developed methodologies have performed well and with further investigations or an optimisation of the parameter values, the performance may be improved further.
In addition to this, the convergence based prediction method offers many benefits over a more traditional surrogate. However, the true impact that these types of surrogates can have on an optimisation task is not completely clear at the conclusion of this research.
Chapter 7
Publications and Awards
Journals
• C. Smith and Y. Jin, “Evolutionary Multi-Objective Generation of Recurrent Neural Network Ensembles for Time Series Prediction”, Neurocomputing, vol. 143, pp. 302 - 311, 2014.
Conferences
• C. Smith, J. Doherty, and Y. Jin, “Convergence Based Prediction Surrogates for High-lift CFD optimization”, Royal Aeronautical Society - Applied Aerodynamics Conference, Bristol, July 2014. • C. Smith, J. Doherty, and Y. Jin, “Multi-objective Evolutionary Recurrent Neural Network En- semble for Prediction of Computational Fluid Dynamic Simulations”, Congress on Evolutionary Computation (CEC), 2014 IEEE WCCI, Beijing, July 2014.
• C. Smith, J. Doherty, and Y. Jin, “Recurrent neural network ensembles for convergence predic- tion in surrogate-assisted evolutionary optimization”, Computational Intelligence in Dynamic and Uncertain Environments (CIDUE), 2013 IEEE SSCI, Singapore, pp. 916, April 2013.
Awards
• Best student Paper Runner up at 2014 IEEE Congress on Evolutionary Computation, Beijing, China, July 6-11, 2014.
Chapter 8
Bibliography
[1] L. J. Clancy, Aerodynamics. Longman Scientific and Technical, 1991.
[2] B. Barzegar, “The evolution of aerodynamics capability,” Flight Physics Integration, Airbus, UK, Tech. Rep., 2014.
[3] FIA, “2015 formula one sporting regulations,” 29 June 2014.
[4] B. Massey, Mechanics of Fluids, 7th ed. Stanley Thornes (Publishers) Ltd, 1998.
[5] R. Bhaskaran and L. Collins, “Introduction to CFD basics,” Cornell University-Sibley School of Mechanical and Aerospace Engineering, 2002.
[6] R. H. Barnard, Road Vehicle aerodynamic Design: An Introduction, 2nd ed. MechAero Publishing, 2001.
[7] H. Versteeg and W. Malalasekera, An Introduction to Computational Fluid Dynamics: The Finite Volume Method. Prentice Hall, 2007.
[8] S. McBeath, Competition Car Aerodynamics: A Practical Handbook. Haynes Publishing, 2006. [9] M. J. Garcia, P. Boulanger, and S. Giraldo, “CFD based wing shape optimization through gradient- based method,” in Proceedings of the International Conference on Engineering Optimization (EngOpt), Brazil, 2008.
[10] E. Benini, R. Ponza, and A. Massaro, “High-lift multi-element airfoil shape and setting optimization using multi-objective evolutionary algorithms,” J. Aircr., vol. 48, no. 2, pp. 683–696, 2011.
[11] D. Hunter, “The ATI uk aero design capability development re-booted,” Aircraft Research Association, Tech. Rep., 2014.
[12] P. Raj and A. Friedman, “On timely and cost-effective prediction of aerodynamic data to meet aircraft design needs,” in Advanced Aero Concepts, Design and Operations–Applied Aerodynamics Conference 2014, UK, 2014.
[13] R. Bond, “The possible time reduction of CFD solutions resulting from grid sequencing,” in Proceedings of the IEEE 1994 National Aerospace and Electronics Conference, NAECON, vol. 2, USA, 1994, pp. 1025–1032.
[14] N. Wyman, “How multigrid solver acceleration works,” November 2001, accessed: 2013-07-29. [Online]. Available: http://www.cfdreview.com
[15] R. Salomon, “Evolutionary algorithms and gradient search: similarities and differences,” IEEE Trans. Evol. Comput., vol. 2, no. 2, pp. 45–55, 1998.
[16] C. A. Coello Coello, G. B. Lamont, and D. A. van Veldhuizen, Evolutionary Algorithms for Solving Multi-Objective Problems, 2nd ed. Springer, 2007.
[17] M.-H. Lin, J.-F. Tsai, and C.-S. Yu, “A review of deterministic optimization methods in engineering and management,” Mathematical Problems in Engineering, vol. 2012, 2012.
[18] C. Othmer and T. Grahs, “CFD topology and shape optimization with adjoint methods,” VDI- Berichte, vol. 1967, no. 1, p. 61, 2006.
[19] O. Tonomura, M. Kano, and S. Hasebe, “Shape optimization of microchannels using CFD and adjoint method,” Computer Aided Chemical Engineering, vol. 28, pp. 37–42, 2010.
[20] M. B. Giles and N. A. Pierce, “An introduction to the adjoint approach to design,” Flow Turbul. Combust., vol. 65, no. 3–4, pp. 393–415, 2000.
[21] S. Eyi, K. D. Lee, S. E. Rogers, and D. Kwak, “High-lift design optimization using navier-stokes equations,” J. Aircr., vol. 33, no. 3, pp. 499–504, 1996.
[22] Y. S. Ong, P. Nair, A. Keane, and K. Wong, “Surrogate-assisted evolutionary optimization frameworks for high-fidelity engineering design problems,” in Knowledge Incorporation in Evolutionary Computation. Springer, 2005, pp. 307–331.
[23] J. E. Gentle, W. K. H¨ardle, and Y. Mori, Handbook of Computational Statistics: Concepts and Methods. Springer Heidelberg, 2004.
[24] J. C. Spall, Introduction to Stochastic Search and Optimization: Estimation, Simulation, and Control. Wiley, 2003.
[25] K. Deb, Multi-Objective Optimization using Evolutionary Algorithms. John Wiley and Sons, Ltd, 2001.
[26] G. Jones, Genetic and Evolutionary Algorithms. John Wiley & Sons, Ltd, 1998.
[27] S. Sharma, G. Singh, and K. V. Singh, “Selection based efficient algorithm for finding non dominated set in multi objective optimization,” Res.J.Recent.Sci., vol. 2, pp. 12–16, 2012. [28] L. Huang, G. Huang, R. LeBeau, and T. Hauser, “Optimization of blowing and suction control
on NACA0012 airfoil using genetic algorithm,” in 42nd AIAA Aerospace Sciences Meeting and Exhibit, no. 0423, USA, 2004.
[29] N. K. Beliganur, R. LeBeau, and T. Hauser, “Application of genetic algorithms and neural networks to unsteady flow control optimization,” in 18th AIAA Computational Fluid Dynamics Conference, USA, 2007.
[30] R. LeBeau, L. Huang, and T. Hauser, “Application of evolutionary algorithms to small jet arrays for flow control,” in 17th AIAA Computational Fluid Dynamics Conference, Canada, 2005. [31] S. Obayashi, S. Takahashi, and I. Fejtek, “Transonic wing design by inverse optimization using
moga,” in 6th Annual Conference of the CFD Society of Canada. Canada: Citeseer, 1998. [32] F. Zhang, S. Chen, and M. Khalid, “Optimizations of airfoil and wing using genetic algorithm,” in
23rd Congress of International Council of the Aeronautical Sciences, Canada, 2002, pp. 8–13. [33] J. J. Doherty, A. Domoney, R. M. Mitchell, and C. Burkett, “Aerodynamic design optimisation
applied to a formula one car,” in MIRA International Vehicle Aerodynamics Conference, UK, 2002.
[34] J. J. Doherty, D. P. Clifton, M. A. Gillan, and F. Ciampoli, “Operations based optimisation using simulation and CFD,” in 45th AIAA Aerospace Sciences Meeting and Exhibit, no. AIAA-2007- 762, USA, 2007.
[35] T. W. Simpson, J. D. Poplinski, P. N. Koch, and J. K. Allen, “Metamodels for computer-based engineering design: Survey and recommendations,” Eng. Comput., vol. 17, no. 2, pp. 129–150, 2001.
[36] A. Ribeiro, A. Awruch, and H. Gomes, “An airfoil optimization technique for wind turbines,” Appl. Math. Model., vol. 36, no. 10, pp. 4898–4907, 2012.
[37] R. M. Greenman and K. R. Roth, “Minimizing computational data requirements for multi-element airfoils using neural networks,” J. Aircr., vol. 36, no. 5, pp. 777–784, 1999.
[38] Y. Jin, “Surrogate-assisted evolutionary computation: Recent advances and future challenges,” Swarm and Evolutionary Computation, vol. 1, no. 2, pp. 61–70, 2011.
[39] D. Lim, Y. Jin, Y.-S. Ong, and B. Sendhoff, “Generalizing surrogate-assisted evolutionary computation,” IEEE Trans. Evol. Comput., vol. 14, no. 3, pp. 329–355, 2010.
[40] A. S´obester, A. I. J. Forrester, D. J. J. Toal, E. Tresidder, and S. Tucker, “Engineering design applications of surrogate-assisted optimization techniques,” Optim. Eng., vol. 15, no. 1, pp. 243– 265, 2014.
[41] Y. Jin, M. H¨usken, and B. Sendhoff, “Quality measures for approximate models in evolutionary computation,” in Proceedings of the Bird of a Feather Workshop, Genetic and Evolutionary Computation Conference (GECCO). USA: AAAI, 2003, pp. 170–173.
[42] M. H¨usken, Y. Jin, and B. Sendhoff, “Structure optimization of neural networks for evolutionary design optimization,” Soft Comput., vol. 9, no. 1, pp. 21–28, 2005.
[43] Y. Jin, M. Olhofer, and B. Sendhoff, “On evolutionary optimization with approximate fitness functions,” in Proceedings of the Genetic and Evolutionary Computation Conference (GECCO), 2000, pp. 786–793.
[44] B. Sendhoff, “Overview evolvable systems engineering,” Honda Research Institute Europe, Tech. Rep., 2012.
[45] S. Haykin, Neural Networks and Learning Machines (3rd Edition), 3rd ed. Prentice Hall, 2008. [46] X. Zhu and A. B. Goldberg, “Introduction to semi-supervised learning,” Synthesis lectures on
artificial intelligence and machine learning, vol. 3, no. 1, pp. 1–130, 2009.
[47] Z.-H. Zhou, Ensemble Methods: Foundations and Algorithms, ser. Chapman & Hall/CRC Data Mining and Knowledge Discovery Serie. Chapman & Hall, 2012.
[48] T. M. Mitchell, “Machine learning, mcgraw-hill higher education,” New York, 1997.
[49] R. Beale and T. Jackson, “Neural computing: an introduction,” Bristol: Hilger, IOP (Institute of Physics) Publication, 1990, vol. 1, 1990.
[50] A. Sharkey and N. Sharkey, “Diversity, selection, and ensembles of artificial neural nets,” Neural Networks and their Applications (NEURAP97), pp. 205–212, 1997.
[51] Y. Jin, “A comprehensive survey of fitness approximation in evolutionary computation,” Soft Comput., vol. 9, no. 1, pp. 3–12, 2005.
[52] C. Sun, J. Zeng, J. Pan, and Y. Jin, “Similarity-based evolution control for fitness estimation in particle swarm optimization,” in IEEE Symposium on Computational Intelligence in Dynamic and Uncertain Environments (CIDUE), Singapore, 2013, pp. 1–8.
[53] R. M. Greenman and K. R. Roth, “High-lift optimization design using neural networks on a multi- element airfoil,” Journal of Fluids Engineering, vol. 121, no. 2, pp. 434–440, 1999.
[54] A. I. Khuri and S. Mukhopadhyay, “Response surface methodology,” Wiley Interdisciplinary Reviews: Computational Statistics, vol. 2, no. 2, pp. 128–149, 2010.
[55] E. Acar and M. Rais-Rohani, “Ensemble of metamodels with optimized weight factors,” Struct. Multidiscip. Optim., vol. 37, no. 3, pp. 279–294, 2009.
[56] K.-S. Zhang, Z.-H. Han, W.-J. Li, and W.-P. Song, “Coupled aerodynamic/structural optimization of a subsonic transport wing using a surrogate model,” J. Aircr., vol. 45, no. 6, pp. 2167–2171, 2008.
[57] A. Forrester, N. Bressloff, and A. Keane, “Response surface model evolution,” in 16th AIAA Computational Fluid Dynamics Conference, vol. 2002–4089, USA, 2003.
[58] M. Y. M. Ahmed and N. Qin, “Comparison of response surface and kriging surrogates in aerodynamic design optimization of hypersonic spiked blunt bodies,” in 13th International Conference on Aerospace Sciences and Aviation Technology, Egypt, 2009.
[59] Y. V. Pehlivanoglu and B. Yagiz, “Aerodynamic design prediction using surrogate-based modeling in genetic algorithm architecture,” Aerosp. Sci. Technol., vol. 23, no. 1, pp. 479–491, 2012. [60] A. Forrester, A. S´obester, and A. Keane, “Multi-fidelity optimization via surrogate modelling,”
Proc. R. Soc. A-Math. Phys. Eng. Sci., vol. 463, no. 2088, pp. 3251–3269, 2007.
[61] Y. Kuya, K. Takeda, X. Zhang, and A. Forrester, “Multifidelity surrogate modeling of experimental and computational aerodynamic data sets,” AIAA J., vol. 49, no. 2, pp. 289–298, 2011.
[62] G. Cybenko, “Approximation by superpositions of a sigmoidal function,” Mathematics of Control, Signals and Systems, vol. 2, no. 4, pp. 303–314, 1989.
[63] R. J. Williams and D. Zipser, “A learning algorithm for continually running fully recurrent neural networks,” Neural Comput., vol. 1, no. 2, pp. 270–280, 1989.
[64] A. P. Engelbrecht, Computational Intelligence: An Introduction, 2nd ed. Wiley, 2007.
[65] R. D. Reed and R. J. Marks, Neural Smithing: Supervised Learning in Feedforward Artificial Neural Networks. Cambridge, MA, USA: MIT Press, 1998.
[66] D. K. Arrowsmith and C. M. Place, Dynamical Systems: Differential Equations, Maps, and Chaotic Behaviour, ser. Chapman and Hall Mathematics Series. Chapman & Hall, 1992. [67] A. Khalkhali, H. Safikhani, A. Nourbakhsh, and N. Nariman-Zadeh, “Modeling and multi-
objective optimization of 4-digit NACA airfoils using genetic algorithms,” in 2nd International Conference on Engineering Optimization, no. 1015, Portugal, 2010.
[68] R. Razaghi, N. Amanifard, and N. Narimanzadeh, “Modeling and multi-objective optimization of stall control on NACA0015 airfoil with a synthetic jet using GMDH type neural networks and genetic algorithms,” IJE Transactions A: Basics, vol. 22, no. 1, pp. 69–88, 2009.
[69] F. Lopez Pe˜na, V. D´ıaz Cas´as, A. Gosset, and R. J. Duro, “A surrogate method based on the enhancement of low fidelity computational fluid dynamics approximations by artificial neural networks,” Comput. Fluids, vol. 58, pp. 112–119, 2012.
[70] T. Rajkumar, C. Aragon, J. Bardina, and R. Britten, “Prediction of aerodynamic coefficients for wind tunnel data using a genetic algorithm optimized neural network,” NASA Ames Research Center, Tech. Rep. 20030107271, 2002.
[71] Y. Cao, Y. Jin, M. Kowalczykiewicz, and B. Sendhoff, “Prediction of convergence dynamics of design performance using differential recurrent neural networks,” in IEEE International Joint Conference on Neural Networks (IJCNN), 2008. (IEEE World Congress on Computational Intelligence)., 2008, pp. 528–533.
[72] L. K. Hansen and P. Salamon, “Neural network ensembles,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 12, no. 10, pp. 993–1001, 1990.
[73] I. A. Gheyas and L. S. Smith, “A novel neural network ensemble architecture for time series forecasting,” Neurocomputing, vol. 74, no. 18, pp. 3855–3864, 2011.
[74] D. Lim, Y.-S. Ong, Y. Jin, and B. Sendhoff, “A study on metamodeling techniques, ensembles, and multi-surrogates in evolutionary computation,” in Proceedings of the 9th annual conference on Genetic and evolutionary computation (GECCO). England: ACM, 2007, pp. 1288–1295. [75] A. Chandra and X. Yao, “DIVACE: Diverse and accurate ensemble learning algorithm,” in
Intelligent Data Engineering and Automated Learning (IDEAL). Springer Berlin Heidelberg, 2004, vol. 3177, pp. 619–625.
[76] X. Yao and M. M. Islam, “Evolving artificial neural network ensembles,” IEEE Comput. Intell. Mag., vol. 3, no. 1, pp. 31–42, 2008.
[77] Y. Tian, J. Zhang, and J. Morris, “Modeling and optimal control of a batch polymerization reactor using a hybrid stacked recurrent neural network model,” Ind. Eng. Chem. Res., vol. 40, no. 21, pp. 4525–4535, 2001.
[78] G. Brown, J. Wyatt, R. Harris, and X. Yao, “Diversity creation methods: a survey and categorisation,” Journal of Information Fusion, vol. 6, no. 1, pp. 1–28, 2005.