DE vs SBX on Aerodynamic Problems

From the academic test cases we have learned that optimal parameterizations differ considerably. As a consequence, there is no general near-optimal default parame- terization. However, this could be due to artificial structures of the academic test

cases. Therefore, the next experiment shall reveal whether our insights gained so far are transferable to real-world problems.

Research Question How does DE compare to SBX variation on examples of real-world problems? How do the results compare to the ones from the previous experiment?

Preexperimental planning As real-world problems from the aerodynamic field, the bi-objective NACA and the 3-objective RAE are chosen (cf. Section 3.5.2). The problems have been subject of previous studies, see Beume et al. (2007)∗, Emmerich et al. (2006), Naujoks et al. (2002). Due to the large calculation times for the computational fluid dynamics simulations, a restricted number of 1000 objective function evaluations is allowed and the SPO budget is slightly decreased to 300 algorithm runs (see Tab. 3.12).

Task See the previous experiment in Section 3.5.3.

Setup The airfoils are represented as Bezier points with 6 degrees of freedom for NACA, and 18 for RAE. For RAE, the initial population always includes the baseline design. The default configurations and regions of interest are identical to the first experiment (see Tab. 3.8). Table 3.12 shows the differences in the experimental setup compared to the experiment in Section 3.5.3. The reference point is set to (0.4,0.4)> for the NACA problem and (10,10,10)> for the RAE problem.

Tab. 3.12:Settings forexperiment on aerodynamic problemsthat differ from Tab. 3.9.

Problems NACA, RAE

SPO budget 300 algorithm runs

Stopping criterion 1000 problem evaluations

Initial experimental design Latin Hypercube (25 points, 4 repeats per point)

Results/Visualization Table 3.13 shows the found optimized configurations, which are also included in Fig. 3.16 as bold lines. Table 3.14 shows the performance re- sults.

Observations All optimized configurations are significant improvements over their default configurations. The difference between SBX* and DE* is not significant on NACA, but on RAE. The possible improvements by parameter tuning can be gleaned from Tab. 3.14: SPO is able to improve the NACA values by 2.7% using

Tab. 3.13:Parameter results on the aerodynamic problems

DE Configuration SBX Configuration

Problem µDE CR F µSBX ηc ηm pc pm

NACA 21 0.90 0.34 10 0.16 15.43 0.06 0.68

RAE 14 0.76 0.71 10 20.50 34.24 0.01 0.48

Tab. 3.14:Mean hypervolume and standard deviation on aerodynamic problems

Problem SBX SBX* DE DE*

NACA 0.1462_± 0.0012 0.1501_±0.0007 0.1467_±0.0009 0.1502 _±0.0007

RAE 993.663 _±0.005 993.844_± 0.022 993.672_±0.033 993.869 _±0.041

SBX* and 2.4% featuring DE*. However, the results on RAE cannot be improved accordingly, here the improvements are about 0.02%.

Interestingly, the same population size is identified for SBX variation on both test cases. For DE*, a roughly similar population size was identified for the RAE case as well, while the best value for the NACA case is about twice as big. Concerning the operators’ probabilities, SBX* variation focuses on mutation. The application probabilities for the recombination operator are very small, which means that ηc

cannot have much influence. Generally, it is remarkable that SBX* and DE* are completely opposed to the default configurations.

Discussion The improvement seems so low on RAE, because the initial popula- tion always contains the mentioned near-optimal baseline solution, which already dominates a hypervolume of 993.662. But in fact, the default SBX configuration fails to find any other feasible solution in 49 of the 50 runs. The default DE config- uration ‘only’ fails in 39 runs. DE* and SBX*, on the other hand, achieve success rates of 100% for this measure.

3.5.5 Conclusions

Responding the recent popularity of DE variation, we studied the performance of the SMS-EMOA with this variation compared to its usual operator combination SBX and PM. Thorough experimental analyses have been performed as a parameter tuning study using SPO, on academic test problems as well as on aerodynamic real- world problems.

The main results are: (1) The performance of the tuned operators improved sig- nificantly compared to the default parameterizations. (2) The performance of the two tuned variation operators is very similar. (3) The optimized parameter config- urations for the considered problems are very different.

So, our experimental analyses could not verify any advantage of the DE variation over the combination of SBX and PM, at least not within the SMS-EMOA. More general than that, our study puts the common practice of benchmarking and tuning studies into question.

The study shows exemplarily that parameter tuning has more potential of perfor- mance improvement than the choice of a supposedly better operator. So, tuning shall always be performed before a conclusive rating of methods; it shall become standard. For academic benchmarks, this means that comparing EMOA without considering parameterizations is unfair and may give misleading hints like in the CEC 2007 competition by Huang et al. (2007) regarding DE variation. As described by Bartz-Beielstein et al. (2010a, Ch. 2.6), a comparison based on equal or untuned parameterizations is inappropriate. Instead the tuning shall be part of the bench- marking. For contests, we recommend to dedicate a budget of function evaluations for parameter tuning and maybe specifying the tuning method. In publications presenting a new optimizer, the new method is typically parameterized with ex- pert knowledge but tested against standard optimizers using default parameters. It is desirable to quantify the effort put in finding good parameterizations and to study ranges of parameters and their sensitivity.

For practitioners our message is analogously: Instead of investing time on a com- prehensive study of methods, quickly choose one optimizer and invest the time to adapt it to your problem. Do not expect default parameterizations to perform well. When parameter tuning is performed, it is still common practice to do this with a chosen optimizer on a test problem which is efficiently evaluable instead of the ac- tual problem. Our study demonstrates that this procedure is not promising, since the optimized parameter configurations for the academic test suite differ a lot from those for the real-world problems. It cannot be expected that parameterization performing well on one problem do so on another.

Although parameter tuning is elaborate, this tool shall not be neglected due to the significant potential. When evaluations of the optimization problem are too time- consuming, an alternative is to perform the tuning on a surrogate model instead. Preuss et al. (2010) achieved successes with this approach and demonstrate that the EA using the gained tuned parameterization outperforms the EA using the default parameterization on the original problem.

Professional tools for parameter tuning are available with SPOT and REVAC (cf. Nannen and Eiben (2007)) but parameter tuning requires considerable resources. To reduce the time spent on expensive function evaluations, further investigations on the suitability of surrogate models instead of original problems for parameter tuning are desirable. Moreover, the interaction of parameters is interesting and dependencies and sensitivity analyses shall give valuable insights.