Pareto front Solutions with Different Models

In order to focus on the overall patterns in the evolved Pareto fronts over all 50 runs, Figures 5.4 and 5.5 (in the previous section) ignore those Pareto front solutions that produce the same performance on the two objectives in a given MOGP run. However, even though two Pareto front solutions may have the same performances on the two objectives in a given MOGP run, it does not necessarily mean that these two solutions will have the same internal models or structures. Solutions with different internal models/structures will not produce the same set of output values when evaluated on all the input instances. The AUC of a given Pareto front solution can determine whether two or more solutions have different internal models, as the AUC calculation examines every output value of a solution to determine how well the solution separates these output values across different classification thresholds.

For example, Figure 5.6(a) shows the output values of two solutions,p1andp2,

when evaluated on eight input instances from two classes. The output values are denoted by+and−for the positive and negative class, respectively. Using zero as the boundary between the two classes, both solutions have the same number of correct predictions for each class, as shown in Figure 5.6(a). This corresponds to a true positive (TP) rate of 0.75 (as 3₄ positive class inputs are correctly classified), and a true negative (TN) rate of 0.5 (as 2₄ negative class inputs are correctly classified).

0 i + + + + − − − − + − − − + − + + + − p1 0 p2 region Target i i + + + + − − − − + − − − + − + + + − 0 p2 p1 region Target i 0 (a) Class threshold is 0 (b) Class threshold isi

Figure 5.6: Output values denoted by+and−for the positive and negative class, respectively, for two solutions (p1andp2). In (a),p1andp2have the same accuracy

on the two classes relative to zero as the class threshold; while in (b), p1 and p2

have different accuracy rates on the two classes relative to class thresholdi.

output values are different. Therefore, when the output values are evaluated relative to another class threshold, i in Figure 5.6(b), these solutions have a different number of correct predictions for each class. For p1, the TP rate is 1

and the TN rate is 0.25. Forp2, the TP rate is 0.75 and the TN rate is 0.25. These

corresponds to different ROC points for solutionp1 and p2 and thus, their AUC

will also be different.

This section compares the number of Pareto front solutions with different internal classification models (i.e. different AUC) for the two MOGP approaches (for a given run). Table 5.2 (left-hand column) shows the average number of distinct points in objective-space (on the test set) produced by the MOGP approaches over 50 runs. This corresponds to the average number of solutions on a given Pareto front which produce different performances on the two objectives (over 50 runs). These figures reflect the average size of an evolved Pareto front

not including those solutions that produce the same performances on the two objectives (in a given run).

The right-hand column in Table 5.2 shows the average sizes of thefullevolved Pareto fronts over 50 runs for the MOGP approaches. These correspond to the number of all Pareto front solutions evolved in a given run including those which produce thesameperformances on the two objectives, i.e., Pareto front solutions with different AUC values on the test set.

As expected, Table 5.2 shows that the number of solutions with different AUC values (right-hand column) is higher than than the number distinct points in objective-space (left-hand column) for both MOGP approaches in all tasks. This confirms that some Pareto front solutions have different internal

5.4. AUC ANALYSIS OF THE PARETO FRONT IN MOGP 131 Table 5.2: The average number of Pareto front solutions that produce distinct points in objective-space(test set), and the number of Pareto front solutions with different internal models (different AUC) over 50 runs for the MOGP approaches.

Task Unique Points inObjective-space Total Pareto Front Size

NSGAII SPEA2 NSGAII SPEA2

Ion 20.08 22.80 21.22 46.54 Spt 19.36 19.22 39.14 91.60 Ped 112.96 134.74 117.30 162.80 Yst1 63.08 58.20 86.74 105.64 Yst2 21.06 29.06 26.66 87.08 Bal 18.02 14.76 29.70 62.28

models/structures even though they produce the same performance on the two objectives for both MOGP approaches.

Interestingly, the number of solutions with different AUC values is always higher in SPEA2 than NSGAII in all tasks, even though both MOGP approaches use the same population size (of 500). This shows that SPEA2 contains more non- dominated solutions (on average) with different internal models/structures in the final generation than NSGAII in all tasks.

In fact, Table 5.2 shows that the average number of distinct points in objective- space (left-hand column) is only larger in NSGAII (than in SPEA2) in exactly three tasks (Spt, Yst1 and Bal). These correspond to the same three tasks where

the average hyperarea for NSGAII and SPEA2 is not statistically different to one another (Table 5.1), and where both systems show similar median attainment surfaces (Figure 5.3). Intuitively, it follows that the MOGP approach with more distinct points in objective-space will also produce a better hyperarea, as more of the objective-space is covered by these solutions. However, in these three tasks, NSGAII only performs as well as SPEA2 (in terms of their hyperarea and median attainment surfaces) but not better than SPEA2. This may be because NSGAII still contains fewer non-dominated solutions with different internal models/structures than SPEA2, as shown in Table 5.2 (right-hand column).

It should be mentioned that Pareto fronts sizes are noticeably larger in Ped (compared to the other tasks) for both MOGP approaches. This is expected due to the very large number of training examples in Ped (more than 12000 examples), where slight variations in the objective performances by the evolved solutions produce more non-dominated points in the objective-space.

(a) Ped (b) Yst1

Figure 5.7: AUC of all Pareto front solutions evolved over 50 MOGP runs for NSGAII (top) and SPEA2 (bottom) on two tasks (Ped and Yst1). Each vertical bar

represents a Pareto front solution (on the two objectives) and the heights of the vertical bars represent the AUC.