This section tries to quantify the run-by-run analysis from the previous section (e.g. Figure 6.4) to further investigate the differences between the MOGP approaches and the ensemble voting/selection strategies. By comparing the classification result of each MOGP approach on a run-by-run basis over the 50 runs, this section investigates which MOGP approach (Baseline, NCL or PFC) and ensemble combination strategy (PF-Wvote or off-EEL) produces better overall performances across all MOGP runs and tasks. These questions are difficult to answer using only the average ensemble performances reported in Tables 6.4 and 6.5 (in the previous section). For clarity, note that in the experimental setup, a single run of the three MOGP approaches (Baseline, NCL and PFC) all use the same random starting seed and initial population.
A run-by-run analysis of the MOGP approaches has a two-dimensional aspect, as both the majority and the minority class accuracies of the ensembles must be taken into account when determining if one approach is better than another (compared to a “single figure” measure such as the overall accuracy). The Pareto dominance relation is used to summarise a single classification result between any two MOGP approaches on a run-by-run basis basis. This allows us to determine if one approach is better than the other, in terms of a “win”, “lose” or “draw” outcome. For two MOGP approaches,gp1 andgp2, the three outcomes
for a particular run can be defined as follows.
• Win forgp1ifgp1dominatesgp2 (loss forgp2).
• Win forgp2ifgp1is dominated bygp2 (loss forgp1).
• Draw otherwise.
These three outcomes (i.e. win, lose or draw) represent a multinomial
distribution over N independent runs. This means that the proportion of wins for one approach (call thisp1), the proportion of wins for the other approach (call
thisp2), and the proportion of draws between them (call thisp3), sums to 1 over
N runs. In a multinomial distribution, a 95% confidence interval of the difference
in the proportion of wins between two approaches (p1−p2) can be calculated for a
particular task. This can be used to determine if one MOGP ensemblesignificantly dominates another over all runs. The 95% confidence interval of this difference between any two MOGP approaches is calculated using Eq. (6.4), wherevar(pi) is the variance ofpi for theithMOGP approach inN = 50runs.
6.6. COUNTING ENSEMBLE “WINS” 167
(p1−p2)±1.96
p
var(p1−p2) (6.4)
where
var(p1−p2) = var(p1) +var(p2)−
−(var(p1+p2)−var(p1)−var(p2))
= 2var(p1) + 2var(p2)−var(p1 +p2)
var(pi) = pi(1−pi) N var(p1+p2) = (p1+p2)(1−p1−p2) N
In the subsequent sections, the ensemble wins between the MOGP approaches are used to explore two main aspects of the ensemble behaviour. The first com- pares which of the three MOGP approaches (Baseline, NCL or PFC) statistically dominates each other on the tasks, and which achieves more overall ensemble wins over all runs and tasks. The second compares which ensemble combination strategy between PF-Wvote and off-EEL statistically dominates each other on the tasks, and which achieves more overall ensemble wins over all runs and tasks.
6.6.1
Wins for Diversity Measure in MOGP
To compare which of the three MOGP approaches (Baseline, NCL or PFC) statistically dominates each other on the tasks, Table 6.5 shows the pairs of ensemble wins between two MOGP approaches, where each is compared with every other (on a run-by-run basis) for 50 runs. Each win-pair in Table 6.5 corresponds to the three pairwise comparisons between the Baseline, NCL and PFC approaches using a particular ensemble combination strategy. Two ensemble combination/selection strategies are examined: fitness-weighted majority vote (PF-Wvote) and off-EEL.
For example, the first win-pair in Table 6.5 for Ion with PF-Wvote is “8,8”. This means that when the Baseline MOGP is compared to the NCL approach on a run-by-run basis over 50 runs, both the Baseline and NCL score 8 wins each (i.e. each approach dominates the other exactly 8 times). In the remaining 34 runs, these approaches are non-dominated with respect to each other (i.e. 34 ”draws”). Similarly, the next win-pair in Table 6.5 for Ion with PF-Wvote is “7,14”.
Table 6.5: ”Win” pairs between two MOGP approaches (on a run-by-run basis) over 50 runs for two ensemble combination strategies (PF-Wvote and off-EEL). Total wins (and draws) is the sum of wins (and draws) over all runs and tasks (50 runs×6 tasks). Bold results indicate a statistically significantly better ensemble performance (95% significance level).
Task PF-Wvote Strategy Off-EEL Ensemble Selection Baseline Baseline NCL Baseline Baseline NCL
vsNCL vsPFC vsPFC vsNCL vsPFC vsPFC Ion 8,8 7,14 8,19 11,6 6,22 3,20 Spt 3,5 1,5 1,3 10,7 3,6 1,9 Ped 11,3 1,25 0,19 15,7 0,20 0,26 Yst1 7,8 4,2 5,7 5,3 6,3 1,3 Yst2 7 / 5 5,2 3,12 5,1 8,7 0,4 Bal 10,7 11,14 10,9 14,8 12,16 4,21 Wins 46,36 29,62 27,69 60,32 35,74 9,83 Draws 218 209 204 208 191 208
This means that when the Baseline is compared to PFC on a run-by-run basis over 50 runs, PFC wins against (dominates) the Baseline in 14 runs, while the Baseline wins against PFC in 7 runs. These approaches are non-dominated in the remaining 29 runs (i.e. 29 ”draws”).
The last two rows in Table 6.5 reports the total number of wins for each pair, and the total number of draws (non-dominated performances), summed overall
runs and tasks. The total number of wins and draws in each column in Table 6.5 sums to 300 (50 runs×6 tasks).
The results of the 95% confidence intervals of the wins between two MOGP approaches are also shown in Table 6.5. The statistically significantly better ensemble performance, denoted by a higher number of wins, is highlighted in bold. It is important to note that as three separate confidence intervals are constructed for each of the three pairwise comparisons, the statistical relationship only applies to a specific pair.
PFC better than NCL over all tasks
According to Table 6.5, the total number of wins (over all tasks) when NCL is compared to PFC is higher in PFC for both ensemble combination strategies. For the PF-Wvote strategy, PFC has 69 total wins while NCL only has 27; for the off-EEL strategy, PFC has 83 total wins while NCL only has 9. This means that in both ensemble combination strategies, the PFC ensembles dominate the NCL
6.6. COUNTING ENSEMBLE “WINS” 169 ensembles more often than the opposite case over all runs and tasks. The very large difference in total wins between PFC and NCL for off-EEL is due to the PFC ensembles achieving statistically significantly better performances than NCL in nearly all tasks. As the PFC approach, particular with off-EEL, produces better ensemble results than NCL in these tasks, this suggests that the PFC ensembles may be more diverse that the NCL ensembles.
A similar conclusion can be drawn when the PFC ensembles are compared to the Baseline MOGP for the two ensemble combination strategies. PFC always scores more total wins (over all tasks) than the Baseline when these two approaches are compared against each for both strategies. For the PF-Wvote strategy, PFC has 62 total wins while the Baseline only has 29; for the off-EEL strategy, PFC has 74 total wins while the Baseline only has 35. As discussed in the previous section (and shown in Figure 6.4), the better PFC performances are due to better cooperation between members than the Baseline MOGP on these tasks.
In contrast, NCL scores fewer total wins (over all tasks) than the Baseline MOGP for the two ensemble combination strategies. For the PF-Wvote strategy, NCL has 36 total wins while the Baseline has 46; for the off-EEL strategy, NCL has 32 total wins while the Baseline has 60. In fact, Table 6.5 shows that for both PF-Wvote and off-EEL strategies, there is no statistically significant difference in wins between the Baseline and NCL in any of the six tasks. This shows that both NCL and Baseline ensembles achieve very similar performances on the tasks.
Further Discussions
The above results show that the PFC ensembles performed better than NCL on these tasks, particularly for the off-EEL selection algorithm. This may be due to two reasons. The first is the different ways NCL and PFC create “spread” (or diversity) in the population (see [126][125] for theoretical insights into how NCL creates spread in a population). The second is the different ways in which NCL and PFC are used in MOGP: NCL is calculated after the population is ranked on the objectives (using SPEA2), while PFC is calculated before Pareto ranking is done.
Developing an approach which incorporates NCL into the objective perfor- mance before Pareto ranking is done (similar to PFC) may improve ensemble performances for NCL. Likewise, new diversity measures (such as the root quartic NCL proposed in [126][125]) may also improve ensemble performances for NCL, due to different ways in which these measures create “spread” in
Table 6.6: ”Win” pairs between the two ensemble combination strategies (PF- Wvote and off-EEL) for the MOGP approaches (on a run-by-run basis) over 50 runs. Bold results indicate a statistically significantly better ensemble perfor- mance (95% significance level) over 50 runs.
Task PF-Wvote vs off-EEL Baseline NCL PFC Ion 4,9 4,15 1,25 Spt 2,2 1,1 10,3 Ped 0,3 1,3 0,5 Yst1 1,12 2,4 3,10 Yst2 0,0 1,4 1,6 Bal 3,5 6,3 3,8 Wins 10,31 15,30 18,57 Draws 259 255 225
a population. However, this is outside the scope of this work which focuses on evaluating the use of the traditional NCL and PFC measures in the fitness function in MOGP for diversity (and not adaptations of these measures such as [126][125]). This will be an interesting exercise for future work.
6.6.2
Wins for Ensemble Combination Strategies in MOGP
To compare which of the two ensemble combination strategies statistically dom- inates each other on the tasks, Table 6.6 shows the pairs of ensemble wins when PF-Wvote is compared to off-EEL (on a run-by-run basis) over 50 runs for the three MOGP approaches (Baseline, NCL and PFC approaches). The statistically significantly better ensemble strategy, denoted by a higher number of wins, is highlighted in bold. For example, the first win-pair in Table 6.6 for Ion is “4,9”. This means that when PF-Wvote is compared to off-EEL for the Baseline MOGP, PF-Wvote wins against (dominates) off-EEL in 4 runs, while off-EEL dominates PF-Wvote in 9 runs. In the remaining 37 runs, these strategies are non-dominated to each other.
The last two rows in Table 6.5 reports the total number of wins for each pair, and the total number of draws (non-dominated performances), summed over all 50 runs and for the six tasks (300 total runs).
Table 6.6 shows that the total number of draws (over all tasks) between these two ensemble combination strategies (for the three MOGP approaches) is higher than the total number of draws in all columns in Table 6.5 (from the previous section). This shows that the two ensemble combination strategies are
6.7. COMPARISON WITH SGP, NB AND SVM 171