Results of Hyper Volume Indicator

In order to further compare the results of the multi-objective algorithms, NSPSOFS, CMDPSOFS, NSGAII, SPEA2 and PAES, the hyper volume in- dicator [174] is used in the experiments. In each run, each method ob- tained two Pareto fronts, which are atraining Pareto frontaccording to the training classification performance and the number of features, and atest-

4.4. RESULTS AND DISCUSSIONS 125

ing Pareto frontaccording to the testing classification performance and the number of features. Therefore, for each method, we calculated two sets of hyper volume values based on the Pareto-fronts on the training process and the testing process, respectively. Therefore, for each method, 40 hy- per volume values on the training process and 40 hyper volume values on the testing process were calculated. As the calculation of hyper vol- ume needs the true Pareto front, which is not available in the datasets, we firstly combine the training (or testing) Pareto front of these five methods into a union, then identify the Pareto front in the union as the “true Pareto front” to calculate the hyper volume values. The hyper volume values of each Pareto front are normalised to hyper volume ratios, which is the division of the hyper volume value of a Pareto front and the hyper vol- ume value of the “true Pareto front”. In order to compare NSPSOFS and CMDPSOFS with the other three algorithms, NSGAII, SPEA2 and PAES, the Student’s T-test was performed on their hyper volume ratios, where the significance level is set as 0.05 (or confidence interval is 95%).

Results of Hyper Volume on Testing Process

Table 4.1 shows the results of the T-test between NSPSOFS, CMDPSOFS, NSGAII, SPEA2 and PAES on the hyper volume ratios in thetesting pro- cess, where “NS” and “CMD” represent NSPSOFS and CMDPSOFS. In Table 4.1, “+” (“-”) indicates that NSPSOFS or CMDPSOFS is significantly better (worse) than another corresponding algorithm. “=” means they are similar. On the WBCD and Lung datasets, the “?” means the hyper vol- ume ratio could not be obtained because the extracted “true Pareto front” only contains two points and its hyper volume value is zero.

Table 4.1 shows that compared with CMDPSOFS, NSGAII, SPEA2 and PAES, NSPSOFS achieved similar results in most cases, although NSP- SOFS achieved better results on the Australian dataset and worse results on the Hillvalley and Musk1 datasets. Table 4.1 also shows that CMDP- SOFS achieved similar results with other methods in most cases. On the

Table 4.1: T-test on Hyper Volume Ratios on Testing Accuracy

Dataset Wine Australian Zoo Vehicle German WBCD

NS CMD NS CMD NS CMD NS CMD NS CMD NS CMD NSPSOFS = - = = = ? CMDPSOFS = + = = = ? NSGAII = = + = = = = = = = ? ? SPEA2 = = + = = = = = = = ? ? PAES = = = = = = = = = = ? ?

Dataset Lung Ionosphere Hillvalley Musk1 Madelon Isolet5 NS CMD NS CMD NS CMD NS CMD NS CMD NS CMD NSPSOFS ? + + + = + CMDPSOFS ? - - - = - NSGAII ? ? - = - + - + = = + + SPEA2 ? ? = + - + - + = = + + PAES ? ? - = - + - + = = - +

datasets with a large number of features, such as Hillvalley, Musk1 and Isolet5, CMDPSOFS achieved significantly better results than NSPSOFS, NSGAII, SPEA2, and PAES.

Results of Hyper Volume on Training Process

Table 4.2 shows the results of the T-test between NSPSOFS, CMDPSOFS, NSGAII, SPEA2 and PAES on the hyper volume ratios in thetrainingpro- cess. It can be seen that NSPSOFS achieved slightly worse results than other methods in most cases, but NSPSOFS achieved better results than NSGAII and SPEA2 on the Isolet5 dataset, where the number of features is large. Table 4.2 also shows that on the datasets with a relatively small number of features, CMDPSOFS usually achieved similar results to NS- GAII, SPEA2 and PAES. On the datasets with large numbers of features, such as Hillvalley, Musk1, Madelon and Isolet5, CMDPSOFS achieved sig- nificantly better results than NSPSOFS, NSGAII and SPEA2. Although CMDPSOFS achieved slightly worse results than PAES on the training set, CMDPSOFS achieved similar or better results than PAES on the test set (shown in Table 4.1), which is considered due to the overfitting problem in PAES.

4.4. RESULTS AND DISCUSSIONS 127 Table 4.2: T-test on Hyper Volume Ratios on Training Accuracy

Dataset Wine Australian Zoo Vehicle German WBCD

NS CMD NS CMD NS CMD NS CMD NS CMD NS CMD NSPSOFS + + + + + + CMDPSOFS - - - - NSGAII - = - = - - - = - = SPEA2 - = - = - = - = - = - = PAES - = - = - - - =

Dataset Lung Ionosphere Hillvalley Musk1 Madelon Isolet5 NS CMD NS CMD NS CMD NS CMD NS CMD NS CMD NSPSOFS + + + + + + CMDPSOFS - - - - NSGAII - - - + - + = + + + SPEA2 - - - - = + - + = + + + PAES - - - -

From the results of the hyper volume ratios, it can be seen that the hy- per volume indicator does not seem a good measure for feature selection problems. Although the results are still consistent, the pattern shown by the hyper volume indicator is less clear than the previous direct compar- isons using figures. A possible reason is that the hyper volume indicator is mainly used for continuous multi-objective algorithms, not for discrete multi-objective algorithms. This is also the case for other multi-objective performance indicators/measures. Therefore, in Chapter 6 which presents the work of multi-objective filter feature selection, only direct comparisons will be used and the hyper volume indicator will not be used.