The goal of this chapter was to develop a new PSO based feature selec- tion approach to investigate and improve the performance of PSO for fea- ture selection, which is expected to select a smaller number of features and achieve better classification performance than using all features. To achieve this goal, we investigated the influence of the fitness function, the initialisation strategy and thepbestandgbestupdating mechanism in PSO for feature selection. The new algorithms were then developed to success- fully improve the performance of PSO for feature selection, and to outper- form the classification performance achieved by using all features and two traditional feature selection algorithms.
This chapter shows that the fitness function can significantly influ- ence the performance of PSO for feature selection. A well-designed fit- ness function can reduce the number of features and/or improve the clas- sification performance. The newly developed two-stage fitness function aims to maximise the classification performance in the first stage and con- siders both the number of features and the classification performance in the second stage. It can further remove redundancy in the feature sub- sets evolved by PSO using the fitness fitness considering the classification performance only. As a result, the number of features is further reduced without significantly decreasing or even improving the classification per- formance.
3.5. CHAPTER SUMMARY 103 This chapter shows that the initialisation strategy in PSO for feature selection can not be ignored. When the number of features is not included in the fitness function, the initialisation of solutions (starting points) can influence the size of final feature subsets, which then influences the clas- sification performance. The results show that the small initialisation usu- ally selected a small number of features, but the classification performance was not as good as the large initialisation, which usually selected a large number of features. The mixed initialisation successfully avoided the lim- itations to select a small number of features, but maintained the classi- fication performance achieved by the large initialisation and traditional random initialisation.
This chapter also shows that the pbestandgbest updating mechanism can significantly influence the performance of PSO for feature selection. By considering the number of features when updating pbest and gbest, which are the leaders of particles, the number of features was significantly reduced and the classification performance was maintained or even in- creased.
By combining the best initialisation strategy andpbestandgbestupdat- ing mechanism, the PSO based algorithm simultaneously improved the classification performance and reduced the number of features, especially on datasets with a large number of features. By reducing the number of features, the computational time can also be reduced. The reason is that the computational time was mainly spent on the classification process in the fitness evaluation procedure and a smaller number of features cost less time for each classification process.
The proposed PSO based algorithms in this chapter focus on the single fittest solution found during the evolutionary search process. Although both the number of features and the classification performance are consid- ered, it is unknown whether the obtained solution still have redundancy. Meanwhile, in real-world applications, it is needed to provide users the trade-off between the two objectives. To achieve this, it is thought to use
evolutionary multi-objective algorithms to evolve a set of Pareto front so- lutions (feature subsets), which allows decision-makers to choose a pre- ferred solution according to their own requirements. Therefore, the next chapter will develop a multi-objective feature selection approach based on multi-objective PSO, where the two objectives are to maximise the classi- fication performance and to minimise the number of features.
Chapter 4
Wrapper Based Multi-Objective
Feature Selection
4.1
Introduction
Feature selection in nature is a multi-objective problem, which is to max- imise the classification accuracy (minimise the classification error rate) and minimise the number of features. These two objectives are usually conflicting to each other and the optimal decision needs to be made in the presence of a trade-off between them. Treating feature selection as a multi-objective problem can obtain a set of non-dominated feature subsets to meet different requirements in real-world applications. Although PSO, multi-objective optimisation, and feature selection have been individually investigated frequently, the use of PSO for multi-objective feature selection has not been investigated.
4.1.1
Chapter Goals
The overall goal of this chapter is to develop a PSO based multi-objective feature selection approach to classification with the expectation of achiev- ing a Pareto front of non-dominated solutions, which hopefully include
a smaller number of features and achieve a lower classification error rate than using all available features. In order to achieve this goal, we investi- gate two Pareto front feature selection algorithms based on multi-objective PSO, which are NSPSOFS using the idea of non-dominated sorting and
CMDPSOFSusing the ideas of crowding, mutation andε-dominance. The two feature selection algorithms will be examined and compared with the best single objective algorithm (PSOIniPG) developed in the previous chapter and three well-known evolutionary multi-objective algorithms (De- tails can be seen in Section 4.3). Specifically, we will investigate:
• whether NSPSOFS can evolve a Pareto front of non-dominated solu- tions, which include a smaller number of features and achieve better classification performance than using all features, and outperform PSOIniPG,
• whether CMDPSOFS can evolve a Pareto front of non-dominated feature subsets and outperform PSOIniPG, and
• whether NSPSOFS and CMDPSOFS can achieve better performance than three well-known multi-objective algorithms, NSGAII, SPEA2 and PAES.
4.1.2
Chapter Organisation
The remainder of this chapter is organised as follows. The second sec- tion describes the new multi-objective feature selection algorithms. The third section describes the design of the experiments. The results and dis- cussions are presented in the fourth section. The fifth section provides a summary of this chapter.