GP for Classification

Classification is an important problem domain, and one which forms the basis of empirical testing in this thesis. Furthermore, the classification tasks used in this thesis are all multiclass classification problems. There- fore in this section we briefly review related work on GP for classification, with a focus on multiclass classification.

There has been an enormous quantity of work in the area of GP for classification, and it is not the intent of this thesis to act as a comprehensive survey paper. To this end we limit ourselves to a representative sampling of important papers. Esperjo et al [24] provides an excellent survey of GP for classification for the interested reader.

GP can be applied to classification tasks in a variety of ways. The three major application categories Preprocessing [65], Model Extraction, and En- semble Classifiers. Most of the papers published related to GP and clas- sification focus on the application of GP to model extraction, that is, the induction of classifiers [24]. As this thesis focuses on Model Extraction via Discriminant Functions in the form of LGP programs, we will limit our re- view appropriately.

TGP for Classification

We begin by reviewing papers related to TGP for classification. While not the focus of this thesis, TGP is the conventional form of GP, used in the vast majority of literature. In addition, the use of LGP for classification is typically motivated by the drawbacks associated with using TGP for clas- sification. Hence it is important give a brief overview of relevant literature related to TGP for classification.

36 CHAPTER 2. LITERATURE SURVEY 52, 68, 86, 96, 98, 100]. TGP has proven to be a highly effective approach for solving binary classification problems. Unfortunately TGP is not well suited to solving multiclass classification problems [99]. TGP programs output a single real number, which is difficult to interpret effectively as a class label in the context of multiple classes. Hence TGP classifiers often perform poorly on multiclass classification problems.

Several attempts have been made to improve the performance of TGP on multiclass classification problems. The majority of these approaches focus on new mapping functions for converting the single program output into a class label.

Loveard [53] applies GP to several binary and multiclass medical data sets. The methods of binary decomposition, static range selection, dy- namic range selection, class enumeration and evidence accumulation are described. Dynamic range selection was found to be the method with the best mix of speed and accuracy.

Zhang and Smart [99, 81] introduce two mapping functions: centered dynamic range selection and slotted dynamic range selection. These two approaches outperform the basic static range selection on more difficult multiclass classification problems with many classes.

Smart and Zhang [80] introduce a probability based mapping function which uses a parameterized Gaussian to map tree output to a class label. This approach gives significantly improved performance on a number of multiclass classification problems, however performance suffers when the underlying output distribution is not a Gaussian.

LGP for Classification

In this subsubsection we review papers related to LGP for classification. While LGP is an effective technique for solving binary classification prob- lems [?] it has particularly come into its own as a method for solving mul- ticlass classification problems. With this in mind this thesis uses multi- class classification problems as the basis of empirical testing. Hence this

2.7. RELATED WORK 37 subsection will review LGP for classification with a focus on multiclass classification.

LGP is particularly well suited to solving multiclass classification prob- lems. It has been demonstrated that LGP has significantly superior perfor- mance to conventional LGP on many multiclass classification problems [26, 67]. Despite this, relatively little work has been done in the area of linear genetic programming for multiclass classification.

Olague et al [67] develop a LGP approach for image classification which simultaneously solves the region selection and feature extraction tasks. The method searches for optimal regions of interest, using texture infor- mation as its feature space and classification accuracy as the fitness func- tion. The paper shows that this LGP based approach gives superior per- formance to previous methods.

Fogelberg and Zhang [97] apply LGP to a number of multiclass image recognition problems with favorable results. The paper demonstrates that a LGP based approach significantly outperforms the basic TGP based ap- proach. It also provides heuristic guidelines for initially setting system parameters.

Downey and Zhang [20] introduce a new mutation operator called se-

lective mutation. Selective mutation identifies “bad”2 instructions within

the program and focuses mutation on these instructions. The paper shows that selective mutation significantly outperforms conventional mutation on several multiclass classification problems.

Downey, Zhang and Browne [21] introduce two crossover operators: Class Graph Crossover, and Selective Crossover. These operators are specifi- cally designed for multiclass classification problems. Class graph crossover examines program struture to determine which instructions are responsi- ble for each output. Selective crossover examines program performance on the training set, and attempts to select the most useful program code from each parent during crossover. The paper shows that these opera-

38 CHAPTER 2. LITERATURE SURVEY tors significantly outperform conventional crossover on several multiclass classification problems.

Chapter 3

Data Sets

This chapter describes the data sets, GP parameters, implementation, and hardware used throughout the experiments in this thesis.

3.1

Data Sets

The algorithms presented in this thesis are evaluated on three tasks from the important problem domain of object classification. Object classification problems are ideal for comparing and contrasting LGP algorithm perfor- mance. They occur naturally in a wide range of applications,

The three tasks chosen are Hand Written Digits, a problem in classify- ing hand written digits; Artificial Characters, a problem in classifying ar- tificial characters; and Yeast, a problem in predicting protein localization sites in yeast. These tasks are highly challenging due to a high number of attributes, a high number of classes, and noise in some sets. All three tasks are sourced from the UCI machine learning database [27].