Timing, resources and opportunity









0.8 1.2 2.3 0.3 1.1 0.2 2.1 0.3 1.2 2.3 0.3 1.1 0.1 0.0 2.1 0.4 2.0 1.2 1.3 0.0 2.1 0.6 0.9 1.4









(3.1)

This set is comprised of three vectors (rows), described by eight features (columns). The bit string genotype describing this data set will subsequently be eight bits long, where the genotype 00110011 describes a feature subset including the third, fourth, seventh and eighth features; 11001010 the first, second, fifth and seventh features; and 11110000 the first four features.

The APIC method searches for the lowest dimensional feature subset, where classifiers trained using the feature subset achieve performance results that equal or exceed those of classifiers trained using the full feature set. In this method, a low dimensional feature subset is encouraged by applying a penalty to each genotype’s fitness, proportional to its dimensionality. This ensures that genotypes with lower dimensionality are more favourable than others, promoting the reduction of unnecessary features within the data.

The initial genotype population describing feature subsets is generated with bits toggled between “1” and “0” randomly. Using these feature subsets, the supplied data set can be partitioned, or divided, into a distinct number of groups. Each of these groups describes a unique pattern (type of object) in the data. A clustering process is used to group (cluster) the data, annotating each datum with its associated cluster identifier. More information on this process is provided in the following section.

Data Set

(Feature Subset) Data Clustering Genetic Algorithm

Hyper-Parameter Tuning

Evaluate

Clustered Data Set

Stop Condition

Figure 3.2: A GA-controlled clustering of data extracted using a feature subset.

The APIC method is capable of grouping feature vectors of a particular type (pattern) using unsupervised learning algorithms, illustrated in Figure 3.1 (2). Each group, orcluster, is automatically annotated with its cluster id, producing training sets for the production of supervised or semi-supervised classifiers. The method allows the number of groups to be set explicitly, or to be inferred automatically, through a discovery process. Where the number of patterns, or targets, are not specified, the method uses algorithms such as Density-Based Spatial Clustering of Applications with Noise (DBSCAN) (Ester et al., 1996) to discover these automatically. Where the number of targets,k, are knowna priori, algorithms such as k-means are used to group datum into k clusters.

Using these algorithms, the input data described by the current feature subset is clustered, grouping related datum. The hyper-parameters controlling this clustering process are unique to each problem.

Accommodating for this, the APIC method uses a GA to search for optimised values for these parameters, ensuring tighter (more optimal) cluster formation. Figure 3.2 provides an outline of the APIC pattern discovery process, where the hyper-parameters controlling the clustering process are optimised automatically using a GA.

The parameters guiding the clustering process of each problem are encoded as bit string genotypes. Where k-means partitive clustering is used, the location of each centroid is encoded in the bit string. At this point, each value of the feature subset is assumed to have been normalised to a value between 0 and 1. The position of each centroid is represented as four bits within the genotype, which reduces genotype complexity and

0 1 1

x

x

0.8

0.1

0.1 0.6

x=0.1, y=0.1: 0001 0001 x=0.6, y=0.8: 1001 1100

(a) Initial Generation

0 1

1

x x

0.8

0.2

0.3 0.5

x=0.5, y=0.2: 0111 0011 x=0.3, y=0.8: 1000 1100

(b) Final Generation

Figure 3.3: A GA encodes the location of two centroids on a two dimensional lattice for a k-means clustering problem. By tuning the locations of each centroid, the GA is able to optimise the search process for centroid positioning. In (a), two centroids (denoted by x) are positioned randomly on the lattice. In (b), the centroids have positioned themselves in the centre of the two distinct patterns present in the data (the diamonds and the stars)

allows for up to 16 distinct positions in the feature space. This granularity is sufficient for the k-means algorithm, as the centroid positions in the feature space are fine-tuned by the k-means algorithm. Figure 3.3 illustrates the encoding of two centroids in an example two dimensional problem. Figure 3.3a illustrates an example of a genotype from the initial population, while Figure 3.3b demonstrates the evolution of centroids in the final generation.

For problems where the number of clusters is unknown, algorithms such as DBSCAN are used to infer these automatically. The DBSCAN algorithm requires values for two hyper-parameters, namely the neighbourhood radius size Epsilon () and the minimum number of points (minP T S) per cluster.

These two parameters are each encoded as eight bit integers, concatenated to form a complete genotype.

For all clustering algorithms, the initial population of genotypes of hyper-parameters is generated randomly. Each genotype is evaluated by a fitness function, which executes an instance of the clustering algorithm configured with the decoded parameters. Each algorithm is permitted to run until the stopping criteria is met. For the k-means algorithm, a stop

A B C D E

Total Fitness ( )

0 Tf Tf

Result

∋

f

Figure 3.4: An example of roulette wheel selection probabilities for a generation consisting of five genotypes.

will occur when no changes are made to centroid locations during a single pass. Algorithms such as DBSCAN, for example, stop once all datum have been passed and evaluated by the algorithm. After clustering is complete, the resulting clustered data set is evaluated using the Silhouette Cluster Analysis (Rousseeuw, 1987) method. The average silhouette for a clustered data set is given by Equation 3.2.

s(i) = b(i)−a(i)

max{a(i), b(i)} (3.2)

Where a(i) is the average measure of dissimilarity between datum i and another datum within the same cluster, b(i) is the lowest average measure of dissimilarity between datum i and any other cluster. The s(i) for the clustered data set is therefore a measure of how successful the clustering process is. These values are represented as values between -1 and 1, with a value toward the latter signifying more appropriate (tight) clustering. Each genotype is assigned a fitness value equal to the silhouette score achieved post-clustering. Once all genotypes have been scored, a succeeding population is generated for the next generation. Evolution of the successive population is ensured by application of EA operators and processes after each generation. First, elitism (Eiben and Smith, 2003) is applied, so that the highest scoring genotypes are transferred directly to the next generation’s population. Next, pairs of “parent” genotypes are selected from the current population using roulette wheel selection (Al Jadaan et al., 2008). This process ensures a high degree of probability for selecting fitter

1 0 0 1 1 0 1

1 1 0 0 1 0 1

Parent 1 Parent 2

Random Crossover Point

0

1

1 0 0 1 1 0 1 1

1 1 0 0 1 0 1 0 Crossover

Random Bit string Mutation

Random Bit string Mutation

1 0 0 1 1 1 1

1 1 0 1 0 1 0

1

1

Child Genotypes

Figure 3.5: Application of EA operators on a pair of parent genotypes

parent genotypes. Where F_i is the fitness of genotype i in the current population, the probability of selection is given by Equation 3.3.

P_i = F_i PN

j=1F_j (3.3)

WhereN is the number of genotypes within the current population. The probability of selecting a genotype based on fitness is illustrated in Figure 3.4

The pairs of parents are recombined using single-point crossover to produce two child genotypes (Eiben and Smith, 2003). For each of these genotypes, bit string mutation is applied to flip one bit, resulting in a slightly altered resultant genotype. Both of these new genotypes are subsequently added to the next generation’s population. These selection and recombination processes, illustrated in Figures 3.4 and 3.5, respectively, continue until the maximum number of allowed genotypes have been added to the successive generation’s population.

Once a sufficient silhouette score has been attained for a clustered data set, the GA is said to have converged and the best scoring solution is forwarded to the classifier production process. A fixed number of generations is specified as a stop condition where, if breached, the best solution at that point is used. The following section describes the third and final process of APIC, the development of customised classifiers.