3.3 Genetic Algorithm for Finding Cluster Hierarchies
3.3.1 Using Genetic Algorithm for Finding Cluster Hierarchies
Each chromosome specifies one solution to a defined problem. For GACH, a chromosome is the encoding of a hierarchical cluster structure (HCS) that was previously defined in ITCH (Section 3.2.2) and has to address the three following features:
• Storage of a set of clusters C1,· · · , Cn.
• Representation of the hierarchical relationship between clusters forming a tree T of clusters.
• Encoding of the cluster representatives, i.e. the parameters of the un- derlying PDF. For GACH we represent each cluster by a Gaussian PDF. Note that our model can be extended to a variety of other PDFs, e.g. uniform or Laplacian.
With this requirements a chromosomal representation of a HCS is defined as follows:
Definition 3 (Chromosomal HCS)
(1) A chromosomal HCS (HCSChrom) is a dynamic list storing a set of cluster objects.
(2) Each cluster C holds references to its parent cluster and to its child nodes. Besides that, the level lC of each cluster defines the height of the descendant subtree in the HCS
(3) The parameters of the underlying Gaussian PDF of cluster C, the mean value µC and σC, are modeled as additional parameters of the cluster C. (4) Each cluster C is associated with a weight WC, where
Pk−1
i=0 WCi = 1.
The underlying PDF of a clusterCis a multivariate Gaussian in ad-dimensional data space which is defined by the parameters µC and σC (whereµC and σC
72 3. Hierarchical Clustering
are vectors from a d-dimensional space) by the following formula:
N(µC, σC, x) = Y 1≤i≤d 1 q 2πσ2 C,i ·e− (xi−µC,i)2 2σ2 C,i
GACH assigns each pointxdirectly to that clusterC ∈HCSChromthe prob- ability density of which is maximal at the position ofx:
C(x) = arg max
Ci∈HCSChrom
{WCi ·N(µCi, σCi, x)}.
The parameters µC and σC of each cluster C are determined based on the hierarchical relationship to all subclusters and calculated analogously to ITCH as described in Section 3.2.4.
Initialization of GACH. Basically the initial set of a population consists of a randomly generated set of individuals. This strategy is also processed by GACH, where in a first step a random number of clusters ˜k is selected for each structureHCSChrom. Then a simple k-means algorithm divides the data set into ˜k clusters that act as the leafs of the initial hierarchy. Finally, these clusters are combined by one additional root cluster. Hence, the initialization process results in a 2-level hierarchy that consists of ˜k+ 1 nodes. Each clus- terCiis described by random parameters and is associated a weightWCi =
1 k.
Reproduction. In order to generate the next population of cluster hierar- chies GACH uses several genetic operators that are particularly defined for the hierarchical clustering problem: mutations (delete, add, demote and
promote) and crossover.
The delete operator deletes a specific cluster C (except the root) with a
deletion ratepdel from the HCS. This results in structure HCS0 that does not contain the cluster C any more. The proceeding of delete is illustrated in Figure 3.8(a). Here, the cluster C is marked in dark blue color. By deleting
3.3 Genetic Algorithm for Finding Cluster Hierarchies 73
delete( ) Level 2
Level 1
Level 0
(a) deleteone node of the hierarchy.
add( )
Level 2
Level 1
Level 0
(b) addone node to the hierarchy.
demote( )
Level 2
Level 1
Level 0
(c) demote two nodes to a lower level.
Level 3
Level 2
Level 1
Level 0
promote( )
(d) promoteone node to a higher level.
Figure 3.8: Summarization of the mutation operators used for GACH.
C the level of each direct and indirect subcluster of C (marked in red) is decreased by 1. The former parent node of C, the root node in our example, becomes the parent node of all direct subclusters of C.
The operator add adds direct subclusters to an arbitrary cluster C of the hierarchy with an add rate padd (normallypdel =padd). The number of addi- tional subclusters is bounded by an upper limit value maxnew. Figure 3.8(b) illustrates an example for the application of theaddoperator to a HCS where the cluster Cadd marked in dark blue color is added as a subcluster of the red cluster C. Since the PDFs of subcluster Cadd should fit into the PDF of
C and therefor to get an valid hierarchical relationship betweenC and Cadd, we calculate random parameters forCadd based onµC and σC. In particular,
we add a random value r to both parameters, where r is a vector from a
d-dimensional space: µCadd =µC+r σCadd =σC+r.
The motivation behind thedemote operator is the following. Assume a data set consisting of three clustersC1,C2 andC3, whereC1 holds a large number of objects, clusters C2 and C3 are smaller ones but they are locally close to
74 3. Hierarchical Clustering crossover C0 C6 C5 C4 C3 C2 C1 C0 C1 C2 C3 C4 C5 C0 C1 C3 C2 C4 C5 C6 C0 C5 C1 C2 C3 C4
(a) Hierarchical representation.
C0 μ0σ0 w0 C1 μ1σ1 w1 C2 μ2σ2 w2 C3 μ3σ3 w3 C4 μ4σ4 w4 C6 μ6σ6 w6 C5 μ5σ5 w5 C0 μ0σ0 w0 C1 μ1σ1 w1 C2 μ2σ2 w2 C3 μ3σ3 w3 C4 μ4σ4 w4 C5 μ5σ5 w5 crossover (b) Chromosomal representation.
Figure 3.9: Thecrossoveroperator for two selected hierarchies. The subtree
T1 of the red hierarchy is exchanged with the subtree T2 of the blue hierar- chy, visualized by a hierarchical (3.9(a)) and a chromosomal representation (3.9(b)).
each other. An intuitive hierarchical representation would be a HCS with one root node andC1, C2 and C3 as direct subclusters (cf. Figure 3.8(c)) which provides only a very coarse view of the data set. But, if we combine the two smaller clusters (marked in dark blue) and demote them with a demote rate
pdem to a lower level with a common parent cluster (marked in dark red), we are able to get a more detailed look on our data. The parameters of the inserted cluster Cin are obtained by the average of the parameters of the demoted clusters. Note that demoting only one cluster corresponds to the
addoperator. Hence, we apply demote on at least two clusters.
The promote operator lifts a cluster C from a lower level to the level right
above with a promotion rate ppro, if and only if C is at least two levels underneath the root cluster. Consequently all subclusters of C are lifted accordingly. In Figure 3.8(d) the dark blue cluster is promoted from level 3 to level 2. Hence, also the red subcluster is lifted to the next higher level. The parent of the parent node of the dark blue cluster (here the root node)
3.3 Genetic Algorithm for Finding Cluster Hierarchies 75
becomes the parent node of C in the resulting hierarchy HCS0, together with the correct rearrangement of all subclusters.
The operator crossover exchanges information among two different struc- tures. In general the information of two different chromosomes is combined in order to obtain a new individual with superior quality. GACH performs a crossover between two selected hierarchiesHCS1 andHCS2 with a crossover rate pco as follows:
1. Remove a selected subtree T1 entirely from HCS1.
2. Remove a selected subtree T2 entirely from HCS2.
3. Select a random node in HCS1 and insert T2.
4. Select a random node in HCS2 and insert T1.
Figure 3.9(a) illustrates this procedure exemplarily for two selected hierar- chies. The subtrees T1 and T2 are removed from the red and the blue HCS respectively. T1 is then inserted into the blue HCS as subtree of the dark blue node. AnalogouslyT2 is inserted as subtree of the dark red cluster in the red HCS. Figure 3.9(b) describes the same procedure w.r.t. a chromosomal representation of both hierarchies. For simplicity, only the pointers to the parent cluster are displayed.
Fitness Function. Following the Darwin‘s principle “Survival of the fittest” naturally only individuals with highest fitness can survive and those that are weaker become extinct. A GA adopts this aspect of evolution by the use of a fitness function. GACH uses the hM DL criterion formalized in Section 3.2.1 which evaluates the fitness of a chromosomal HCS by relating the clustering problem to that of data compression by Huffman Coding:
76 3. Hierarchical Clustering
hM DLHCS =
X
C∈HCS
cost(C)−nWClog2(WC)−log2(
X
xvparent of C
Wx)
The coding cost for each cluster C ∈ HCS is determined separately and
summed up to the overall coding cost of the completeHCS. Points that are directly assigned to the cluster C together with the parameters µC and σC of the underlying Gausssian PDF are coded bycost(C). The point to cluster assignment is coded by the so-called ID cost of each data point x ∈ C and is given by −nWClog2(WC) where WC is the weight of cluster C and n the number of points. The binary logarithm is used to represent the code length in bits. Clusters with higher weight are coded by a short code pattern whereas longer code patterns are assigned for smaller clusters with lower weight. The ID costs for the parameters are formalized by −log2(P
xvparent of C Wx) whereas constant ID costs are defined for the parameters of the root node.
The better the statistical model (the HCS) fits to the data the higher the compression rate thus the lower the coding costs are. Using this cod- ing scheme as fitness function ensures the selection of that chromosome
HCSChrom that fits best to the data.
Selection. The selection function chooses the best individuals out of a set of given individuals to form the offspring population according to their fitness. For GACH we use the well-known weighted roulette wheel strategy [117]. Imagine that eachHCSChromrepresents a number on a roulette wheel, where the amount of numbers refers to the size of the population. In addition we assign a weight to each number on the roulette wheel, depending on the fitness of the underlying chromosome. That means the better the fitness of a chromosome the higher its weight on the roulette wheel will be, i.e. the higher the chance to get selected for the offspring population. Note that there is the chance that one chromosome is selected multiple times. GACH forms
3.3 Genetic Algorithm for Finding Cluster Hierarchies 77
a new population that has as much individuals as the former population.
3.3.2
Algorithm GACH
The algorithm GACH is based on the combination of a genetic algorithm, information theory and model-based clustering as described in the previous Section. An initial population is built as described in Section 3.3.1. This population is evaluated according to the fitness function hM DLHCS which means that GACH determines the coding cost for each cluster hierarchy of the population. The lower the coding costs the better the HCS fits to the data. In order to optimize the point to cluster assignment of each HCS and to provide an additional model of the data, we apply the same hierarchical E- and M-steps formalized in ITCH (cf. Section 3.2.4) on each cluster structure. The formalization of GACH is presented in Algorithm 1. The population resulting from the initialization undergoes several mutation and crossover operations within popmax number of generations in an iterative way. In each iteration the next population is selected according to the weighted roulette wheel strategy and undergoes several reproduction procedures as described in the previous Section. Each operation (mutation or crossover) is processed with a certain probability which is extensively evaluated in Section 3.3.3. After optimizing the point to cluster assignment using E- and M-step as de- scribed in ITCH (cf. Section 3.2.4), GACH determines the fitness of each
HCSChrom in the population by calculating thehM DLHCS value. The algo- rithm terminates if a specified maximum number of new populations popmax is reached. The experiments show that the HCS can be optimized even with small generation sizes.
3.3.3
Experiments
Now we demonstrate that the genetic parameters (mutation rate, crossover rate and population size) do not affect the effectiveness of GACH in a major
78 3. Hierarchical Clustering
Algorithm 1GACH 1: countpop ←0
2: initialize population(countpop) 3: evaluatepopulation(countpop) 4: while (countpop ≤popmax)do 5: countpop ←countpop+ 1
6: select population(countpop) from population(countpop−1) 7: reproduce population(countpop)
8: evaluate population(countpop) 9: end while
way. Nevertheless, we provide a suitable parametrization that enables the user to receive good results independent of the used data set. Based on this, we compare the performance of GACH to several representatives of various clustering methods on synthetic and real world data. We selected the hierarchical clustering method Single Link [153], the more outlier-robust hierarchical clustering algorithm OPTICS [4], with optimal parameters w.r.t. accuracy. Furthermore, we chose RIC [18], an outlier-robust and information- theoretic clusterer, and finally ITCH (cf. Section 3.2). As ITCH strongly depends on its initialization, we used the best out of 10 runs in this case. In order to facilitate interpretation of the clustering result, we added color bars below the plots of Single Link and OPTICS, where the colors refer to the original class labels of the points in the data set. Furthermore, we chose the measures described in Section 2.2 to provide a quantitative comparison of the clustering results.
Evaluation of Genetic Parameters
We applied GACH on two different data sets to evaluate the mutation and crossover rates and the impact of the population size on the quality of the results w.r.t. the fitness function, introduced in Section 3.3.1. One data set consists of 1360 2-dimensional data points that form a true hierarchy of six clusters. The second data set covers 850 2-dimensional data points
3.3 Genetic Algorithm for Finding Cluster Hierarchies 79
that are grouped in two flat clusters. For each experiment, we present the mean hM DLvalue and the corresponding standard deviation over ten runs. GACH turned out to be very robust and determines very good clustering results (P rec > 90%,Rec >90%) indepent of the parametrizations.
Different Mutation Rates. We evaluated different mutation rates ranging from 1% to 5% on two different population sizes and a fixed crossover rate of 15%. As a mutation within a HCS is performed by one of the four operations
delete, add, demote or promote the mutation rate is the sum of pdel, padd,
pdem and ppro (cf. Section 3.3.1). As demote and promote turned out to be essential for the quality of the clustering results pdem and ppro are typically parametrized by a multiple of pdel or padd. This is due to the fact that the optimal number of clusters which is influenced by pdel andpaddis determined very fast by the fitness function, but pdem and ppro have an impact on the hierarchical structure of the clusters that has to be adjusted during the run of GACH. Figures 3.10(a) and 3.10(d) demonstrate that the mutation rate has no outstanding effect on the clustering result, neither on a hierarchi- cal nor on a flat data set. Higher mutation rates result in higher runtimes (3388 ms for mutation rate = 0.05 vs. 1641 ms for mutation rate = 0.01 on hierarchical data set, population size = 5). However, a higher mutation rate provides more flexibility. Hence, we achieved slightly better results with a mutation rate of 0.05 (hM DL= 10520) compared to a mutation rate of 0.01 (hM DL= 10542).
Different Crossover Rates. We compared the clustering result for different crossover rates pcoranging from 0.05 to 0.25 in combination with a mutation rate of 0.03 on two different population sizes. Figures 3.10(b) and 3.10(e) show that the performance of GACH is almost stable w.r.t. the different parameterizations of pco. Especially on the flat data set a higher value of
80 3. Hierarchical Clustering
population size: 5 chromosomes l i i 20 h
10700
u
e population size: 20 chromosomes
10650 10700 val u 10550 10600 D L 10500 10550 h M D 10500 0.01 0.02 0.03 0.04 0.05 ean h m e mutation rate (a) H: mutation rate
population size: 5 chromosomes l i i 20 h
10700
u
e population size: 20 chromosomes
10650 10700 val u 10550 10600 D L 10500 10550 h M D 10500 0.05 0.10 0.15 0.20 0.25 ean h m e crossover rate (b) H: crossover rate 10500 10550 10600 10650 10700 5 10 15 20 25 mean hMDL value population size (c) H: population size 7540 7560 7580 7600 D L value
population size: 5 chromosomes population size: 20 chromosomes
7500 7520 0.01 0.02 0.03 0.04 0.05 mean hM D mutation rate (d) F: mutation rate
population size: 5 chromosomes l i i 20 h
7600
u
e population size: 20 chromosomes
7560 7580 7600 val u 7540 7560 D L 7500 7520 h M D 7500 0.05 0.10 0.15 0.20 0.25 ean h m e crossover rate (e) F: crossover rate
7540 7560 7580 7600 D L value 7500 7520 5 10 15 20 25 mean hM D population size (f) F: population size
Figure 3.10: Mean fitness of resulting clusterings over ten runs on
(H)ierarchical and (F)lat data sets w.r.t. the genetic parameters mutation rate, crossover rate and population size.
3.3 Genetic Algorithm for Finding Cluster Hierarchies 81
hM DL value in almost every run, even for relatively small population sizes. Higher pco values enable GACH to search the data space more effectively as the crossover between two strong individuals produces an even fitter individ- ual. Therefore, we need less generations to find good clustering results, e.g. the result of GACH on the hierarchical data set using five structures was determined after 75 generations (1993 ms per generation) with pco = 0.05, and after 61 generations (2553 ms per generation) with an crossover rate of
pco = 0.25.
Different Population Sizes. We tested the impact of the population size on the quality of the clustering result. We used populations that cover 5, 10, 15, 20 and 25 hierarchical cluster structures in combination with a mu- tation rate of 3% and a crossover rate of 15%. Figures 3.10(c) and 3.10(f)
show again the mean hM DL value over ten runs for each population size
on two different data sets. Both plots demonstrate that a higher popula- tion size tends to produce better results, which can be explained by the fact that a higher population size provides more variation opportunities whereby a global optimum can be reached easier. However, a large number of chro- mosomes cause a considerable amount of runtime. One generation using 5 chromosomes took 2462 ms on average, the computation of a generation on 25 chromosomes took 9229 ms.
Hence we use a population size consisting of ten cluster structures in