2.8 Conclusions
3.1.1 Hierarchical Agglomerative Methods
In general the steps of an agglomerative algorithm are the following:
1. Initially, the n objects are being split into n clusters and the symmetric proximity matrix DIST = dist(di, dj) is constructed.
2. We search DIST for the closest clusters. Let K and L be the clusters that are more close to each other, then min(DIST ) = dist(dK, dL).
3. The clusters K and L are being merged and the new cluster is named KL. Also the matrix DIST is updated by removing lines and columns that corresponds to K and L, and by adding a line and a column that displays the distances of the new cluster KL from the rest of the clusters.
4. Finally, repeat steps 2 and 3 n − 1 times in total. At the final state of the algorithm all objects belongs to one clusters. The clusters created during the algorithmic process are being recorded along with the level of similarity of each merging.
3.1. Hierarchical Clustering Algorithms 21
Defining proximity between clusters The hierarchical agglomerative methods differentiates by the way they compute the proximity between two clusters. While the calculation of the distance between two elements is clear enough, the calculation of the distance between clusters that contain more than one elements can be done in many ways. Cluster proximity is typically defined with a particular cluster type in mind (see Section2.5). For example many agglomerative hierarchical techniques, such as Single Link, Complete Link and Group Average, come from a graph based view of clusters. Single Link defines proximity between two clusters as the distance between the two closest points that are in different clusters. This yields contiguity-based clusters as shown in Section2.5. On the other hand, Complete Link consider cluster proximity as the distance between the two farthest points that are in different clusters. One more graph based approach is the Group Average, which defines cluster proximity to be the average of all pairwise proximi-ties of all points that belong to different cluster. In Figure 3.2 the above approaches are illustrated.
Figure 3.2: The three graph based proximities described. Single Link (left), Complete Link (middle) and Group Averege (right).
When each cluster is represented by a centroid (centroid method), the cluster proximity is defined as the proximity between cluster centroids.
The median method it was first proposed in order to alleviate some disadvantages of the centroid method. In the centroid method, if the sizes of the two groups to be merged are quite different, then the centroid of the new group will be very close to that of the larger group and may remain within that group. In the median method, the centroid of a new group is independent of the size of the groups that form the new group. On the other hand, the Ward’s method, also assumes that a cluster is represented by a centroid, but it measure the proximity between two clusters in terms of the increase in the error sum of squares (ESS) that results from merging the two clusters.
The proximity calculation methods described above satisfy a recursive equation for the distance dM (KL)between the cluster M and the cluster KL which was created by merging cluster K and L at a previous step of the procedure. That is the following equation:
distM (KL)= αkdistM K+ αldistM L+ βdistKL+ γ|distM K− distM L| (3.1) where distKL the distance between the cluster K and L and α, β, γ the parameters that define equation 3.1.1according to Table3.1.
22 Chapter 3. Clustering Algorithms
Single Link αK = αL= 12 β= 0 γ = −12
Complete Link αK = αL= 12 β= 0 γ = 12
Group Average αK = nnK
K+nL, αL= n nL
K+nL β= 0 γ = 0
Centroid clust. αK = nnK
K+nL, αL= n nL
K+nL β = −αKαL γ = 0
Median clust. αK = αL= 12 β = −14 γ = 0
Ward’s method αK= nnM+nK
M+nK+nL, αL= n nM+nL
M+nK+nL β = n −nM
M+nK+nL γ = 0 Table 3.1: Parameters for Equation 3.1.1.
Agglomerative methods
Geometric methods Graph methods
Ward’s method Centroid method Median method
Single link method
Complete link method Group average method
Figure 3.3: Commonly used hierarchical agglomerative methods.
3.1. Hierarchical Clustering Algorithms 23
3.1.1.1 BIRCH
Balanced Iterative Reducing and Clustering using Hierarchies (BIRCH) is a widely use agglomerative hierarchical algorithm proposed in [ZRL96] for clustering very large numerical data sets in Euclidean spaces. In the al-gorithmic procedure a Clustering Feature (CF) is used to summarise the information of each cluster. For a cluster C of an a-dimensional dataset, CF is defined as follows:
CF(C) = (|C|, S1, S2),
where |C| is the number of instances in C and S1,S2 are a-dimensional vectors defined as: initially builds a CF tree dynamically as the new data points are inserted.
The parameters of the CF are the branching factor B, the leaf factor L and the threshold T . The nodes that are not leafs contain at most B subnodes of the form [CFi, childi] while the leaf nodes contain a most L entries fo the form [CFi]. Finally, the diameter of each entry in a leaf node has to be less than T . The determination of outliers is based on the density of each entry in a leaf node, entries of low density are considered to be outliers. Then the outliers are remove from the tree and are being stored to disk in order to reduce its size. Later during the process the outliers are scanned in order to examine if they can enter the tree form again without causing it to grow in size. Finally, when the CF tree is built, an agglomerative hierarchical clustering algorithm is applied to the nodes and a centroid is obtained for each cluster. The new clusters are formed by reassigning each data point to its nearest centroid.
The BIRCH algorithm has the advantage to perform very well in cases where clusters have spherical shape and uniform size, but it is not recom-mended for use with clusters of different sizes or irregular shapes.
3.1.1.2 CURE
Clustering Using Representatives (CURE) [KHK99] is an agglomerative hi-erarchical clustering algorithm that can deal with large data sets that con-tain nonspherical clusters of different sizes. The basic characteristic of this algorithm is that each cluster is represented by a set of points that is well
24 Chapter 3. Clustering Algorithms
scattered in the cluster. CURE make use of random sampling and partition-ing in order to handle large databases. The basic steps of the algorithmic procedure are the following:
1. Draw a random sample: To handle large databases a random sample is drawn. Here in order to analytically derive values for sample sizes, such that the probability of missing clusters is low, the Chernoff bounds [MR95] are used.
2. Partition the sample: In cases where the input sizes become large, a partitioning scheme is needed to speed up CURE. The samplespace is being partitioned into p partitions, each of size sp, where s is the sample size.
3. Partially cluster the partitions: Each partition is clustered until the cluster number for each partition reduces topqs for some constant q ≥ 1.
4. Eliminate the outliers: The outliers do not merge with other points in the same rate as actual clusters due to the greater distances from other points and for this reason grow much slower in agglomerative hierarchical clustering.
5. Cluster the partial clusters: A second clustering pass is run on the sp partial clusters for all the partitions.
6. Label the data: Each of the remaining data points is assigned to the cluster containing the representative point closest to it.
The complexity of the cure algorithm is O(n2log n), but it can be reduced to O(n2) for the 2-dimensional case.