Computational Complexity - Projection methods for clustering and semi-supervised classification

In this subsection we analyse the computational complexity of MDP2. At each iteration the algorithm projects the data sample onto v(θ) which in-

volvesO(nd)operations. To compute the projection index,φ(θ), we need to

minimise the penalised density integral, f(v(θ),b). This can be achieved by

first evaluating f(v(θ),b)on a grid ofmpoints, to bracket the location of the

minimiser, and then applying bisection to compute the minimiser(s) within the desired accuracy. The main computational cost of this procedure is due to the first step which involvesmevaluations of a kernel density estimator with n kernels. Using the improved fast Gauss transform (Morariu et al., 2008) this can be performed in O(m+n) operations, instead ofO(mn). Bisection requiresO(−log₂ε)iterations to locate the minimiser with accuracyε.

If the minimiser of the penalised density integralb?=argmin_b_∈_Af(v(θ),b),

is unique the projection index is continuously differentiable at θ. To obtain

the derivative of the projection index it is convenient to define the projection function,P(v) = (x1·v,. . .,xn·v)>. An application of the chain rule yields,

d_θφ=D_θf(v(θ),b?) =DPf(v(θ),b?)DvPDθv (2.26)

where the derivative of the projections of the data sample with respect to v is equal to the data matrix, DvP= (x1,. . .,xn)>; andDθv is the derivative of

v with respect to the projection angle, which yields ad×(d−1)matrix. The computation of the derivative therefore requiresO(d(n+d))operations.

The original GS algorithm requiresO(d)gradient evaluations at each iteration which is costly. Curtis and Que (2013) have developed an adaptive gradient sampling algorithm that requiresO(1)gradient evaluations in each iteration. More recently,Lewis and Overton(2013) have strongly advocated that for the minimisation of nonsmooth, nonconvex, locally Lipschitz functions, a simple BFGS method using inexact line searches is much more effi- cient in practice than gradient sampling, although no convergence guarantees have been established for this method. BFGS requires a single gradient evaluation at each iteration and a matrix vector operation to update the Hessian matrix approximation. In our experiments we use this algorithm.

5 Experimental Results

In this section we assess the empirical performance of the minimum density hyperplane approach for clustering and semi-supervised classification. We compare performance with existing state-of-the-art methods for both problems on the following 14 benchmark datasets: Banknote authentication (banknote), Breast Cancer Winsconsin original (br. cancer), Forest type mapping (forest), Ionosphere, Optical recognition of handwritten digits (optidigits),

Table 2.1:Details of Benchmark Data Sets n d c banknotea 1372 4 2 br. cancera 699 9 2 foresta 523 27 4 ionospherea 351 33 2 optidigitsa 5618 64 10 pendigitsa 10992 16 10 seedsa 210 7 3 smartphonea 10929 561 12 image seg.a 2309 18 7 satellitea ₆₄₃₅ ₃₆ ₆ syntha 600 60 6 votinga 435 16 2 winea 178 13 3 yeastb 698 72 5

a. UCI machine learning repositoryhttps://archive.ics.uci.edu/ml/datasets.html

b. Stanford Yeast Cell Cycle Analysis Projecthttp://genome-www.stanford.edu/cellcycle/

Pen-based recognition of hand-written digits (pendigits), Seeds, Smartphone- Based Recognition of Human Activities and Postural Transitions (smartphone), Statlog Image Segmentation (image seg.), Statlog Landsat Satellite (satellite), Synthetic control chart time series (synth control), Congressional voting records (voting), Wine, and Yeast cell cycle analysis (yeast). Details of these data sets, in terms of their sizen, dimensionalityd and number of clustersc, can be seen in Table2.1.

5.1 Clustering

Since MDHs yield a bi-partition of a dataset rather than a complete clustering, we propose two measures to assess the quality of a binary partition of a dataset containing an arbitrary number of clusters. Both take values in[0,1] with larger values indicating a better partition. These measures are motivated by the fact that a good binary partition should (a) avoid dividing clusters between elements of the partition, and (b) be able to discriminate at least one cluster from the rest of the data. To capture this we modify the cluster labels of the data by assigning each cluster to the element of the binary partition which contains the majority of its members. In the case of a tie the cluster is assigned to the smaller of the two partitions. We thus merge the true clusters into two aggregate clusters,C1andC2.

The first measure we use is the binary V-measure which is simply the V-measure (Rosenberg and Hirschberg, 2007) computed on C1,C2 with re-

spect to the binary partition, which we denote Π1,Π2. The V-measure is the

harmonic mean of homogeneity and completeness. For a dataset containing clusters C1,. . .,Cc, partitioned as Π1,. . .,Πk, homogeneity is defined as

the conditional entropy of the cluster distribution within each partition, Πi.

Completeness is symmetric to homogeneity and measures the conditional entropy of each partition within each cluster,Cj. An important characteristic

of the V-measure for evaluating binary partitions is that if the distribution of clusters within each partition is equal to the overall cluster distribution in the data set then the V-measure is equal to zero (Rosenberg and Hirschberg,

2007). This means that if an algorithm fails to distinguish the majority of any of the clusters from the remainder of the data, the binary V-measure returns zero performance. Other evaluation metrics for clustering, such as purity and the Rand index, can assign a high value to such partitions.

To define the second performance measure we first determine the number of correctly and incorrectly classified samples. The error of a binary partition, E(Π1,Π2), given in Eq. (2.27), is defined as the number of elements

of each aggregate cluster which are not in the same partition as the majority of their original clusters. In contrast, the success of a partition, S(Π1,Π2),

Eq. (2.28), measures the number of samples which are in the same partition as the majority of their original clusters. The Success Ratio, SR(Π1,Π2),

Eq. (2.29), captures the extent to which the majority of at least one cluster is well-distinguished from the rest of the data.

E(Π1,Π2) =min{|Π1∩C1|+|Π2∩C2|,|Π1∩C2|+|Π2∩C1|}, (2.27)

S(Π1,Π2) =min{max{|Π1∩C1|,|Π1∩C2|},max{|Π2∩C1|,|Π2∩C2|}},

(2.28) SR(Π1,Π2) = S(Π1,Π2

)

S(Π1,Π2) +E(Π1,Π2). (2.29)

Similarly to the binary V-measure defined above, Success Ratio takes the value zero if an algorithm fails to distinguish the majority of any cluster from the remainder of the data.

Parameter Settings for MDP2

The two most important settings for the performance of the proposed approach are the initial projection direction, and the choice ofα, which controls

the width of the interval F(v) within which the optimal hyperplane falls. Despite the ability of the MDP2 formulation to mitigate the effect of local minima in the projected density ˆpv(b), the problem remains non-convex and

We have found that this effect is amplified in general when either or both the number of dimensions, and the number of high density clusters in the dataset is large. To better handle the effect of local optima, we use multiple initialisations and select the MDH that maximises therelative depth criterion, defined in Eq. (2.30). The relative depth of an MDH,H(v,b), is defined as the smaller of the relative differences in the density on the MDH and its two adjacent modes in the projected density, ˆpv(·),

RelativeDepth(v,b) =min ˆ pv(ml)−pˆv(b) ˆ pv(b) , ˆ pv(mr)−pˆv(b) ˆ pv(b) , (2.30) where ml and mr are the two adjacent modes in ˆpv(·). If an MDH does not

separate the modes of the projected density, ˆpv(·), then its relative depth is

set to zero, signalling a failure of MDP2to identify a meaningful bi-partition. The relative depth is appealing because it captures the fact that a high quality separating hyperplane should have a low density integral, and separate well the modes of the projected density ˆpv(·). Note also that the relative depth

is equivalent to the inverse of a measure used to define cluster overlap in the context of Gaussian mixtures (Aitnouri et al.,2000). In all the reported experiments we initialise MDP2to the first and second principal component and select the MDH with the largest relative depth. For the data sets listed above it was never the case that both initialisations led to MDHs with zero relative depth.

The choice ofαdetermines the trade-off between a balanced bi-partition

and the ability to discover lower density hyperplanes. The difficulties asso- ciated with choosing this parameter are illustrated in Figure 2.2. In each sub-figure the horizontal axis is the candidate projection vector, v, while the right vertical axis is the direction of maximum variability orthogonal to v. Points correspond to projections of the data sample onto this two- dimensional space, while colour indicates cluster membership. The solid line depicts the projected density on v, ˆpv(·), while the dotted line depicts the

penalised function, fUL(v,·). The scale of both functions is depicted in the

left vertical axis. The solid vertical line indicates the MDH alongv. Setting

α to a large value can cause MDP2 to focus on hyperplanes that have low

density because they partition only a small subset of the dataset as shown in Figure 2.2(a). In contrast smaller values ofαmay cause the algorithm to

disregard valid lower density hyperplane separators (see Figure 2.2(b)), or for the separating hyperplane to not be a local minimiser of the projected density (see Figure2.2(c)).

Rather than selecting a single value forα we recommend solving MDP2

repeatedly for an increasing sequence of values in the range {αmin,αmax},

where each implementation beyond the first is initialised using the solution to the previous. Setting αmin close to zero forces MDP2 to seek low density

−5 0 5 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7

(a) MDH separating few observations −5 0 5 0.0 0.1 0.2 0.3 0.4

(b) Lower density hyperplane beyond feasible re- gion −4 −2 0 2 4 6 8 0.0 0.1 0.2 0.3 0.4 (c) MDH not a minimiser of the projected density

Fig. 2.2:Impact of choice ofαon minimum density hyperplane.

tions which display strong multimodal structure, yet prevents convergence to hyperplanes that have low density because they partition a few observations, as in the case shown in Figure2.2(a). Increasingαprogressively fine-tunes the

location of the MDH. To avoid sensitivity to the value ofαmax(set to 0.9) the

output of the algorithm is the last hyperplane that corresponds to a minimiser of ˆpv(·). Figure2.3illustrates this approach using the optical recognition of

handwritten digits dataset from the UCI machine learning repository (Lich- man,2013). Figure2.3(a)depicts the projected density on the initial projection direction, which in this case is the second principal component. As shown, the density is unimodal and the clusters are not well separated along this vector. Although not shown, if a large value of α is used from the outset,

MDP2 will identify a vector for which ˆpv(·) is unimodal and skewed. Fig-

ure2.3(b)shows that after five iterations withα=10−2 MDP2has identified

a projection vector such that ˆpv(·) is bimodal. In subsequent iterations the

two modes become more clearly separated, Figure2.3(c), while increasingα

enables MDP2to locate an MDH that corresponds to a minimiser of ˆpv(·), as

illustrated in Figure2.3(d). Performance Evaluation

We compare the performance of MDP2 for clustering with the following methods:

1. k-means++ (Arthur and Vassilvitskii,2007), a version ofk-means that is guaranteed to beO(logk)–competitive to the optimal k-means clustering.

2. The adaptive linear discriminant analysis guidedk-means (LDA-km) (Ding and Li,2007). LDA-km attempts to discover the most discriminative linear subspace for clustering by iteratively usingk-means, to assign labels

−10 −5 0 5

0.0

0.1

0.2

0.3

(a) Initial projection

−6 −4 −2 0 2 4 6 0.0 0.1 0.2 0.3 0.4

(b) Projection after 5 iterations withα= 10−2 −5 0 5 0.0 0.1 0.2 0.3 0.4

(d) Final projection withα=αmax

Fig. 2.3:Evolution of the minimum density hyperplane through consecutive iterations.

to observations, and LDA to identify the most discriminative subspace. 3. The principal direction divisive partitioning (PDDP) (Boley,1998), and the density-enhanced PDDP (dePDDP) (Tasoulis et al., 2010). Both methods project the data onto the first principal component. PDDP splits at the mean of the projections, while dePDDP splits at the lowest local minimum of the one-dimensional density estimator.

4. The iterative support vector regression algorithm for MMC (Zhang et al.,

2009) using the inner product and Gaussian kernel, iSVR-L and iSVR-G respectively. Both are initialised with the output of 2-means++.

5. Normalised cut spectral clustering (SCn) (Ng et al., 2002) using the Gaussian affinity function, and the automatic bandwidth selection method of Zelnik-Manor and Perona(2004). This choice of kernel and band-

width produced substantially better performance than alternative choices considered. For datasets that are too large for the eigen decompo- sition of the Gram matrix to be feasible we employed the Nyström method (Fowlkes et al.,2004).

We also considered the density-based clustering algorithm PdfCluster (Menardi and Azzalini, 2014), but this algorithm could not be executed on the larger datasets and so its performance is not reported herein. With the exception of SCn and iSVR-G, the methods considered bi-partition the data through a hyperplane in the original feature space. For the 2-means and LDA-2m algorithm the hyperplane separator bisects the line segment joining the two centroids. iSVR-L directly seeks the maximum margin hyperplane in the original space, while iSVR-G seeks the maximum margin hyperplane in the feature space defined by the Gaussian kernel. PDDP and dePDDP use a hyperplane whose normal vector is the first principal component. PDDP uses a fixed split point while dePDDP uses the hyperplane with minimum density along the fixed projection direction.

Table2.2reports the performance of the considered methods with respect to the success ratio (SR) and the binary V-measure (V-m) on the fourteen datasets. In addition Figures2.4(a)and2.4(b)provide summaries of the overall performance on all datasets using boxplots of the raw performance measures as well as the associatedregret. The regret of an algorithm on a given dataset is defined as the difference between the best performance attained on this dataset and the performance of this algorithm. By comparing against the best performing clustering algorithm regret accommodates for differences in difficulty between clustering problems, while also making use of the mag- nitude of performance differences between algorithms. The distribution of performance with respect to both SR and V-m is negatively skewed for most methods, and as a result the median is higher than the mean (indicated with a red dot).

It is clear from Table2.2that no single method is consistently superior to all others, although MDP2 _{achieves the highest or tied highest performance}

on seven datasets (more than any other method). More importantly MDP2 is among the best performing methods in almost all cases. This fact is better captured by the regret distributions in Figure 2.4(b). Here we see that the average, median, and maximum regret of MDP2 is substantially lower than any of the competing methods. In addition MDP2achieves the highest mean and median performance with respect to both SR and V-m, while also hav- ing much lower variability in performance when compared with most other methods.

Pairwise comparisons between MDP2and other methods reveal some less obvious facts. SCn achieves higher performance than MDP2in more examples (six) than any other competing method, however it is much less consis-

MDP2 iSVR-L iSVR-G SCn LDA-2m 2-means++ PDDP dePDDP Dataset SR V-m SR V-m SR V-m SR V-m SR V-m SR V-m SR V-m SR V-m banknote 0.79 0.55 0.00 0.00 0.35 0.00 0.46 0.10 0.00 0.01 0.37 0.01 0.40 0.03 0.00 0.03 br. cancer 0.91 0.79 0.73 0.56 0.73 0.56 0.00 0.13 0.87 0.71 0.87 0.72 0.91 0.78 0.90 0.77 forest 0.78 0.67 0.90 0.72 0.91 0.74 0.56 0.41 0.76 0.63 0.72 0.58 0.64 0.36 0.00 0.00 image seg. 0.89 0.72 0.82 0.59 0.88 0.71 0.92 0.87 0.78 0.58 0.78 0.71 0.87 0.67 1.00 1.00 ionosphere 0.48 0.13 0.47 0.13 0.47 0.13 0.55 0.22 0.47 0.12 0.47 0.12 0.47 0.12 0.42 0.09 optidigits 0.93 0.85 0.63 0.29 0.82 0.60 0.00 0.00 0.81 0.62 0.92 0.82 0.68 0.30 0.00 0.00 pendigits 0.74 0.39 0.79 0.55 0.88 0.68 0.80 0.68 0.79 0.55 0.78 0.57 0.79 0.54 0.61 0.42 satellite 0.89 0.75 0.73 0.40 0.73 0.40 0.92 0.86 0.73 0.40 0.87 0.81 0.71 0.37 0.00 0.00 seeds 0.88 0.73 0.71 0.53 0.71 0.53 0.89 0.76 0.96 0.90 0.86 0.70 0.75 0.59 0.73 0.60 smartphone 0.99 0.97 0.99 0.95 0.99 0.96 0.99 0.94 0.99 0.97 0.99 0.94 0.99 0.95 0 .00 0.00 synth 0.98 0.94 0.94 0.83 0.94 0.83 1.00 1.00 0.88 0.76 1.00 1.00 0.69 0.51 1.00 1.00 voting 0.70 0.43 0.46 0.09 0.00 0.00 0.00 0.05 0.69 0.41 0.00 0.00 0.70 0.40 0.68 0.38 wine 0.77 0.61 0.70 0.52 0.69 0.50 0.67 0.48 0.66 0.48 0.68 0.49 0.65 0.46 0.68 0.49 yeast 0.92 0.76 0.89 0.68 0.91 0.72 0.84 0.61 0.86 0.63 0.91 0.73 0.87 0.65 0.00 0.00 Average Improvement 0.13 0.18 0.12 0.14 0.22 0.16 0.10 0.11 0.10 0.08 0.11 0.18 0.40 0.32

Table 2.2: Performance on the task of binary partitioning. (Ties in best performance were re-

solved by considering more decimal places)

tent in its performance, obtaining very poor performance on five of the data sets. The iSVR maximum margin clustering approach is arguably the closest competitor to MDP2. iSVR-L and iSVR-G achieve the second and third highest average performance with respect to V-m and SR respectively. The PDDP algorithm is the second best performing method on average with respect to SR, but performs poorly with respect to V-m. The density enhanced variant, dePDDP, performs on average much worse than MDP2. This approach is similarly motivated by obtaining hyperplanes with low density integral, and its low average performance indicates the usefulness of searching for high quality projections as opposed to always using the first principal component. Finally, neither of thek-means variants appears to be competitive with MDP2 in general.

5.2 Semi-Supervised Classification

In this section we evaluate MDHs for semi-supervised classification. We compare MDHs against three state-of-the-art semi-supervised classification methods: Laplacian Regularised Support Vector Machines (LapSVM) (Belkin et al.,

2006), Simple Semi-Supervised Learning (SSSL) (Ji et al.,2012), and Corre- lated Nyström Views (XNV) (McWilliams et al., 2013). For all methods the inner product kernel was used to render the resulting classifiers linear, and

● ● ● ● ● ● ● ● ●●

Success Ratio V−measure

0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 MDP2 iSVR iSVR−G SCn LD A−2m 2−means++ PDDP

dePDDP MDP2 iSVRiSVR−G SCn LD A−2m 2−means++ PDDP dePDDP

(a) Raw Performance Measure

● ● ● ● ● ● ● ● ● ● ● ● ●

Success Ratio V−measure

0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 MDP2 iSVR iSVR−G SCn LD A−2m 2−means++ PDDP

dePDDP MDP2 iSVRiSVR−G SCn LD A−2m 2−means++ PDDP dePDDP (b) Regret

Fig. 2.4:Performance and Regret Distributions for all Methods Considered

thereby comparable to the minimum density hyperplane approach. As the MDH is asymptotically equivalent to a linear S3VM we also considered the continuous formulation for the estimation of a S3_{VM proposed by}_Chapelle and Zien (2005). These results are omitted as this method was not competitive on any of the considered datasets.

Parameter Settings for MDP2

The existence of a few labelled examples enables an informed initialisation of MDP2. We consider the first and second principal components as well as the weight vector of a linear SVM trained on the labelled examples only, and

● ● ● ● ● ● ● 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45 510 20 40 60 80 100

Number of Labelled Examples

Classification Error (a) voting ● ● ● ● ● ● ● 0.05 0.10 0.15 0.20 0.25 510 20 40 60 80 100

Number of Labelled Examples

Classification Error (b) banknote ● ● ● ● ● ● ● 0.05 0.10 0.15 0.20 0.25

In document Projection methods for clustering and semi-supervised classification (Page 58-68)