We introduce relevant research questions in SSC: the use of soft partitions as regularisation for multi-class classification; the impact of unlabelled data in ensemble design; and en- semble techniques that enable SSC for large-scale datasets. We describe the contributions of this Thesis as follows.
1. Algorithms for multi-class semi-supervised classification.
Most existing SSC methods in literature are binary classifiers, therefore such al- gorithms depend on suboptimal decomposition techniques, such as one-vs-all and
one-vs-one, to perform multi-class classification. And they are prone to issues with imbalanced classes and different output scales of binary classifiers [Valizadegan et al.,
2008]. Thus, in this Thesis, we focus on developing cluster-based classifiers that can perform multi-class classification effectively and efficiently.
2. Semi-supervised classification with cluster regularisation.
In order to design an ensemble technique to solve the issues mentioned in Section
1.5, we design a new multi-class semi-supervised single classifier that overcomes the limitations of existing methods highlighted in Section 1.5.1.
As mentioned before, most cluster-based methods attempt to find the largest margin between high-density regions (clusters). When overlapping high-density regions are present, with sparse labelled instances on their borders, these classifiers may not produce good predictions, although these inherent clusters might be easily identified. In this Thesis, we propose a cluster-based multi-class algorithm, ClusterReg (Cluster- based Regularisation). Unlike other cluster-based algorithms, it does not depend on gaps between potential clusters, but captures the partition information, as posterior
probabilities, from a clustering algorithm in order to improve the decision boundary. By employing soft partitions in our method, the generalisation becomes more robust to the position of the few labels in the clusters. Unlike other cluster-based methods, our algorithm can easily establish a decision boundary between clusters without being misguided by neither the overlapping classes nor the few labels, when the dataset possesses a cluster structure for classes. ClusterReg achieves such robustness by incorporating soft clustering into a new regularisation technique.
ClusterReg regards the structure arising from the clustering algorithm as a soft partition. That is, each instance is assigned a probability of belonging to a given cluster, unlike hard partition where clusters are strictly disjoint. By using soft partitions (also known as soft clustering), we can address uncertain instances (likely lying on low density region, that is, on the border of clusters) differently from the more confident ones (likely lying on denser regions of clusters). Soft clustering helps the algorithm to address uncertain instances (with low probabilities for all clusters) as instances lying in gaps, therefore helping the classifier to generate the decision boundary in low-density regions.
One contribution of this work is to introduce a classifier that employs any clustering algorithm into SSC to regularise its decision boundary. The proposed algorithm (i) is robust in the presence of fewer labelled points, and is robust to the position of labelled data in clusters by considering the strength of clustering algorithms in a natural way and (ii) is able to improve the performance of a given classifier when the classes or clusters overlap, compared to other cluster-based algorithms.
ClusterReg can use any clustering algorithm with a proper processing of its output. It can employ any classifier that is able to minimise the proposed loss function. Therefore, ClusterReg can be seen as a framework for SSC methods.
3. A fully semi-supervised ensemble for multi-class classification.
In Section 1.5.2, we raised the question of which level of an ensemble should use the available unlabelled instances. Such unlabelled data can be considered at the ensemble level, the base classifier level, or both. To our knowledge, such issue has not been addressed until now, as most algorithms use unlabelled data only at the ensemble level and employ supervised base classifiers. In this Thesis, we present a study on the usefulness of employing unlabelled data at both ensemble and base learner levels. We compare such an approach to the use of unlabelled instances at the ensemble level only. We propose the Cluster-based Boosting algorithm (CBoost) for multi-class classification. Such a method extends ClusterReg and, unlike most semi-supervised ensembles in the literature, is composed of semi-supervised base classifiers.
Unlike other ensemble classifiers where unlabelled data is presented to the base classifiers as pseudo-labelled instances, both ensemble algorithm and base classi- fiers optimise the loss function introduced in Chapter 3, which uses both clustering algorithm and unlabelled data in a regularisation mechanism.
CBoost is able to learn from the clustering neighbourhood structure of pseudo- labels assigned by the ensemble, which leads to better generalisation when compared to learning the exact pseudo-labels individually. CBoost can overcome incorrect pseudo-label assignments used in the training of a new base classifier. CBoost is robust to the position of labelled data within a cluster and is able to handle the potential presence of overlapping classes. Experiments in Chapter 4confirmed that the proposed method is significantly superior to state-of-the-art ensemble methods and can improve the generalisation of single classifiers.
As discussed in Section 1.5.3, due to the large number of available unlabelled data, a semi-supervised training set can often have tens of thousands of instances. There- fore, the learning algorithms must be able to handle such large-scale datasets. Re- cently, various ensemble algorithms have been introduced with improved gener- alisation performance when compared to single classifiers. However, the existing ensemble algorithms are not able to handle typical large scale datasets.
In order to perform classification in large datasets, we propose the Efficient Cluster- based Boosting (ECB) algorithm. ECB uses a regularisation technique, based on posterior cluster probabilities, to avoid generating a decision boundary in high- density regions. In order to reduce the computational complexity of ECB, (i) base learners are trained with a subset of the unlabelled data along with all available la- belled instances at each iteration; (ii) we also employ an approximation technique to increase the efficiency of time and memory in the computation of nearest neighbours; (iii) and use an efficient clustering algorithm. We provide a theoretical discussion on the reasons why ECB might be able to achieve good performance with small amounts of sampled data and a relatively small number of base learners. Our ex- periments confirmed that ECB scales to large datasets whilst delivering comparable generalisation to state-of-the-art methods.