Multiclass Weak Learner

Schapire and Singer [142] present a method for generating a weak hypothesis based on partitioning the domain ~xf _{∈R that comprises of all sample values x}

i from a dataset in

respect to a given feature f into distinct and disjoint intervals, referred to also as bins. The idea is, if sample values representing the input space~xis seen as a distribution, then it can be partitioned into confidence intervals. As a result, a meaningful confidence value can be associated with each valuexidepending on what bin it falls into which is otherwise

non-existent when only hard thresholds are used. The authors state that this additional information about a sample can be extracted and used to improve both learning and detection.

This general principle is extended here to the problem of generating multiclass weak hypotheses and is also inspired by the work of [173], who demonstrated how optimal thresholds for simple decision stump learners can be calculated with speed and efficiency. A variety of existing methods for generating multiclass hypotheses could have been used instead. Of these, C4.5 was a candidate. However, such sophisticated methods, though fast on small and moderately sized datasets [149], can be computationally expensive on larger datasets [150] and are hard to callibrate so that they do not overfit the data [59].

For these reasons the motivation here was to examine how well a cascaded framework can learn and generalize given a weak classifier, that is both inexpensive and unlikely to overfit. With that in mind, the aim was to design a multiclass weak learner for the cascaded framework, that complements the ’separate-and-conquer’ strategy, by producing high accuracies for at least one class label. Though overall error rates that are only marginally better than random guessing as in K−1

K forK classes was desirable, it was not

absolutely required for every weak classifier.

While Schapire and Singer [142] divide the binary-class domain~xinto a predetermined number of disjoint bins, the learner presented here divides the vector ~x into K possibly

4_{Both IREP and RIPPER employ pruning while RIPPER uses sophisticated global optimization after}

Algorithm 11: Domain-partitioning Weak Learner

Given: T_k+= total weights for all samples (xi, yi) wherek=yi,Tk− = total weights for all

samples (xi, yi) wherek6=yi, ω

+

k,ω−k = current sum of weights,Zk =

normalization coefficient for classk,ε→

k = current error for threshold pointing

right,ε←

k = current error for threshold pointing left,Ek = minimum error for class

k

Input: training setD={(~x1, y1), ...,(~xn, yn)}where~xf is a feature vectorf of a totalF

features and yi is a class label whereyi∈Y andY ∈ {0, ..., K}. Each samplexi

has an associated weightwi.

Output: multiclass weak classifierhconsisting of (t~1f, ~t1f, ~E) wheret~1f_k, ~t2f_k are primary

and secondary thresholds respectively for a feature f and classk

∀k,initializeTk+,Tk− 1 ∀k,computeZk 2 foreachfeaturef toF do 3 ~x=∀i,(xfi, yi) 4 sort~x 5 foreachvalue xi to xn do 6 foreachclassk to Kdo 7 ω_k+= K P k=0 Zk·wi,whereyi=k 8 ω− k = K P k=0 Zk·wi,whereyi6=k 9 ε← k =ω + k +Tk−−ω−k 10 ε→ k =ω−k +T + k −ω + k 11 if E_kf < M in(ε← k , ε→k )then 12 E_kf =M in(ε← k , ε→k ) 13 t1f_k =xi, GetBestErrorDirection(ε←k , ε→k ) 14

∀k,re-initialize Tk+,Tk− with respect tot1

f k 15 ∀k,computeZk 16 foreachvalue xi to xn do 17 foreachclassk to Kdo 18 if (t1f_k is ← ∧xi< t1fk)∨(t1 f k is → ∧ xi> t1fk)then 19 ω_k+= PK k=0 Zk·wi,whereyi=k 20 ω− k = K P k=0 Zk·wi,whereyi6=k 21 if t1f_k is ←then 22 ε→ k =ωk−+T + k −ω + k 23 else 24 ε← k =ω + k +Tk−−ωk− 25 if E_kf< V alid(ε← k , ε→k )then 26 E_kf =V alid(ε← k , ε→k ) 27 t2f_k =xi 28 returnf =M in(Ef₎ 29

7.3. Experiment Design ₁₀₉

overlapping bins, where each bin jk represents a bin that is associated with a single

class label. The weak learner proceeds by first sorting the feature vector ~x. The learner then traverses ~x, searching forK optimal thresholds tk for each class with an associated

direction. At each potential threshold point xi, the error is calculated K times for each

class. However, the error is calculated with respect to a binary distribution, where an even sum of weights is assigned to both the positive and the negative sets. In effect, the error calculation reduces to a form of one-against-all training. The first traversal of ~x is sufficient to generateK optimal threshold values with respect to a binary distribution for each class, together with their directions. Algorithm 11 details these steps.

An additional traversal of ~x is necessary to generate a bound on the first threshold so that bins and partitions can be created. This bound is referred to as a secondary threshold and it represents an optimal threshold with respect to the first threshold, being based on the error of the binary distribution. Each secondary threshold must lie within the value range that corresponds to the direction of the first threshold. The direction of the secondary threshold needs to be opposite to the direction of the first threshold so that a coherent partition of a class can be created (Figure 7.2).

Each partition jk is assigned a confidence value based on its accuracy. The average

error rate of all partitions defines the overall error rate for the given hypothesis and a given feature. Predictions for samples on overlapping partitions is awarded to the partition with the highest confidence.

Given a total pool of F features, for each feature vector~xf _{the weak learner executes}

a quicksort function which is of orderO(N log(N)) forN samples and traverses the vector ~xf _{twice. This results in a total number of iterations amounting to}

F(N log(N) + 2N) (7.5)

which can be expressed in terms of complexity without the linear component as

O(F N log(N)) (7.6)

Under usual conditions, the size of the training samples therefore dominates the complex- ity; however, exceptions can occur when training images using Haar-like features, whose total feature space can be in excess of 200,000 features.

7.3

Experiment Design

Experiments were performed using the above cascading algorithm, combined with the domain-partitioning (DP) weak-learner and AdaBoost.M15_{[44] for multiclass nodes, while}

Discrete AdaBoost and decision stumps were used for auxiliary nodes. The experiments were conducted on 18 multiclass datasets from the UCI machine learning repository. The dataset attributes are described in Chapter 3.2. The combination of the two is henceforth referred to as the cascaded.DP algorithm. AdaBoost.OC, ECC and M2 were implemented in C to compare with the proposed algorithm.

5_{AdaBoost.M1 is identical to Discrete AdaBoost and differs only in that it is able to handle multiple}

classes. Its only point of difference is in the evaluation of the ensemble, whereby the class label with the maximum sum of weighted votes becomes the predicted label.

Figure 7.2: An example of the proposed multiclass weak learner selecting the best features on the first 8 boosting iterations of the Pendigit dataset from left to right.

7.3. Experiment Design ₁₁₁

The justification for selecting AdaBoost.OC and AdaBoost.M2 for comparisons is be- cause they have been shown to clearly perform better than other contending methods like naive ECOC and Bagging in experiments carried out by Schapire [140]. Experi- ments on similar datasets by Jin and Zhang [72] also indicated a stronger performance by AdaBoost.OC over ECOC methods. Both AdaBoost.OC and AdaBoost.ECC are still widely used [133] and AdaBoost.ECC has been until recently a subject of research and extension [154]. In addition, the experimental results from Li [86] have shown that the performance strength of AdaBoost.ECC is greater than OAO and OAA approaches which would also have been suitable candidates for comparisons. In addition, Guruswami and Sahai [58] claim that AdaBoost.ECC is both theoretically and experimentally superior than AdaBoost.OC.

For datasets with both training and test sets, experiments were executed ten times; otherwise, 10-fold cross-validation was employed in conjunction with 10 training repeti- tions for a total of 100 runs. All results were averaged and in the presence of randomness, standard error was reported. Four different cascaded.DP classifiers were trained for each dataset. These classifiers were trained with different parameter settings for the value Φ that determines the maximum number of weak classifiers per multiclass node. The value forΦwas set to 5, 10, 25 and 50. The training terminated once a layer for each class label was constructed.

For a fair comparative analysis, ECC, OC and M2 classifiers were also trained using weak inducers in the form of decision stumps. The terminating condition for these algo- rithms was a predetermined maximum number of boosting iterations. This upper limit was in line with experiments in [35, 86] and is shown alongside the test error rates in the results section6.

The AdaBoost.M2 algorithm requires that the entire search space of all possible com- binations of class labels be examined for all feature types at every boosting iteration in order to find the lowest pseudoloss combination. Thus, the complexity of an exhaustive search becomes F ×2k_{, where F is the number of features and k is the number of class}

labels7_{. This is the primary performance bottleneck for AdaBoost M2; however, since the}

number of class labels is usually not greater than 10, it is feasible to perform an exhaustive search over all possible class combinations. In the case of the Letter dataset though, there are 26 class labels which makes the comprehensive evaluation over the complete search space unfeasible. In order to achieve practical training runtimes, the total search space of functionally unique set combinations for the Letter dataset was minimized from the total possible size of 33,554,431 to 1,0008 _{randomly sampled class combinations.}

6_{Experiments by Schapire [140], Sun et al. [154] set the ensemble size for similar datasets to 500;}

however, they used stronger inducers than the decision stumps here and so were able to achieve faster convergences in fewer boosting rounds.

7_{While the exhaustive number of possible permutations is 2}k_{, not all permutations form sets}

which are functionally unique. Due to this, the total number of unique set combinations can be re-

duced. Calculating this figure becomes a combinatorial problem and can be calculated as k!

1!(k−1)! + k! 2!(k−2)!+, ...,+ k! 2(k 2)!( k 2)! up to k

2 class labels whenkis an even number. Whenkis odd, then the last

term is not used and the terms up to to k−1

2 class labels are summed.

8_{The considerable reduction of the search space was justified on the basis that a comparative training}

runtime to other algorithms was required as a major component of these experiments. In addition, the assumption was made which expected that a set of sufficiently diverse colourings would be generated, with

For M2, a fast method for determining all plausible class label combinations was devised using bitwise operators. Firstly, each class label was assigned a value∀i∈ {1..k} : (yi, i). Each value j → 2k was then cast into an unsigned integer value. The component

bits{1..k}of the unsigned valuej1..k were then associated with (yi, i) labels. These could

then be efficiently extracted at each iteration through a shift operator. The plausible set µt was constructed consisting of class labelsy where the bit valuesji..k →(yi, i) and the

bit ji = 1.

The same basic approach was used to determine the colouring µt for AdaBoost.OC

and ECC with slight modifications. Schapire [140] showed that the size of the colouring set µt needs to be k₂. This time an array ~a was generated consisting of n ∈ {1..2k}

possible number of integer values, whose number of positive bits equaled k₂. In the case that there was an odd number of class labels, then the value with positive bits k₂+ 1 was selected. At training, a new random number jn _{was generated at each boosting round}

which determined the colouring and from which k₂ number of class labels were extracted as before.

It was not straightforward devising a fair and a completely accurate method for com- paring the training and execution runtimes of cascaded and their associated single-layer classifiers. This was so since both runtimes are directly affected by the size of a classifier’s ensemble and the point on their receiver operating curves (ROC) that they are being com- pared to. In the case of cascaded.DP classifiers, the ensemble size is not predetermined prior to training, while for the single-layer classifiers it is, since very low training errors on complex datasets with weak learners are often hard to attain. Since determining the op- timal criterion for halting the training of single-layer classifiers is subjective, it challenges the validity of comparing classifiers only on their absolute training and execution run- times and gives an occasion to unfairly prejudice the runtime performance of the control algorithms if their predetermined ensemble size is unnecessarily large.

Consequently, two values were used to provide a balanced comparison between the pairs of classifiers. One value reports a factor by which one algorithm trains or executes

a single weak classifier faster than the other. This figure provides fairness to single-layer

algorithms which could have been trained with fewer boosting iterations in order to attain a comparable generalization. The second value provides the same kind of comparison, however this time from the perspective of the entire classifier in order to provide a balanced view. Therefore, conclusive statements about which algorithm performs more efficiently will only be warranted if a superior performance is measured on both metrics.