Notation

We introduce the notation for the this dissertation.

Bold capital letters denote matrices (e.g. X), bold lower-case letters represent vectors (e.g.,x). All non-bold letters denote scalar variables. xi is thei−th row of the matrixX.

14 INTRODUCTION

xjis thej−th column of the matrixX. 1is a matrix or vector of all ones of the appropriate

size. Iis the identity matrix. diag(d) denotes a matrix withdin the main diagonal. xij

denotes the scalar in thei−th row andj−th column ofX. kXkF = tr(X>X) denotes the

Frobenius norm. k·kpis used to denote the Lp-norm. x⊕yis an operator which concatenates

vectorsxandy. rank(X) denotes the rank ofX. X≤0 denotes the point-wise inequality. Table 1.2 shows the different symbols used in this dissertation.

Matrix of training samples X i−th training sample xi Vector of labels y label of thei−th samples yi

i-th category ci _{Indicator function I}

ECOC coding matrix M i-th codeword mi

i−th dichotomy mi i−th classifier fi

Number of classes k Number of dichotomies l

Vector of classifier predictions

for thei−th sample f(xi) Decoding measure δ

Chromosome s Error function E

Number of Generations G i−th individual Ii

Confusion matrix C Performance of binary classifiers p

Dichotomy selection order t Mutation control value mtc

Separability matrix H i-th row ofH hi

Design Matrix D i−th row ofD di

Minimum distance between

pairs of codewords P i−th row ofP p

i

Matrix of eigenvectors V Vector of eigenvalues λ

Likelihood map B Weight matrix W

Part I

Ensemble Learning and the

ECOC framework

Chapter 2

2.1

The Error-Correcting Output Codes Framework

ECOC is a general multi-class framework built on the basis error-correcting principles in communication theory [39]. This framework suggests to view the task of supervised multi- class classification as a communication problem in which the label of a training example is being transmitted over a channel. The channel consists of the input features, the training examples, and the learning algorithm. Because of errors introduced by the finite number of training samples, choice of input features, and flaws in the learning process, the label information is corrupted. By coding the label information in an error-correcting codeword and transmitting (i.e learning) each bit separately (i.e., via uncorrelated independent binary classifiers), the algorithm may be able to recover from the errors produced by the number of samples, features or learning algorithm.

The ECOC framework is composed of two different steps: coding [3, 39] and decoding [47, 146]. At the coding step an ECOC coding matrixM∈ {−1,+1}k×l_{is constructed, where} k denotes the number of classes in the problem and l the number of bi-partitions defined to discriminate thekclasses. In this matrix, the rows (mi_{’s, also known as error-correcting}

codewordsor codewords for shorter) are univocally defined, since these are the identifiers of each category in the multi-class categorization problem. On the other hand, the columns of Mdenote the set of bi-partitions, dichotomies, or meta-classes to be learnt by each binary classifierf(also known as dichotomizer). In this sense, classifierfjis responsible of learning the bi-partition denoted on thej−th column ofM. Therefore, each dichotomizer learns the classes with value +1 against the classes with value−1 in a certain column. Note that the ECOC framework is independent of the binary classifier applied. For notation purposes in further sections we will refer to the entry ofMat thei-th row and thej-th column asmij.

Following this notation thei-th row (codeword of classci_{) will be referred asm}i _{and, the} j-th column (j-th bi-partition or dichotomy) will be referred asmj.

Originally, the coding matrix was binary valued (M∈ {−1,+1}k×l

). However, [3] intro- duced a third value, and thus,M∈ {−1,+1,0}k×l_{, defining ternary valued coding matrices.}

In this case, for a given dichotomy categories can be valued as +1 or−1 depending on the meta-class they belong to, or 0 if they are ignored by the dichotomizer. This new value allows the inclusion of well-known decomposition techniques into the ECOC framework, such has One vs. One [125] or Sparse [3] decompositions.

At the decoding step a samplextis classified among thekpossible categories. In order to

perform the classification task, each dichotomizer infjpredicts a binary value forxtwhether

it belongs to one of the bi-partitions defined by the correspondent dichotomy. Once the set of predictionsf(xt)∈Rlis obtained, it is compared to the codewords ofMusing a distance

18 ERROR-CORRECTING OUTPUT CODES REVIEW AND PROPERTIES

metric δ, known as the decoding function. The usual decoding techniques are based on well-known distance measures such as the Hamming or Euclidean distances. These measures were proven to be effective in binary valued ECOC matrices{+1,−1}. Nevertheless, it was not until the work of [47] that decoding functions took into account the meaning of the 0 value at the decoding step. Generally, the final prediction forxt is given by the classci,

where argmin

i

δ(mi,f(xt)),i∈ {1, . . . , k}. Figure 2.1 shows an example of ECOC matrices

for a toy problem.

By analysing the ECOC errors it has been demonstrated that ECOC corrects errors caused by the bias and the variance of the learning algorithm [81]. The variance reduction is to be expected, since ensemble techniques address this problem successfully [104] and ECOC is a form of voting procedure. On the other hand, the bias reduction must be interpreted as a property of the decoding step. It follows that if a point xt is misclassified by some

of the learnt dichotomies, it can still be classified correctly after being decoded due to the correction ability of the ECOC algorithm. Non-local interaction between training examples leads to different bias errors. Initially, the experiments in [81] show the bias and variance error reduction for algorithms with global behavior (when the errors made at the output bits are not correlated). After that, new analysis also shows that ECOC can improve performance of local classifiers (e.g., the k-nearest neighbor, which yields correlated predictions across the output bits) by extending the original algorithm or selecting different features for each bit [3].

x

t

f(x

t

)

3 44 00 1 22

(a)

(b)

(c)

f3 f2

f

1

f

4

f

5

m

1

m

2

m

3

m

4

m

5

m

1

m

2

m

3

m

4

m

5

m

1

m

2

m

3

m

4

m

5

m

1

m

2

m

3

m

4

m

5

δ

Figure 2.1: (a) Feature space and trained boundaries of dichotomizers. (b) Coding matrix M, where black and white cells correspond to{−1,+1}, denoting the two partitions to be learnt by each base classifier (white cells vs. black cells) while grey cells correspond to 0 (ignored classes). (c) Decoding step, where the predictions of classifiers, {h1, . . . , h5}for samplext are compared to the codewords{m1, . . . ,m5}

In document Learning error-correcting representations for multi-class problems (Page 35-41)