Linear Discriminant Analysis (LDA)

Linear Discriminant Analysis (LDA), also known as the Fischer Linear Discriminant technique is another method for projecting data to a lower dimensionality. When LDA is used to project data with more than two classes it is also occasionally re-

(a) The normalized two class data. (b) The first two PCs (eigenvectors) from PCA, overlaid on the normalized data set.

(c) The data is projected to one dimension using the largest eigenvalue’s eigenvector (PC) from PCA. Then creating a histogram of the pro- jected data the separability of the two classes is clearly shown.

(d) The data is projected to one dimension us- ing the second largest eigenvalue’s eigenvec- tor from PCA. Then creating a histogram of the projected data the separability of the two classes is clearly shown.

Figure 3.1: Using the simple two class, two feature data shown in (a) the first two PCs, which are eigenvectors from performing PCA, are shown overlaid on top of data in (b). Creating a histogram from the data after it is projected to one dimension allows the retained variance to easily be seen as well as the separability between the two classes. (c) is from using the primary PC and (d) is from using the second PC.

to project data to a lower dimensionality but differ significantly in their affect on the data. The difference is that LDA seeks a projection that allows for the best discrimi- nation between classes, while PCA seeks a projection that represents the data in the

fewest dimensions i.e. the most dimensionally efficient representation [27]. Another,

significant, difference is that LDA is a projection from a d-dimensional space to at most a (c − 1) dimensional space, where c is the number of classes represented in the data set. For example, the feature set at the point when LDA is applied, in this research, has 13 dimensions (features) and represents three unique classes. Therefore

the input to LDA, a 13 × n-signature feature matrix, is reduced to a 2∗× n-signature

feature space. Where as PCA could be used to project this data to any dimensionality, from 1 to 13.

To apply LDA a projection matrix is calculated in the form of a classic eigen-

problem, Eq. (3.3). This calculation is formulated using the between-class scatter

matrix, SB, calculated using Eq. (3.4), the with-in class scatter matrix, Sw, Eq. (3.5),

the scatter matrix of the i th class, (3.6), the mean vector of the entire data set, m,

Eq. (3.8), and the mean feature vector using just the i th class’s signatures Eq. (3.7).

The eigenvectors resulting from this calculation are sorted into descending value via the descending order of the eigenvalues. Then the (c − 1) eigenvectors, which are column vectors, are used to from the projection matrix by “stacking” the eigenvectors side-by-side, such that the eigenvectors remain columns.

SBwi = λiSwwi (3.3) SB = c X i=1 ni(mi− m)(mi− m)T (3.4) Sw = c X i=1 Si (3.5) Si = X x∈Di (x − mi)(x − mi)T (3.6) mi = 1 ni X x∈Di x (3.7) m = 1 n c X i=1 nimi (3.8)

∗ _{In this situation you could also use LDA to reduce to single dimension from 13 dimensions.}

where,

c is the number of classes,

x is the feature vector of single signature,

mi is the mean feature vector using only the i th class’ samples,

m is a vector containing the mean of each feature using the entire data set,

n is the number of samples in the entire data set,

ni is the total number of samples belonging to the i th class,

λi is the i th largest eigenvalue,

wi is the i th eigenvector (associated with the i th largest eigenvalue).

Performing LDA on the same two-class, two-feature data set used in the simplistic PCA example and overlaying the LDA projection vector with the data results in

Figure 3.2(a). By overlaying the vectors that are the three possible projections from

two dimensions down to one with the data set it is clear that the clearest separation

between Class 1 (blue) and Class 2 (red) will result from using the LDA projection

vector, Figure 3.2(b). A histogram of the 1-D LDA projected data set is shown in

Figure 3.2(c). LDA separated the data so well you can actual see that only three

samples from the red class (Class 2) would be miss-classified as the blue class (Class 1).

The LDA 1-D projection can be easily compared to the PCA 1-D projection (the

primary PC from PCA) using the histograms in Figure 3.3. Below each histogram,

in Figure 3.3, is a scatter plot of the actual 1-D projected samples from which the

histogram above each scatter plot was generated. A different marker is used to differ- entiate the two classes and also shows the ability to separate the two classes using only one dimension (feature). It is important to remember that while in this simplistic example LDA clearly separates the two classes using only one dimension better than PCA; this is a very idealistic example. PCA and LDA each provide benefits that the other does not. For example, PCA is good for removing stochastic noise and requires no training set i.e. it is an unsupervised dimensionality reduction technique. LDA will improve the classification results by maximizing the between-class variance while minimizing the in-class variance, but requires a training set i.e. it is a supervised technique.

(a) The normalized, simplistic two class data with the one-dimensional LDA projection vector overlaid.

(b) The two class data with the one-dimensional LDA projection vector overlaid along with the first two one-dimensional PCs from PCA.

(c) The data is projected to one dimension using the LDA projection matrix, then a histogram of the projected data was created to show the separability of the two classes.

Figure 3.2: Using the same simple two class, two feature data shown in Figure 3.1(a)

the vector along which the data will be projected to is shown overlaid on the data in (a). In (b) the one-dimensional projection vector for LDA, the primary PC from PCA, and the second PC from PCA are all shown overlaid together. Creating a histogram from the data after it is projected to one dimension via LDA allows the separability between the two classes to be seen in (c).

Figure 3.3: The histograms show the number of data points “in the same location” and the scatter plot underneath the histogram shows the actual 1-D projected samples the histogram was generated from. The blue bars and dots are Class 1. The redbars and circles are Class 2.