Classification Based on Mahalanobis Distance

With Fisher’s linear discriminant function, the classification consists of projecting p-dimensional observed data from both classes onto axes ob-tained by linear combination of variables and selecting as the optimum projection axis the one that maximizes the ratio of between-class vari-ance to within-class varivari-ance. In this section, we discuss a method in which the distance of an observation to the class sample mean vector is defined and the observation is assigned to the class being the closest dis-tance. The sample mean vector representing each class can be considered as its center of mass.

6.2.1 Two-Class Classification

Suppose that we have n1 p-dimensional data from class G₁ and n2 p-dimensional data from class G2, and represent the sample mean vector and variance-covariance matrix for each class as

G₁: x1= 1

The (squared) Mahalanobis distance of x to the i-th class sample mean vector xiis then defined by

D²_i = (x − xi)^TS⁻¹_i (x− xi), i= 1, 2. (6.34) A new observed data x is assigned to the class whose sample mean vector is closest to x in Mahalanobis distance. The classification rule assigns x to G₁if

Q(x)= D²₂− D²₁ (6.35)

= (x − x2)^TS⁻¹₂ (x− x2)− (x − x1)^TS⁻¹₁ (x− x1)≥ 0, and to G2otherwise, which gives a quadratic discriminant function with quadratic decision boundaries.

The sample variance-covariance matrices S1 and S2 are combined to obtain an unbiased estimator of the common population variance-covariance matrixΣ in the following:

S = 1

n₁+ n2− 2{(n1− 1)S1+ (n2− 1)S2} . (6.36) Then the quadratic function Q(x) = D²₂− D²₁ in (6.35) can be rewritten as

h(x)= (x1− x2)^TS⁻¹x−1 2

x^T₁S⁻¹x₁− x^T2S⁻¹x₂

. (6.37) Noting that

x^T₁S⁻¹x₁− x^T₂S⁻¹x₂

= (x1− x2)^TS⁻¹(x1+ x2), the discrim-inant function based on Mahalanobis distance with the pooled sample variance-covariance matrix corresponds to Fisher’s linear discriminant function in (6.31).

In quadratic discriminant analysis, in contrast to linear discriminant analysis, it is difficult to assess the relative importance of the variables from their estimated coefficients. To investigate variable x1, for example, it is necessary to consider the estimated values of the coefficients of both x²₁and x₁, x₂in the quadratic equation at the same time, making it quite difficult to determine whether it is effective for separation of the two classes. If the sample variance-covariance matrices of the two classes differ substantially and the number of samples from each class is quite large, on the other hand, quadratic discriminant analysis is more effective than linear discriminant analysis for good separation of the two classes.

6.2.2 Multiclass Classification

When training data from three or more classes have been observed, the construction of a classification procedure based on Mahalanobis distance is straightforward. The basic concept is similar to that of two-class clas-sification: when an observation x has been obtained, the distance to the center of mass of each class (the sample mean vector) is computed and x is assigned to the class achieving the shortest distance. ɹ

Suppose that we have nj p-dimensional data from class G_j ( j = 1, 2,· · · , g)

G_j: x^{( j)}₁ , x^{( j)}₂ , · · · , x^{( j)}n_j, j= 1, 2, · · · , g. (6.38) For each class Gj the sample mean vector and the sample

variance-covariance matrix are given by Then the classification rule is constructed by assigning x to class Gj,

j= 1, 2, · · · , g, for which

D²_j= (x − xj)^TS⁻¹_j (x− xj), j= 1, 2, · · · , g (6.40) is smallest, which gives quadratic discriminant functions with quadratic decision boundaries

Qjk(x)= D²j− D²k j k; j, k = 1, 2, · · · , g. (6.41) Taking the pooled sample variance-covariance matrix as

S = 1 n− g

(n1− 1)S1+ (n2− 1)S2+ · · · + (ng− 1)Sg

, (6.42)

where n = n1+ n2 + · · · + ng, we have linear discriminant functions with linear decision boundaries. The g predicted classes are separated by linear functions

h_jk(x)= D²_j− D²_k j k; j, k = 1, 2, · · · , g. (6.43)

What is Mahalanobis distance? Figure 6.4 shows the data dispersion in the form of an ellipse. This probability ellipse is analogous to the representation of elevation in a mountain contour map and shows that the data concentration increases as the center of mass is approached. In the left panel the sample mean vector and variance-covariance matrix are

x=

The squared Euclidean distances from the points xP = (9.5, 11.8)^T and xQ = (9.5, 8.2)^T to x are (xP− x)^T(xP− x) = (xQ− x)^T(xQ− x)

= 6.85. In contrast the squared Mahalanobis distances are D²_P = (xP − x)^TS⁻¹(xP− x) = 2.48 and D²_Q= (xQ− x)^TS⁻¹(xQ− x) = 12.61. The Ma-halanobis distances of points P and Q from x are clearly different; point P is substantially closer. The inclusion of point P in the class is stronger than that of Q, and this is effectively captured by the Mahalanobis dis-tance. In the mountain analogy, the Mahalanobis distance is an explicit expression of the relatively gentle gradient (slope) of a climb from point P to the center of mass, and the relatively steep gradient from point Q.

䞉

Figure 6.4 Mahalanobis distance and Euclidean distance.

6.2.3 Example: Diagnosis of Diabetes

Reaven and Miller (1979) have described a classification for 145 non-obese adult subjects by three variables, with assignment to three classes:

a normal class (G1), a chemical diabetes class (G2) with no finding of clinical abnormality but with a glucose tolerance test finding of abnor-mality, and a clinical diabetes class (G3) showing clinical symptoms characteristic of diabetes. Here, we first configure the three-class classi-fication based on two of the test variables. The two diabetes test variables are as follows.

x₁: Glucose is ingested after fasting by each subject, the plasma glucose level is then measured every 30 min for 3 h, and the area under the resulting curve is taken as a test value.

x₂: In the same way, the plasma insulin level is measured and the area under the resulting curve is taken as a test value.

Figure 6.5 shows the plot of 145 training data for a normal class G₁ (◦), a chemical diabetes class G2 () and clinical diabetes class G3

(×), in which G1, G2, and G3 include n1 = 76, n2 = 36, and n3 = 33



Figure 6.5 Plot of 145 training data for a normal class G1(◦), a chemical dia-betes class G₂(), and clinical diabetes class G3(×).

observations, respectively. The objective is to formulate a method for effective separation of these three classes.

The sample mean vectors and sample variance-covariance matrices are Let us consider an individual who newly undergoes the tests with result-ing test values of x= (x1, x₂)^T= (820, 150)^T. The question is then which of the three classes, normal, chemical diabetes, or clinical diabetes, this person belongs to. For this diagnosis of diabetes, we use linear discrim-inant analysis based on a pooled sample variance-covariance matrix for the three classes.

It follows from (6.45) and (6.42) with g= 3 that the pooled sample variance-covariance matrix is

S =

 23046.8 −3901.4

−3901.4 10610.9

 .

The squared Mahalanobis distances of x= (x1, x₂)^T= (820, 150)^T to xi, the sample mean vector of Gi, are given by

G₁: D²₁= (x − x1)^TS⁻¹(x− x1)= 9.91,

G₂: D²₂= (x − x2)^TS⁻¹(x− x2)= 5.3, (6.46) G₃: D²₃= (x − x3)^TS⁻¹(x− x3)= 2.18.

The sample mean vector x₃ in G₃ is closest to x= (820, 150)^T in Ma-halanobis distance, and thus we assign this data x as belonging to the clinical diabetes class.

Figure 6.6 shows the lines separating the three classes based on Mahalanobis distance. With line L12, the Mahalanobis distances D²₁ to the center of mass x1 = (350.0, 172.6)^T of class G1 and D²₂ to that x₂ = (493.9, 288.0)^T of class G2 each represents the set (x1, x₂) such that D²₁ = D²₂ (D²₂− D²₁ = 0), and the locus is clearly a straight line.

The L12represents a linear decision boundary that separates the normal class G1and the chemical diabetes class G2. Similarly, L23separates the chemical diabetes class G2and the clinical diabetes class G3, and L31the clinical diabetes class G3and the normal class G1. These linear discrim-inant functions are given by

L₁₂= −0.0086x1− 0.014x2+ 6.873, L₂₃= −0.0223x1+ 0.0089x2+ 15.418, L₃₁= 0.031x1+ 0.0051x2− 22.291.

As shown in Figure 6.6, the separation of the regions in the input space by a straight line clearly demonstrates that we are in this case es-sentially performing linear discriminant analysis of multiple class com-binations. If such predicted class regions are formed in classification based on two variables, then the class membership of new test values can be readily determined by plotting them in the figure. The course of the symptoms can also be ascertained to some degree from the positional relationship of each class. Finally, by narrowing the normal class region, it is possible to reduce the probability of diagnostic failure by assign-ing an individual actually havassign-ing an abnormality to the normal class. For data dispersed in a higher-dimensional space we consider a method of projection onto a two-dimensional plane or a three-dimensional space for visualization in Section 6.4.





















;

       

;

⩌

⩌

⩌

ᩓᕸᅗ

ศ๭ኚᩘ㸸&ODVV

[

/

/

/

[

Figure 6.6 Linear decision boundaries that separate the normal class G1, the chemical diabetes class G₂, and the clinical diabetes class G3.