classification
The ROC surface is a useful tool to assess the accuracy of a diagnostic test when three ordered groups are involved. To introduce the ROC surface let there be three separately ordered groups, denoted by X, Y and Z. Assume that we have real-valued data from diagnostic tests on individuals from the three groups; groupX withnx observations, group
Y with ny observations and group Z with nz observations. Assume that a continuous
diagnostic test is used to distinguish the individuals from the three groups. Suppose that
the measurements from group X tend to be smaller than those from group Y, which
in turn tend to be smaller than those from group Z. Let the cumulative distribution functions (CDFs) for the test outcomes of the three groups X, Y and Z be denoted by Fx, Fy and Fz, respectively.
For a decision rule, two thresholds c1 < c2 are required to classify individuals, based
on their diagnostic test results, into one of the three groups, such that a test value which is less than or equal toc1is an indication that this individual belongs to groupX, a test value
3.2. Thresholds selection in three-group classification 35
which is greater than c1 and less than or equal to c2 is an indication that this individual
belongs to group Y, and a test value which is greater than c2 is an indication that this
individual belongs to groupZ. The probability of correct classification for the three groups with thresholds c1 < c2 are as follows; p1 = P(X ≤ c1) = Fx(c1) is the probability of
correct classification for individuals from groupX,p2 =P(c1 < Y ≤c2) =Fy(c2)−Fy(c1)
for individuals from groupY andp3 =P(Z ≥c2) = 1−Fz(c2) for individuals from group
Z. The three-class ROC surface is a plot of these probabilities of correct diagnosis for all possible values c1 < c2 [52, 53, 56]. For three-group classification problems, the volume
under the ROC surface (VUS) has been extensively studied for assessment accuracy of a diagnostic test to differentiate among the three groups [1, 52, 70, 71]. The VUS is equal to the probability that three randomly selected measurements (one from each disease group) are ordered correctly. It takes the value 1 if the three groups are perfectly ordered and the value 1
6 if the diagnostic test results for the three groups are identical.
Once the accuracy of a diagnostic test is determined over all the possible thresholds, the selection of optimal thresholds is required to discriminate between the three groups. The common approach is the generalization of the Youden index as introduced by Nakas et al. [54], which is an extension of the two-group Youden index, discussed in Section 1.2, to the three-group setting. The three-group Youden index (3-YI) is defined as
3-YI = max
(c1<c2)
{Fx(c1) +Fy(c2)−Fy(c1) + 1−Fz(c2)} (3.1)
The optimal thresholds are the values of c1 and c2 which maximise the 3-YI, with the
constraintc1 < c2, where 3-YI is equal to 1 when the three groups are identical, and equal
to 3 where they are perfectly distinguished. In order to obtain the empirical estimator for the 3-YI, replace the CDFs by the corresponding empirical CDFs. The empirical estimate of the Youden index (3-EYI) is given by
3-EYI(c) = 1 nx nx X i=1 1{xi ≤c1}+ 1 ny ny X j=1 1{c1 < yj ≤c2}+ 1 nz nz X l=1 1{zl > c2}. (3.2)
Other methods for three-group thresholds selection based on ROC analyses are the closest to perfection method (3-MD) and the maximum volume method (3-MV), as
introduced by Attwood et al. [4]. Both approaches are generalisations of corresponding methods in two-group classification, namely the closest-to-(0,1) method [11, 65] and the maximum area method, respectively.
The 3-MD approach selects the optimal thresholds which generate the point on the ROC surface closest to the point of perfection (1,1,1) (i.e. the point closest to perfection with p1(c1) = 1, p2(c1, c2) = 1 and p3(c2) = 1). The optimal thresholds are the values of
c1 and c2, which minimise the distance, and this method is given by
3-MD = min
(c1<c2)
{q(1−p1(c1))2+ (1−p2(c1, c2))2+ (1−p3(c2))2} (3.3)
The 3-MV method can be defined as the maximum product of the correct classification probabilities for the three groups as follows
3-MV = max
(c1<c2)
{p1(c1)×p2(c1, c2)×p3(c2)} (3.4)
Attwood et al. [4] did not mention the empirical estimate of the maximum volume (3-EMV) method in their paper, which is defined by
3-EMV(c) = 1 nx nx X i=1 1{xi ≤c1} × 1 ny ny X j=1 1{c1 < yj ≤c2} × 1 nz nz X l=1 1{zl> c2}. (3.5)
Attwood et al. [4] also discussed a comparison of optimal thresholds selected by their methods and the three-group Youden index method (3-YI). We review some results obtained by [4]. Although the 3-YI approach maximises the total number of correct classification rates for the three groups, it tends to have limitation in selecting the thresholds c1 and c2. The maximisation problem in Equation (3.1) can be written as
the maximisation of two two-group problems, one between the healthy and intermediate groups and the other between the intermediate and diseased groups, given that c1 < c2.
This can lead to imbalanced classification rates between the three groups, in favour of identifying healthy and diseased groups but poor identification of the intermediate group. This aspect is in line with the results of the simulation study, which will be presented it in Section 3.7.