8.3
Assessment Metrics for Model Quality
The validation methods often employed to measure the accuracy of the approx- imation function, implicitly assumes that all types of misclassification incur an equal cost, this however, is not always the case. For example, considering a model for a safety-critical software system which predicts either a system failure- inducing or non-failure-inducing state. Predicting a failure-inducing state as being non-failure-inducing, will typically result in a much more significant cost than classifying a non-failure-inducing state as being failure-inducing. In such situations, the predictions of a model on a test data set can be cross-tabulated with the actual classes assigned to the instances by the target function to pro- duce a confusion matrix. Table 8.1 shows the general form of a confusion matrix for a binary classification problem (though it can easily be extended to the case of more than two classes). Classification tasks involving two classes is known as binary classification and the corresponding function approximation is known as a binary function. Typically, in binary classification, one class is the class of interest and is referred to as the “concept”. The instances belonging to this con- cept are referred to as positive instances or positives for short. On the contrary, all other instances not belonging to the concept are called negative instances or negatives for short. In the context of the aforementioned example, (and in most software dependability analysis that uses for binary classification), the function predicts either a system state is going to lead to a system failure or a successful execution and the positives are the instances labelled as those leading to system failure.
In Table 8.1, TP is the number of positive correctly labelled as positives by ˆf, known as true positives, whilst FN is the number of positives misclassified as negatives, known as false negatives. Further, FP is the number of negatives incorrectly classified as positives, known as false positives, whilst TN is the number of negatives correctly classified as negatives, known as true negatives.
8.3. ASSESSMENT METRICS FOR MODEL QUALITY 146
Finally,nposandnnegare the respective total number of positives and negatives
in the test data and ˆnpos and ˆnneg are the total number of instances predicted
as positives and negatives, respectively. In software dependability analysis, it is natural to seek out models that maximise true positives and minimise false pos- itives, not least because these correspond closely with the concepts of accuracy and completeness. However, as a balance must be struck between these related concerns, it is appropriate to identify an aggregated measures of model quality.
Table 8.1: The general form of a confusion matrix for binary classification.
Predicted Class
Positives Negatives Marginal Sums
Actual Class
Positives TP TN npos
Negatives FP FN nneg
Marginal Sums nˆpos ˆnneg n
A variety of metrics for model assessment are derived from a confusion matrix. The most commonly used of these metrics are the True Positive Rate (TPR), also known as sensitivity or recall, and true negative rate (TNR), also known as specificity. Sensitivity measures how often a model predicts an instance to be positive, when it is actually positive and it is computed as shown in Equation 8.1. Specificity measures how often a model predicts an instance to be negative when it is actually negative and it is computed as shown in Equation 8.2.
T P R = sensitivity = T P
T P + F N (8.1)
T N R = specif icity = T N
T N + F P (8.2)
8.3. ASSESSMENT METRICS FOR MODEL QUALITY 147
acteristic (ROC) Curve (or ROC for short) and area under the curve (AUC), F-score, precision, false positive rate (FPR), mcost, accuracy and misclassifica- tion rate.
Accuracy measures how often the model is correct overall and it is computed as shown in Equation 8.3. Misclassification Rate, measures how often the model is wrong overall and it is given as shown in Equation 8.4. Misclassification rate is also known as error rate and is equivalent to 1 minus accuracy.
accuracy = T P + T N
T P + T N + F N + F P (8.3)
error rate = 1−accuracy = F P + F N
T P + T N + F N + F P (8.4)
FPR, measures how often a model predicts an instance to be positive, when it is actually a negative. FPR is equivalent to 1 minus specificity, and its computation is shown in Equation 8.5.
F P R = 1 −specif icity = F P
F P + T N (8.5)
ROC analysis combine the FPR and the TPR into one single metric and it is based on a 2-dimensional graph that summarises the performance of a model over possible thresholds. It is generated by plotting the TPR (y-axis) against the FPR (x-axis) as the threshold for assigning instances to a given class is varied. It shows the tradeoff between sensitivity and specificity (any increase in sensitivity will be accompanied by a decrease in specificity). The closer the curve follows the left-hand border and then the top border of the ROC
8.3. ASSESSMENT METRICS FOR MODEL QUALITY 148
space, the more accurate the test. The closer the curve comes to the 45-degree diagonal of the ROC space, the less accurate the test. For different thresholds, the same classifier will produce multiple points on such a plot. The AUC it is obtained by joining these plotted points to (0,0) and (1,1). The AUC is a measure of expected model accuracy for the classifier. For a single model, the simple trapezium obtained by connecting the coordinates (0,0), (FPR,TPR), (1,1) and (1,0) has an area given by Equation 8.6. An area of 1 represents a perfect model.
AU C = T P R−F P R+ 1
2 (8.6)
The Euclidean distance from the perfect classifier, which has coordinates (0, 1), i.e, FPR = 0 and TPR = 1, may be used in the ranking of single models. This measure is given by the well known formula in Equation 8.7.
euclidean distance =p(F P R−0)2 + (1−T P R)2 (8.7)
Precision measure how often a model prediction is correct when it predicts yes and is given by Equation 8.8. Precision and Recall together with their harmonic mean F1 measure or balanced F-Score (F1-Score) are often used as a model quality metric in the domain of information retrieval. The F-Score is a weighted average of the recall and precision, and it is measured as shown in Equation 8.9.
precision = T P
8.3. ASSESSMENT METRICS FOR MODEL QUALITY 149
F1−Score = 2· precision·recall
precision+recall (8.9)
When specific classification errors are more severe than others, e.g., When the cost associated with a false positive is different from that of a false negative, a more appropriate measure of quality is expected, misclassification cost, rather than the expected error. This requires the definition of a cost matrix. Assuming there aremclass labels,Li, anm×mcost matrix,C, needs to be defined such
that the value C(i, j) is the cost of misclassifying an instance of class Li to
the class Lj. Clearly C(i, i) = 0 as there should be no cost associated with
correctly classifying an instance. The model tries to avoid classification errors with a high error weight. Weights specified must be greater than or equal to zero. The default weight is 1. The cost matrix diagonal must be zero. Minimising the error is a special case of minimising misclassification cost when the cost matrix is defined as C(i, j) = 1, where i 6= j and C(i, i) = 0. The expected misclassification cost,mcost, can then be calculated as shown in Equation 8.10, whereCM(i, j) represents index access to the associated confusion matrix using iandj. mcost = m X i m X j C(i, j)·CM(i, j) (8.10)
The proposed approach in this chapter focuses on the generation of efficient fault injection points, which means that the model quality is evaluated with respect to the efficiency, i.e., the levels of accuracy and completeness, that can be achieved by these points. With this in mind, the AUC measure, which is an aggregate representative of accuracy and completeness in the form of TPR and FPR, is used in measuring model quality. However, as misclassification