Final model training and evaluation

The final phase of the ANN model development process entailed training the ANN models for each of the wheel wear measurements on the entire train data set, with the hyperparameters set to those in Table 17. As with the logistic regression models, each model’s performance was evaluated based on confusion matrix statistics, ROC curves and the AUC measure.

i) FH model evaluation

The confusion matrix for the FH wear prognostic ANN model is provided in Table 18. From Table 18, accuracy, sensitivity, specificity, and the F1 measure were calculated. These values are provided in Table 19. An ROC curve, which is illustrated in Figure 39, was produced for the ANN model performance on the test set. The associated AUC value was 0.874.

Table 18: Confusion matrix for ANN model of FH wear prognostics

Predicted Class = 1 Predicted Class = 0

Actual Class = 1 4’908 5’130

Table 19: Confusion matrix metrics for ANN model of FH wear prognostics Metric Value Sensitivity 0.489 Specificity 0.996 Accuracy 0.934 F1 0.964

Figure 39: ROC curve for ANN model of FH wear prognostics ii) TD model evaluation

The confusion matrix for the TD wear prognostic ANN model is provided in Table 20. From Table 20, accuracy, sensitivity, specificity, and the F1 measure were calculated. These values are provided in Table 21. An ROC curve, which is illustrated in Figure 40, was produced for the ANN model performance on the test set. The associated AUC value was 0.977.

Table 20: Confusion matrix for ANN model of TD wear prognostics

Predicted Class = 1 Predicted Class = 0

Actual Class = 1 38’500 2’898

Table 21: Confusion matrix metrics for ANN model of TD wear prognostics Metric Value Sensitivity 0.930 Specificity 0.996 Accuracy 0.963 F1 0.964

Figure 40: ROC curve for ANN model of TD wear prognostics iii) HW model evaluation

The confusion matrix for the HW wear prognostic ANN model is provided in Table 22. From Table 22, accuracy, sensitivity, specificity, and the F1 measure were calculated. These values are provided in Table 23. An ROC curve, which is illustrated in Figure 41, was produced for the ANN model performance on the test set. The associated AUC value was 0.794.

Table 22: Confusion matrix for ANN model of HW wear prognostics

Predicted Class = 1 Predicted Class = 0

Actual Class = 1 1’390 5’710

Table 23: Confusion matrix metrics for ANN model of HW wear prognostics Metric Value Sensitivity 0.196 Specificity 0.993 Accuracy 0.925 F1 0.960

Figure 41: ROC curve for ANN model of HW wear prognostics iv) FS model evaluation

The confusion matrix for the FS wear prognostic ANN model is provided in Table 24. From Table 24, accuracy, sensitivity, specificity, and the F1 measure were calculated. These values are provided in Table 25. An ROC curve, which is illustrated in Figure 42, was produced for the ANN model performance on the test set. The associated AUC value was 0.969.

Table 24: Confusion matrix for ANN model of FS wear prognostics

Predicted Class = 1 Predicted Class = 0

Actual Class = 1 50’171 5’501

Table 25: Confusion matrix metrics for ANN model of FS wear prognostics Metric Value Sensitivity 0.901 Specificity 0.968 Accuracy 0.923 F1 0.892

Figure 42: ROC curve for ANN model of FS wear prognostics v) FT model evaluation

The confusion matrix for the FT wear prognostic ANN model is provided in Table 26. From Table 26, accuracy, sensitivity, specificity, and the F1 measure were calculated. These values are provided in Table 27. An ROC curve, which is illustrated in Figure 43, was produced for the ANN model performance on the test set. The associated AUC value was 0.772.

Table 26: Confusion matrix for ANN model of FT wear prognostics

Predicted Class = 1 Predicted Class = 0

Actual Class = 1 1’677 5’729

Table 27: Confusion matrix metrics for ANN model of FT wear prognostics Metric Value Sensitivity 0.226 Specificity 0.995 Accuracy 0.926 F1 0.960

Figure 43: ROC curve for ANN model of FT wear prognostics

4.2.3.4 ANN performance summary

The ANN model type performed very well on the FH wear measurements. The model achieved an accuracy rate of over 90% and its AUC score was 0.87, which is high. The model’s ROC curve was indicative of a healthy ML model, that is behaving as expected, with a smooth curve that thends toward the (0,1) point before curving toward the (1,1) point. This indicates that the model was able to separate the target variable classes well.

As with the logistic regression model, the ANN performed extremely well on the TD wear measurements. The model attained an accuracy rate of over 95% and had an extremely high AUC of 0.977. The extremely good performance leads to the same concerns that was raised with the logistic regression model, which was that the extremely good performance might be due to either easy seperability of the TD target variable classes, or that the TD related data exhibited strange behaviour.

The ANN had difficulty with providing HW prognostics. Although the model achieved an accuracy rate of 92.5%, it had a relatively low sensitivity rate and, therefore, a relatively low AUC of 0.794. This indicates that the model performed moderately well when it came to separating the target variable classes.

The ANN model performed extremely well when it came to providing FS wear prognostics. The ROC curve tended to very near to the (0,1) point, before elbowing sharply toward the (1,1) point. This indicates that the model was capable of separating the target variable classes very well. The model achieved an accuracy rate of 92.3% and had an AUC of 0.969, which is high. Although the model performance and its ROC curve is similar to that of the TD case, the concern that was raised for the TD case is not raised here, because FS is a directly measured metric, as opposed to TD, which is an extrapolated metric. Therefore, the risk of a model achieving exceptional performance due to anomlies in the data is less in the FS case than in the TD case.

Finally, the ANN model performed moderately well when it came to FT wear prognostics. The model attained a high accuracy rate of 92.6%, however, the specificity rate was relatively low, at 22.6%. This indicates that the model struggled to separate the target variable classes. This notion is supported by the relatively flat shape of the ROC curve and its relatively low AUC of 0.727.