Metrics for Prognostics Algorithm Performance

Uncertainties arise from various sources in a PHM system (Coppe, 2009), (Hastings and Mcmanus, 2004) & (Orchard et al., 2008). Some of these sources include:

• Model uncertainties (errors in the representation and parameters of both the system model and fault propagation model),

• Measurement uncertainties (these arise from sensor noise, ability of sensor to detect and disambiguate between various fault modes, loss of information due to data pre- processing, approximations and simplifications),

• Operating environment uncertainties,

• Future load profile uncertainties (arising from unforeseen future and variability in usage history data),

Intercept of failure

The potential failure degradation loci RUL

Feature vector representing degradation

Potential Failure Interval

Band of normal operation

Threshold of fault detection

172

• Input data uncertainties (estimate of initial state of the system, variability in material properties, manufacturing variability), etc.

Performance metrics for prognostics can be classified into accuracy, precision, and robustness. The definition of accuracy is the degree of closeness of the prediction to the actual failure time. Where the precision is defined as the spread of prediction performed at the same time and the robustness is the sensitivity of the predictions changes of algorithm parameter variations or external disturbances. A comprehensive list of performance metrics, is given by (Saxena, et al., 2008), (Saxena, et al., 2009) & (Saxena, et al., 2009)of which the most prevalent are discussed below.

5.9.1 Performance Metrics

Over the years, various metrics have been developed for the measure of accuracy, namely; Average Bias, Mean Square Error (MSE), Root Mean Squared Error (RMSE) and Median absolute percentage error (MdAPE).

Average bias is a conventional metric that has been used in many ways as a measure of accuracy as well as standard deviation which allows the dispersion/spread of the error with respect to the sample mean of the error to be realised. However, simple average bias metric suffers from the fact that negative and positive errors cancel each other and high variance may not be reflected in the metric. Therefore, MSE averages the squared prediction error for all predictions and encapsulates both accuracy and precision. A derivative of MSE, often used, is Root Mean Squared Error (RMSE). MAPE weighs errors with RULs and averages the absolute percentage errors in the multiple predictions. For prediction applications it is important to differentiate between errors observed far away from the EoL and those that are observed close to EoL. Smaller errors are desirable as EoL approaches.

𝑀𝑆𝐸 =1_𝑙∑𝑙𝑖=1∆(𝑦𝑖 − 𝑓𝑖)2 (140)

𝑀𝐴𝑃𝐸 =1_𝑙∑ |100∆(𝑦𝑖−𝑓𝑖)

𝑛𝑦𝑖 |

𝑙

𝑖=1 (141)

Where 𝑦_𝑖 : is the actual RUL value at time point ‘𝑖’ 𝑓𝑖 : is the predicted RUL value at time point ‘𝑖’

173

One of the major downfalls of the above metrics is that they are not designed for applications where RULs are continuously updated as more data is available. It is desirable to have metrics that can characterize improvement in the performance of a prognostic algorithm as time approaches near end-of-life. The next section describes the work of the extensive studies carried out by the NASA IVHM research group to solve this problem.

5.9.2 Prognostic Horizon (PH)

The Prognostic Horizon is defined as the range between the points where the predictions fall under the allowable error bound (α) for the first time and the end-of-life time point. This metric basically shows that the predicted estimates are within specified limits around the actual EoL and may be considered trust worthy within these bounds. While comparing algorithms, an algorithm with longer prediction horizon would be preferred.

𝐻 = 𝐸𝑜𝑃 − 𝑖 (142)

where 𝑖 = 𝑚𝑖𝑛 {𝑗|(𝑗 ∈ 𝑙)⋀(𝑟_∗(1 − 𝛼) ≤ 𝑟𝑙_{(𝑗) ≤ 𝑟}

∗(1 + 𝛼))}

𝑖 : The first time index when predictions satisfy 𝛼-bounds 𝛼 : Accuracy modifier

𝐸𝑜𝐿 : The ground truth end-of-life

ℓ : Set of all RUL estimation point time indexes 𝑙 : Test sample or specimen number

𝑟∗ : Actual RUL

(𝑗) : Predicted RUL at time instance ‘𝑗’ for the test sample number ‘𝑙’ (i.e. can be mean or median of prediction RUL distribution).

For instance, a PH with error bound of α = 20% identifies when a given algorithm starts predicting estimates that are within 20 % of the actual EoL.

5.9.3 α-λ Accuracy

It may be of interest whether the prediction is within a specified accuracy level at a particular time, this metric provides a way to quantify the prediction at certain time instances. For example it may be required that a prediction falls within 20% accuracy (i.e., α=0.2) halfway to failure from the time the first prediction is made (i.e., λ=0.5).

174

λ is the time and may be specified as the percentage of total remaining life from the point the first prediction is made or a given absolute time interval before EoL is reached. For the tests carried out in this thesis, α-λ accuracy is defined as the prediction accuracy to be within α*100% of the actual RUL at specific time instance tλ

expressed as a fraction of time between the point when an algorithm starts predicting and the actual failure. For example,

[1 − 𝛼]𝑟∗(𝑡) ≤ 𝑟𝑙(𝑡𝜆) ≤ [1 + 𝛼]𝑟∗(𝑡) (143)

where α :accuracy modifier λ :time window modifier 𝑡𝜆 = 𝑡𝑝+ 𝜆(𝐸𝑂𝐿 − 𝑡𝑝)

𝜆 : Time window modifier 𝑡_𝑝 : Prediction time

The higher the percentage the better the ability of the algorithm is at prognosis.

5.9.4 Relative Accuracy (RA)

Relative Accuracy is similar to the α-λ accuracy metric. However, instead of finding out whether the predictions fall within a given accuracy levels at a given time instant, the accuracy level is measured. The time instant is again described as a fraction of actual remaining useful life from the point when the first prediction is made. An algorithm with higher relative accuracy is desirable.

𝑅𝐴_𝜆 = 1 − 𝑟∗(𝑡𝜆)−𝑟𝑙(𝑡𝜆)

𝑟∗(𝑡𝜆) (144)

175