Summary of the experiments

2.5.6.1 Discrepancies in data labelling

Results that were achieved with two types of automatic labelling procedures were very different. For most of the point systems the classification rates were over 90%, when the labelling of movements was done using alarm dates. In comparison, classification rates of only around 60% were achieved, when the labelling was done using failure dates. This difference in results suggested that in some instances movements that were very similar were labelled differently, when the labelling was done according to failure dates. A detailed analysis of data labelling results with both labelling techniques was performed.

118 It was found that only significant changes in the value of current raised an alarm and thus, when the data was labelled according to alarm dates, only the movements which were very different from the normal behaviour, i.e. usually the duration of POE operation was significantly longer, were labelled as faulty. Thus it was concluded that data labelling using alarm dates should not be performed, since it does not allow identifying more subtle changes in the profile of current trends, necessary for incipient fault detection.

From a closer look at failure dates, it was found that a lot of movements that represented the already rectified state of the railway point system (therefore, they were similar to the good state) using an automatic labelling approach were still labelled as faulty. Moreover, the failure dates recorded in the database were usually later than the actual failure date, i.e. the failure date was recorded after the engineers found the failure cause, but the movements that were performed prior to the recorded date were already similar to the faulty state. Such discrepancies in the labelling technique using failure dates resulted in poor classification rates and none of the combinations of pre-processing options or different similarity measures used in the analysis helped to achieve better results.

Due to the discrepancies found in both labelling options, the automatic labelling of the data could not be used. In the further study, the movements of one point system were therefore labelled manually, by considering movements that looked abnormal and that looked fault-free. After the manual editing of the discrepancies of the labelling it was shown that the OCSVM model can perform well, as shown in section 2.5.4 and published in (Vileiniskis et al., 2013, Vileiniskis et al., 2015). Ideally, specific RPS engineering knowledge is needed for such data cleansing process, and it would be impractical to relabel a large number of movements.

2.5.6.2 Training and testing procedures

Two different training and testing techniques were used to evaluate the performance of the OCSVM classifiers. The 5-fold cross-validation showed that the results obtained with this technique are more reliable than the ones obtained with the proportional training. It was shown that the best classifier obtained with the proportional training method achieved good classification accuracies by allowing big misclassification rates of the training dataset: a lot of movements representing the fault-free condition of RPS were classified as faulty in the training dataset. Such an approach would lead to a situation when newly recorded good movements would be classified as abnormal if they were similar to the ones used in the training phase. Thus the proportional training and testing technique should not be used to obtain the OCSVM model. However, in order to obtain the OCSVM models with the 5-fold cross-validation method more computational time is required, since the validation needs to be performed repetitively, as discussed in section 2.4.5.1.

If only a single iteration of 5-fold cross-validation is performed to reduce the time, the results might be biased, since the estimates of the classification accuracy depend on the random division of data.

2.5.6.3 Pre-processing techniques

The influence of different pre-processing techniques on classification accuracy has been considered. The use of estimates of the derivatives, in addition to absolute values, for the alignment of data time series did not increase the classification

119 accuracies in most of the cases, despite the evidence provided in section 2.4.3.5, when it overcomes some weaknesses of the alignment using only absolute values of current measurements.

The classification accuracy usually dropped after downsampling of the data. However, the drop in the accuracy was only around 1% and, as it was shown in section 2.5.5.1, the downsampling can decrease the computational time significantly. If such loss of accuracy is acceptable, the data could be downsampled when the method is applied in practice. Moreover, in some cases the downsampling of data even increased the classification accuracy. Thus the down-sampling might be a good step of the analysis to implement if this approach was to be considered for the online fault detection. The smoothing parameter had some influence on OCSVM performance, when two time series were aligned by the estimates of the derivatives. Almost in all cases, the accuracy of the model increased if the data was smoothed, when the alignment of time series was based on the estimates of the derivatives.

The effect of rescaling the data was not considered, since the rescaling was only done as an intermediate step to compare the Euclidean distance and elastic similarity measures on identical data.

2.5.6.4 Different ways of using similarity measures

Two approaches of using time series similarity measures were tested in this study. The first approach was to use a similarity measure as a kernel distance function. The second approach was to calculate the similarity measures with several previous movements and then use them as a feature input to OCSVM. Initially, the latter approach was considered to be more useful, since by comparing consecutive movements the model could update the knowledge on the changing behaviour of good movements and in this way seasonal effects could be included in the model. Different numbers of consecutive movements for comparison were considered for the analysis and the comparison of one movement with 5 previous movements proved to be the best from the ones tested. However, using this approach to classify the movement did not provide good classification accuracy. To improve the accuracy of this approach, some sort of weighting of the previous movements could be considered or some additional information used in the model, such as the increase/decrease of values of the measurements of current for every movement, the information about the class of the previous movement etc.