Choosing the proper method

The first part of the experiment tests the influence of different methods using multiple feature count on each risk level separately. This experiment was applied using Algorithm 1 in chapter 4, where the different thresholds were not included. The tested methods are Corr, MI and mRMR. All three methods were previously discussed in section 4.3.1.

The results are shown in Figures 5.1, 5.2, 5.3, 5.4, 5.5 and 5.6. The figures included 11 charts, one for each risk level. Each figure presents two curves per method, one showing its accuracy (Acc) and the other its shifted accuracy (S. Acc) giving a total of six curves per figure. For each curve, 20 different runs were performed, one for each feature count shown by the x-axis, resulting in either the Acc or S. Acc results for a specific risk. This means that each point on the curve shows the result of a risk level when running Algorithm 1 using the defined method (identified by its colour) and the number of features defined by the x-axis.

Analysing Figure 5.1a shows that for risk 0 applying feature counts 1 to 5 have not produced a significant number of patients’ records. This improved when feature count exceeded 6. Correlation showed the best accuracy results for this risk level using different feature counts. Shifted accuracy results were almost constant with 7 features or more but a slight degradation can be noticed as feature count reached 18 in the mRMR curve. This chart shows that as more features are used, better classification for the zero risk patients occurs for both Corr and MI.

5.2 Experiment 1: Methods and parameters

(a) Risk 0 results

(b) Risk 1 results

Fig. 5.1 Accuracy and shifted accuracy for suicide risk levels 0 patients (Figure a) and 1 patients (Figure b) using feature counts ranging from 1 to 20. The results are shown for mutual information MI, correlation Corr and minimum redundancy maximum relevance mRMR criteria.

(a) Risk 2 results

(b) Risk 3 results

Fig. 5.2 Accuracy and shifted accuracy for suicide risk levels 2 patients (Figure a) and 3 patients (Figure b) using feature counts ranging from 1 to 20. The results are shown for mutual information MI, correlation Corr and minimum redundancy maximum relevance mRMR criteria.

5.2 Experiment 1: Methods and parameters

(a) Risk 4 results

(b) Risk 5 results

Fig. 5.3 Accuracy and shifted accuracy for suicide risk levels 4 patients (Figure a) and 5 patients (Figure b) using feature counts ranging from 1 to 20. The results are shown for mutual information MI, correlation Corr and minimum redundancy maximum relevance mRMR criteria.

(a) Risk 6 results

(b) Risk 7 results

Fig. 5.4 Accuracy and shifted accuracy for suicide risk levels 6 patients (Figure a) and 7 patients (Figure b) using feature counts ranging from 1 to 20. The results are shown for mutual information MI, correlation Corr and minimum redundancy maximum relevance mRMR criteria.

5.2 Experiment 1: Methods and parameters

(a) Risk 8 results

(b) Risk 9 results

Fig. 5.5 Accuracy and shifted accuracy for suicide risk levels 8 patients (Figure a) and 9 patients (Figure b) using feature counts ranging from 1 to 20. The results are shown for mutual information MI, correlation Corr and minimum redundancy maximum relevance mRMR criteria.

Fig. 5.6 Accuracy and shifted accuracy for risk 10 using feature counts ranging from 1 to 20.

The results are shown for mutual information MI, correlation Corr and minimum redundancy maximum relevance mRMR criteria.

Figure 5.1b shows that using a single feature, the algorithm was able to classify risk 1 patients correctly with a high accuracy. There was a decrease in the accuracy as more features were added. On the other hand, the S.Acc curves were almost constant along most feature counts. This shows that some of the records initially classified as risk 1 (whether they were actually risk 1 or not) are now classified as risk 0. Thus showing the improvement in Figure 5.1a versus the degradation in Figure 5.1b. The shifted accuracy curves were almost constant with more than 6 features indicating that any alterations are within the 1 class tolerance.

Figures 5.2a to 5.5a show that risks ranging between 2 and 8 were almost constant using the three methods when 7 features or more were chosen. The difference between the three methods were minimal when it comes to accuracy and shifted accuracy. For the 7 figures, using a single feature, correlation has produced the best accuracy results. Thus Corr was stronger than MI and mRMR in identifying a stronger initial feature in the feature set.

5.2 Experiment 1: Methods and parameters

Risk 9 and 10 results are shown in Figures 5.5b and 5.6. The two curves show an obvious variation for each addition of a feature. The reason behind this is that risk 9 population is represented by 353 records and risk 10 by 99 records resulting in a significant change in the chart with minor changes in the results. This was not obvious in lower risk levels because they have much higher records count, such as risk 0 having 13627 records and risk 1 having 15358 records. Although the higher risk patients are a much smaller count that the lower risk, there is an increased importance of getting their assessment correct.

To sum up, 0 risk might need more variables to be accurately classified but fewer variables are better for minimising the missing information, increasing effectiveness of regression and maximising speed. Correlation showed a small improvement in the accuracy in risk levels 0 and 10. It chooses the features with highest linearity with the risk level, which would explain why using a single feature in Corr curves was better than MI and mRMR. Another advantage of correlation is that it produced the best results given feature count lower than 10, which could be the case for some patients whom did not provide enough information.

Finally correlation managed to have the most stable results through the different feature counts therefore it is the method to be used in the rest of the experiments.