Decomposition Significance Testing

Other than that, the use of critical function was beneficial to inspection of the re- sults presented in Tables 6.1 and 6.2, using the HOG representation, PCA and SVM classifier generation, suggesting that there is no significant difference with respect to the operation of the different decomposition techniques considered although some pro- duced better results than the others. This subsection reports on the ANOVA that was applied to all the generated results (including results using all other combination of the proposed techniques not included in Section 6.2.1) to determine if there was any statistically significant difference in the operation between the different techniques considered in this section.

As already noted in the previous subsection, 64 different combinations of techniques were considered: 4 (levels)×2 (standard and overlapping decomposition)×8 (critical functions) = 64 combinations. ANOVA was applied to check if there was any significant difference between the operation of these 64 different techniques. The analysis was conducted with respect to AUC values.

In the context of comparing the use of the critical functions against each other and without a critical function (0NCF), Table 6.4 shows the ANOVA table (more detail about ANOVA was presented in Subsection 2.8.2). The ANOVA table has six columns:

1. Source: The source of variation in terms of: (i) the differences between groups (Between-Groups), and (ii) the change within each group (Error).

2. SS: The sum of squares value for each source.

3. df: The degrees of freedom for each source.

4. MS: The mean square value associated with each source.

5. F: The F ratio of the mean square values.

6. p-value: The p-value obtained by the cumulative distribution function of F.

From the table, it can be noted that the calculated p-value (2.8315e-20) is less than 0.05, indicating that there is indeed a statistical significance between the results obtained. From the table, ErrorSS = 14.02, indicating that there are a wide range of values for each critical function but with a smaller range in terms of the difference between critical functions (Between-GroupsSS = 0.7024).

Figure 6.1(a) shows the significant difference between the results of the critical functions where the x-axis represents the AUC results and the y-axis lists the methods. The lines represent the “comparison interval” around the AUC mean of each group (critical function); the dot in the middle of each line makes the mean of each group. When the lines of two methods do not overlap, the operation (in terms of AUC) between the two methods can be said to be significantly different.

Figure 6.1(b) shows boxplots of the confidence intervals for each method. In the diagram, the x-axis lists the methods and the y-axis the AUC value. The red line in each box represents the median value of the AUC results while the top and bottom of the box represent the 75% and 25% quartile of the AUC results. The notch in each box represents the 95% confidence intervals of the measured median value. The whiskers mark the highest/lowest AUC values of each group of AUC results that are within 1.5 times the interquartile range of the box edges. The red plus signs represent the outliers beyond the data range. When the notches of two methods do not overlap, the medians of them will be significantly different at the 0.05 significance level. From both figures it can be observed that:

1. The results obtained using 0NCF were statistically different from the results ob- tained using critical functions (except for the DTW and GLCM critical functions) where the range of AUC results for 0NCF were less than the results of the pro- posed critical functions, a lower mean (0.86) than the critical functions and a lower median (0.87) than the rest of the critical functions. This result supports the conjecture that the use of critical functions for measuring the regional homo- geneity produces a more concise and descriptive representation.

2. The AUC results recorded using the ED critical function were statistically dif- ferent from those recorded using 0NCF, DTW, GLCM, KCC and KLD because the range of the confidence interval for the ED result was higher than the other critical functions.

3. The LCS results were not statistically different from AIV, ED and KCC but were statistically different from the other critical functions.

4. The AUC results obtained using GLCM were statistically different from AIV, ED and LCS.

5. The results obtained using KCC were statistically different from ED, DTW and 0NCF.

6. The KLD results were statistically different from the 0NCF, ED and LCS results.

In the context of level of decomposition, the ANOVA table presented in Table 6.5 shows that there was a statistical significance between the results obtained for the

Table 6.4: ANOVA table for comparing critical functions

Source SS df MS F p-value

Between-Groups 0.7024 7 0.1003 15.7415 2.8315e-20 Error 14.0246 2200 0.0064

Total 14.7271 2207

(a) Significance differences (b) Confidence intervals

Figure 6.1: Significance differences and confidence intervals for comparing critical functions

different levels of decomposition in the context of the recorded AUC values (the p- value was 6.9209e-04). Note that in the table the groups are the techniques used with respect to each level. From the table, it can be seen that the range of values was wide (ErrorSS=14.69), while the difference in values between the groups was slightly smaller (Between-GroupsSS=0.1). The significant difference and confidence interval diagrams shown in Figure 6.2(a) illustrate the significant differences between the levels of decomposition. From the figure, it can be observed that the AUC results obtained using L = 3 were statistically different from those obtained using L = 5 and L = 6; while the AUC results obtained using L = 4 were not statistically different from the others because the range of recorded AUC results overlaps with the ranges of the results associated with the other groups. The confidence intervals shown in Figure 6.2(b) reveal that there was a similarity between the results with slight improvement with respect to L = 5, which had a slightly higher median AUC result of 0.91, thus contradicting the previously conducted but less sophisticated, analysis presented in Table 6.3.

In terms of comparing standard and overlapping decomposition, Table 6.6 gives the ANOVA table, where the p-value of 1.7955e-25 indicates that there was a statistical difference between the standard and overlapping decomposition, as also demonstrated in Figure 6.3. From the table, it can be seen that there was similarity between the results of the standard and overlapping decomposition as Between-GroupsSS=0.6906

Table 6.5: ANOVA table for comparing levels of decomposition

Source SS df MS F p-value

Between-Groups 0.1099 3 0.0366 5.7014 6.9209e-04 Error 14.6999 2288 0.0064

Total 14.8098 2291

(a) Significance differences (b) Confidence intervals

Figure 6.2: Significance differences and confidence intervals for comparing levels of decompo- sition

which is slightly low. The table revels also that the differences within each decomposi- tion method with ErrorSS=14.1132 were higher than between groups, indicating that there were large differences between the classification results. The confidence interval, shown in Figure 6.3, indicates that the overlapping decomposition outperformed the standard decomposition in a statistically significant manner, thus confirming the earlier conclusions presented in Subsection 6.2.1.

Table 6.6: Comparison of decomposition techniques

Source SS df MS F p-value

Between-Groups 0.6906 1 0.6906 111.4646 1.7955e-25 Error 14.1132 2278 0.0062

Total 14.8038 2279

Thus in conclusion we can consider that the best critical function to use is either the ED or LCS critical function together with L = 5 and overlapping decomposition. Note that both ED and LCS are amongst the critical functions proposed by the author.

Figure 6.3: Confidence intervals for overlapping with standard decomposition.