In this section, the results obtained comparing the operation of the seven proposed region-based representation techniques are considered. Recall that the objective was to identify the most appropriate region representation in the context of classification performance based on the decomposition process. As in the previous section, results with respect to both classification performance and significance testing are presented (Subsections 6.3.1 and 6.3.2 respectively). For the performance evaluation, four levels of decomposition were again considered,L= 3,L= 4,L= 5 andL= 6, together with the ED critical function and overlapping decomposition because the experiments presented in the previous section (and others not specifically reported in this thesis) indicated that these later two worked well. Although in the previous section it was demonstrated that L = 5 tended to produced a best performance, the reason for using a range of levels with respect to the experiments reported here was because it was conjectured that the operation of the different region representation techniques might be affected by the nature of the level of decomposition used. As previously, with respect to the other two stages not considered in this section (feature vector generation and classification), dimensionality reduction using PCA and SVM classification were adapted.
6.3.1 Classifier Performance in the Context of Region Representation In this section, we present the results obtained to compare the operation of the seven proposed region-based representation techniques. The results are presented in Table 6.7. Out of the seven representation techniques considered, the results presented in the table indicate that the HOG representation method produced the best results with a best average AUC of 0.96. From the table, the best accuracy and AUC value were produced using the HOG representation with L = 4 (97.14% and 0.97 respectively). Overall, use of the HOG representation produced the average best results with respect
to the all the metrics considered.
Considering the three statistical representations (FOR, VCM and VRLM), they produced a similar performance to each other. With respect to the FOR representation, it is interesting to note that when a higher level of decomposition was used a better performance was obtained. The AUC with respect to FOR representation whenL= 3 was 0.83 while forL= 6 the AUC was 0.92. It was conjectured that this was because FOR can produce a better performance with smaller regions than with larger regions. With respect to the histogram based representation techniques (HOG, LBP, HOG- LBP and LPQ) the best performing method as already noted above was HOG. The lowest recorded AUC using the HOG representation was 0.96 while the lowest recorded AUC for LBP, HOG-LBP and LPQ were 0.8, 0.84, and 0.73 respectively. When compar- ing the LBP and HOG-LBP results, HOG-LBP produced better average results where the average AUC for HOG-LBP is 0.85 while the average AUC for LBP is 0.82. It was conjectured that this was because of the positive effect of combing the HOG and LBP representations. With respect to the LPQ representation, the recorded performance was worse than that for the LBP and HOG-LBP representation. It was conjectured that this was because LPQ generated less effective histograms when a higher level of decomposition is used than in the case of LBP and HOG-LBP. The AUC for LPQ with L = 3 was 0.94 while with L= 6 it was 0.75. This is because LPQ is based on the Fourier Transform which is a more effective representation when larger regions are used, but the effectiveness is decreased when smaller regions are used. However, when using the HOG representation the changes in the intensity values of the regions (sub- volumes) in different directions are considered. The HOG representation improved the discriminative power when applied to homogeneous and small regions as in the case of the proposed decomposition. The reason is that the HOG representation uses the gradient (change) in the intensity value even when the region is small; while other rep- resentation techniques, such LPQ which relies on the frequency of the neighbours, tend to work better with larger regions.
6.3.2 Region Representations Significance Testing
The objective of this subsection is to consider whether the result presented above, that HOG representation is the most effective, is indeed statistically significant. ANOVA was conducted using seven groups, where each group represents the AUC results for a specific region representation. The results from the ANOVA, as shown in Table 6.8, demonstrated that there was a statistical difference in the effectiveness of the seven representations (from Table 6.8 it can be seen that the p-value was 3.4274e-119 which is less than 0.05). From the table, it can also be seen that the difference between the groups and within the groups were very similar (Between-GroupsSS = 2.5 and ErrorSS = 2.2).
Table 6.7: Classifier performance results using overlapping decomposition, an ED critical function, dimensionality reduction using PCA and SVM classification in the context of region representation methods (Stage 2) using: (i)) a range of decomposition levels (L), (ii) the seven region-based representation techniques.
Method L Acc Sen Spec PPV NPV EER AUC
FOR 3 82.86% 81.43% 84.29% 83.82% 81.94% 0.16 0.83 4 84.29% 83.82% 84.72% 83.82% 84.72% 0.16 0.84 5 95.00% 94.20% 95.77% 95.59% 94.44% 0.04 0.95 6 92.14% 92.54% 91.78% 91.18% 93.06% 0.09 0.92 Ave. 88.57% 87.99% 89.14% 88.60% 88.54% 0.11 0.88 VCM 3 87.86% 86.96% 88.73% 88.24% 87.50% 0.12 0.88 4 90.00% 89.71% 90.28% 89.71% 90.28% 0.10 0.9 5 85.71% 84.29% 87.14% 86.76% 84.72% 0.14 0.86 6 85.71% 84.29% 87.14% 86.76% 84.72% 0.14 0.86 Ave. 87.32% 86.31% 83.32% 87.84% 86.80% 0.12 0.87 VRLM 3 85.71% 86.36% 85.14% 83.82% 87.50% 0.16 0.86 4 84.29% 84.85% 83.78% 82.35% 86.11% 0.17 0.84 5 85.71% 86.36% 85.14% 83.82% 87.50% 0.16 0.86 6 85.71% 86.36% 85.14% 83.82% 87.50% 0.16 0.86 Ave. 85.35% 85.98% 84.80% 83.45% 87.15% 0.16 0.85 HOG 3 97.86% 98.51% 97.26% 97.06% 98.61% 0.03 0.98 4 96.43% 97.01% 95.89% 95.59% 97.22% 0.04 0.96 5 95.71% 95.59% 95.83% 95.59% 95.83% 0.04 0.96 6 95.71% 96.97% 94.59% 94.12% 97.22% 0.06 0.96 Ave. 96.42% 97.02% 95.89% 95.59% 97.22% 0.04 0.96 LBP 3 86.43% 83.56% 89.55% 89.71% 83.33% 0.11 0.87 4 82.86% 80.56% 85.29% 85.29% 80.56% 0.15 0.83 5 80.00% 77.03% 83.33% 83.82% 76.39% 0.17 0.8 6 80.00% 77.03% 83.33% 83.82% 76.39% 0.17 0.8 Ave. 82.32% 79.54% 85.37% 85.66% 79.16% 0.15 0.82 HOG-LBP 3 87.86% 94.74% 83.13% 79.41% 95.83% 0.18 0.88 4 84.29% 91.07% 79.76% 75.00% 93.06% 0.21 0.84 5 85.00% 91.23% 80.72% 76.47% 93.06% 0.20 0.85 6 85.00% 91.23% 80.72% 76.47% 93.06% 0.20 0.85 Ave. 85.53% 92.06% 81.08% 76.83% 93.75% 0.19 0.85 LPQ 3 94.29% 94.12% 94.44% 94.12% 94.44% 0.06 0.94 4 72.86% 66.67% 84.00% 88.24% 58.33% 0.17 0.73 5 75.00% 68.97% 84.91% 88.24% 62.50% 0.16 0.75 6 75.00% 68.97% 84.91% 88.24% 62.50% 0.16 0.75 Ave. 79.28% 74.60% 87.06% 89.71% 69.44% 0.13 0.79
Figure 6.4(a) shows the significant differences and confidence intervals between the region-based representation methods. From the figure, it can be seen that the operation of three of the methods was not statistically different from each other: LBP-HOG, VCM and VRLM - but they are different from the rest of the representation methods. A statistical difference in operation can be noted between FOR, LPQ, LBP and HOG. The results from the confidence interval diagram (Figure 6.4(b)) indicate that the LPQ results had the lowest median AUC value (0.7), while the HOG representation produced the highest median AUC value (0.9). From Figure 6.4(b), it can be seen that the HOG representation has the narrowest confidence interval, which is indicative of the effectiveness of the representation. Overall, the statistical significance result confirms the results presented in the previous subsection that the HOG representation is the most appropriate with respect to the proposed 3D image classification based on spatial decomposition and region-based representation.
Table 6.8: Comparison of region-based representation methods.
Source SS df MS F p-value
Between-Groups 2.5743 6 0.4290 140.7131 3.4274e-119 Error 2.2837 749 0.0030
Total 4.8580 755
(a) Significance differences (b) Confidence intervals
Figure 6.4: Significance differences and Confidence intervals for comparing representation techniques.