Evaluation Result

Experiment 1: No Signal Anomaly Injected

In Experiment 1, only the No Signal anomaly is injected into the source signals.

The overall results achieved by AID and AWDD are shown in Table 3.2. Overall, there are 4047 detections have been performed with 200 injected anomalies. When No Signal anomaly is injected, AID can improve the TP and TN rate by 66.67% and 5% respectively compared to AWDD. AID also reduces the FP and FN rate by

TP TN FP FN Rec

AWDD 33.33% 93.01% 6.81% 0.39%

-AID 100% 98.01% 0.52% 0% 100%

Improvement 66.67% 5% 6.29% 0.39%

-p-Value 6.9e-09 0.004 0.003 1.1e-06

-Table 3.2: Overall result achieved for AWDD and AID when only the Blank Signal anomaly is injected

6.29% and 0.39% respectively. It can correctly recognise all injected No Signal anomalies. The calculated p-Values for each rate are all well below 0.05. It indi-cates that the improvement achieved by AID over AWDD is statistically significant.

Figure 3.16 shows the boxplot of TP, TN, FP, FN when the Blank Signal anomaly is injected into the source signals. It can be seen from the plots that the AID can achieve similar results for the true-positive and false-negative rates across 20 sub-jects. That means for the Blank Signal anomaly, AID can detect all possible Blank Signal anomaly without any omission. For the true-negative and false-positive rate, if not considering the statistical outlier, the variation between difference subjects are also less than AWDD.

Experiment 2: Inverted Signal Anomaly Injected

In Experiment 2, only the Inverted Signal anomaly is injected into the source sig-nals. The overall results achieved by AID and AWDD are shown in Table 3.3. In total, 4019 detections have been performed with 271 Inverted Signal anomalies being injected. Overall, AID has managed 80.07% TP, 12.86% TN, 6.39% FP, and 7.49% FN improvement over AWDD. Also, it managed to recognise 96.31% of all the injected anomalies. By carrying out the Wilcoxon Rank Sum Test, the p-Values of each rates are all below 0.05. Subsequently, it can be said that the detection accuracy improvements over AWDD are significant.

The boxplot for the results in Experiment 2 is shown in Figure 3.17. Except the statistical outliers, the AID has achieved better TP and FN with less variation be-tween different test subjects than the AWDD. For those non-outliers TP and FN, the AID can achieve nearly 100% and nearly 0% respectively. On the other hand,

Figure 3.16: Boxplot of TP, TN, FP, FN showing the maximum, minimum, median upper quartile, lower quartile, and interquartile range with the statistical outliers (plus sign) when Blank Signal anomaly is injected to the source signals

TP TN FP FN Rec

AWDD 18.45% 84.73% 7.02% 7.75%

-AID 98.52% 97.59% 0.66% 0.26% 96.31%

Improvement 80.07% 12.86% 6.39% 7.49% -p-Value 2.3e-08 2.5e-04 0.002 2.8e-06

-Table 3.3: Overall result achieved for AWDD and AID when only the Inverted Signal anomaly is injected

the TP and FN achieved by AWDD have much wider variation across different test subjects. For the TN and FP, the AWDD achieved slightly less variation across dif-ferent test subjects than the AID. However, the result from AID is generally better than the AWDD despite the slightly wider range.. For example, the minimum value from AID is almost equal to the upper quartile achieved by AWDD.

Experiment 3: Noisy Signal Anomaly Injected

In Experiment 3, the Noisy Signal anomaly is injected into the source signals. The noise strength is varied between −10dB and 0dB. The overall results achieved by AID and AWDD are shown in Table 3.4. In general, the results show that when the noise is much stronger than the signal, AID can achieve much better detection accuracy than AWDD. When the noise strength decreased, the TP rate decreases an FN rate increases. More specific results are presented below for each noise strength level.

When the noise strength is at −10 dB, the AID can improve the detection accuracy by 56.03%, 3.78%, 3.67%, and 1.55% respectively for TP, TN, FP, and FN. The Wilcoxon Rank Sum Test returns a smaller than 0.05 p-Value for each rate. That means the improvements achieved by the AID are all statistically significant. From the boxplot of these results shown in Figure 3.18, it can be seen that the results from AID are better than AWDD.

When the noise strength is −6 dB, the AID can also manage an improvement of 52.3%, 4.08%, 4.07%, and 1.42% respectively for the TP, TN, FP, and FN. The p-Values from Wilcoxon Rank Sum Test also show that the improvement for each rate is significant. The boxplot of the results is shown in Figure 3.19. Similar observation

Figure 3.17: Boxplot of TP, TN, FP, FN showing the maximum, minimum, median upper quartile, lower quartile, and interquartile range with the statistical outliers (plus sign) when Inverted Signal anomaly is injected to the source signals

Noise Strength

TP TN FP FN Rec

AWDD 31.91% 91.18% 6.25% 2.32%

--10dB AID 87.94% 94.96% 2.58% 0.77% 99.19%

Improvement 56.03% 3.78% 3.67% 1.55% -p-Value 5.9e-08 0.002 0.001 2.5e-06

-AWDD 32.33% 90.76% 6.66% 2.33%

--6dB AID 84.21% 94.84% 2.59% 0.9% 99.11%

Improvement 52.3% 4.08% 4.07% 1.42% -p-Value 5.9e-08 0.002 0.002 1.1e-04

-AWDD 29.81% 90.11% 7.42% 2.44%

-0dB AID 51.92% 92.96% 2.64% 2.83% 100%

Improvement 22.11% 2.85% 4.78% -0.39% -p-Value 5.3e-05 0.003 0.001 0.67

-Table 3.4: Overall result achieved for AWDD and AID when only the Noisy Signal anomaly is injected with strength between −10dB to 0dB

to the results from the noise strength of −10 dB can be seen from Figure 3.19.

The AID generally perform better than AWDD with less variation across different subjects.

When the noise strength reduced to 0 dB, the improvements start to decrease.

The AID achieved 22.11%, 2.85%, and 4.78% respectively for TP, TN, and FP. How-ever, the FN is decreased by −0.39% compared to AWDD. Through the Wilcoxon Rank Sum Test, the p-Values of 5.3249e−05, 0.0029, and 0.0012 for TP, TN, and FP respectively show that the improvements for these three rates are significant. How-ever, the p-Value for the FN is 0.6749, which means that the performance decrease compared to AWDD is not statistically significantly.