Message Integrity and Authentication

Model W=0 W=2 W=5

SVM 66.0% (p=0.0014) 66.3% (p=0.1271) 69.3% (p=0.1665) CRF 55.3% (p=0.0007) 53.2% (p=0.013) 57.4% (p=0.0222) HMM 69.6% (p=0.1704) 69.5% (p=0.0876) 70.4% (p=0.3361) HMM-C 69.8% (p=0.0577) 66.9% (p=0.0491) 66.1% (p=0.0194)

FHCRF 75.9% 74.0% 75.1%

Model W=10 FFT (W=16)

SVM 74.8% (p=0.4369) 73.6% (p=0.5819) CRF 56.3% (p=0.1156) 69.8% (p=0.0494) HMM 73.7% (p=0.7358) 68.4% (p=0.4378) HMM-C 70.8% (p=0.6518) 70.4% (p=0.4429)

FHCRF 72.4% 73.4%

Table 3.2: Accuracy at equal error rate for different window sizes on the MelHead dataset. The best model is FHCRF with window size of 0. Based on paired t-tests between FHCRF and each other model, p-values that are statistically significant are printed in italic.

subsequences varied between 30-60 frames. The resulting training set had the same number of gesture and non-gesture subsequences which is equal to the number of ground truth gestures in the original training set.

Each experiment was also repeated with different input feature window sizes. A window size equal to zero (W=0) means that only the velocity at the current frame was used to create the input feature. A window size of two (W=2) means that the input feature vector at each frame is a concatenation of the velocities from five frames:

the current frame, the two preceding frames, and the two future frames.

0.5 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9

0 1 2 3 4 5 6 7 8 9 10

Window size

Accuracy at equal error rate

SVM CRF

FHCRF HMM

HMM-C

Figure 3-14: Accuracy at equal error rates as a function of the window size, for dataset MelHead.

MelHead

In the MelHead dataset, the human participants were standing in front of the robot, and were able to move freely about their environment, making this dataset particu-larly challenging.

Figure 3-13 displays the ROC curves of the four models after training with different window sizes. The last graph of Figure 3-13 shows the performance of all four models when trained on input vectors created by applying an FFT filter over a window of size 32 (W=16). Table 3.2 provides a summary of Figure 3-13 and displays the recognition rate at which both the true positive rate and the true negative rate are equal. This recognition rate (true positive rate) is known as equal error rate (EER). A paired t-test was performed between the FHCRF model and all three other models. The p-values are shown in Table 3.2 next to the equal error rates for the SVM, CRF and HMM models. The difference between FHCRF and SVM is statistically significant at W=0 but is not statistically significant for W=10 which is when SVM accuracy is slightly higher then FHCRF. Figure 3-14 shows a plot of the EER values displayed

0 0.2 0.4 0.6 0.8 1 0

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

False positive rate

True positive rate

SVM CRF FHCRF HMM

0 0.2 0.4 0.6 0.8 1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

False positive rate

True positive rate

SVM CRF FHCRF HMM

(a) W=0 (b) W=2

0 0.2 0.4 0.6 0.8 1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

False positive rate

True positive rate

SVM CRF FHCRF HMM

0 0.2 0.4 0.6 0.8 1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

False positive rate

True positive rate

SVM CRF FHCRF HMM

(c) W=5 (d) W=10

Figure 3-15: ROC curves for the WidgetsHead dataset. Each graph represents a different window size: (a) W=0, (b) W=2, (c) W=5 and (d) W=10.

in Table 3.2 as a function of the window size for each model.

In Table 3.2, the largest EER for all the models and all window sizes is the FHCRF model with a window size of 0. As expected the FHCRF model performs quite well even when given a small window size. This is a direct result of the FHCRF model’s capability to simultaneously model the intrinsic and extrinsic dynamics of the input signal. With only the velocity of the current frame as input, the FHCRF is able to outperform other models with input vectors that are 10 to 20 times larger. This makes FHCRF more suitable for online applications where knowledge for future frames is not available.

Model W=0 W=2 W=5

SVM 53.1% (p=0.0109) 65.5% (p=0.0107) 74.7% (p=0.129) CRF 67.0% (p=0.1161) 66.8% (p=0.0076) 68.1% (p=0.083) HMM 64.5% (p=0.0546) 63.2% (p=0.0071) 69.03% (p=0.0016) HMM-C 68.9% (p=0.0132) 72.1% (p=0.0716) 76.9% (p=0.4142)

FHCRF 73.7% 80.1% 79.6%

Model W=10 W=15

SVM 77.5% (p=0.4953) 77.8% (p=0.0484) CRF 70.07% (p=0.1338) 71.7% (p=0.6592) HMM 71.3% (p=0.3359) 69.6% (p=0.6616) HMM-C 79.2% (p=0.2751) 73.8% (p=0.9377)

FHCRF 75.2% 69.3%

Table 3.3: Accuracy at equal error rate for different window sizes on theWidhetsHead dataset. The best model is FHCRF with window size of 2. Based on paired t-tests between FHCRF and each other model, p-values that are statistically significant are printed in italic.

WidgetsHead

Figure 3-15 shows the ROC curves of the four models after training with different window sizes. Table 3.3 displays the equal error rates (EER) for each ROC curve in Figure 3-15. A paired t-test was performed between the FHCRF model and all 3 other models. The p-values are shown in Table 3.3 next to the equal error rates for the SVM, CRF and HMM models. For a window size of 2, the difference between the accuracy of FHCRF and every other model is statistically significant. Figure 3-16 plots the EER values of Table 3.3 as a function of the window size for each model.

By looking at Figure 3-16, we can see that for this dataset, the FHCRF works best with a small window size between 2 and 5. Again the FHCRF outperforms the other models, even when they are using a larger window size.

AvatarEye

Figure 3-17 shows the ROC curves of the four models after training with different window sizes. Table 3.4 displays the equal error rates (EER) for each ROC curve in Figure 3-17. A paired t-test was performed between the FHCRF model and all 3 other models. The p-values are shown in Table 3.4 next to the equal error rates for

0.5 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9

0 2 4 6 8 10 12 14

Window size

Equalerrorrate

SVM CRF

FHCRF HMM

HMM-C

Figure 3-16: Accuracy at equal error rates as a function of the window size, for dataset WidgetsHead.

the SVM, CRF and HMM models. We can see in Table 3.4 the FHCRF outperforms all three other models for both window sizes. Figure 3-18 shows a plot of the EER values displayed in Table 3.4 as a function of the window size for each model. Note that a problem with memory allocation made it impossible to run the experiment with W=10.

The AvatarEye dataset had only 6 participants and 77 eye gestures. We can see in Figure 3-18 how this small dataset affects the FHCRF model when the window size increases. This effect was not as prominent for larger datasets, as observed in Figures 3-16 and 3-14. Even with this small dataset, FHCRF outperforms the three other models with a maximum accuracy of 85.1% for a window size of 0.