4.3 Methodology of the User Study
4.4.4 Clearing Up the Misconceptions
Overall 72.8% of the responses pertaining to the misconceptions were correct in the pre-treatment ques- tionnaire. However, as becomes apparent from figure 4.15, most of the misconceptions were prevalent in the sample of SME employees during the first phase of the study (i.e. before the treatment). Some of the misconceptions appeared in the majority of the participants. A visual inspection of the data indicates that misconceptions M1, M2, M3, M4, M10, and M11 show low portions of correct responses and seem to be especially prevalent in the sample before the intervention. In contrast, ceiling effects (i.e. no significant improvement possible) appear for the misconceptions M5, M9, M13, M15, M16, M19, and M20.
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% M1 M2 M3 M4 M5 M6 M7 M8 M9^ M10 M11 M12 M13^ M14 M15^ M16^ M17 M18 M19^ M20 M21 M22 M23 correct incorrect
Figure 4.15: The correct and incorrect responses for each of the misconceptions before reading through the awareness material. The misconceptions affected by ceiling effects are marked with a ‘^’.
4 Development and Evaluation of Effective Password Security Awareness Materials 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% M1* M2* M3* M4* M5* M6* M7* M8* M9^ M10* M11* M12 M13^ M14 M15^ M16^ M17* M18* M19^ M20 M21* M22* M23* pre-correct/post-correct pre-incorrect/post-correct pre-incorrect/post-incorrect pre-correct/post-incorrect
Figure 4.16: The correct and incorrect responses for each of the misconceptions before and after reading through the aware- ness material. Each colour represents the respective number of participants with “pre-treatment/post-treatment” responses, e.g. “pre-incorrect/post-correct” is the number of participants having responded incorrectly in the pre- treatment questionnaire and correctly in the post-treatment questionnaire. The misconceptions with significant improvements are marked with a ‘*’. The misconceptions affected by ceiling effects are marked with a ‘^’.
Pre-Treatment to Post-Treatment Analysis
After the intervention, participants performed better regarding the prevalence of misconceptions. The overall portion of correct responses increases from 72.8% in the pre-treatment questionnaires to 89.3% in the post- treatment questionnaires.
Analysis of H3a. A Wilcoxon signed rank test with continuity correction shows a significantly higher number of correct responses per participant in the post-treatment questionnaire than in the pre-treatment question- naire (V = 35, p < .001). An effect size of r = 0.562 indicates a large effect. Therefore, the results of this study support H3a.
Further Findings. Figure 4.16 shows for each misconception the individual differences in correct and incor- rect responses between the pre-treatment questionnaire and the post-treatment questionnaire. The individual differences were evaluated with McNemar’s test.
On the one hand, all of the misconceptions which stood out with high numbers of incorrect responses before the treatment show a significant improvement: M1 (χ2(1)= 41.02, p < .001), M2 (χ2(1)= 28.27, p < .001), M3 (χ2(1) = 38.03, p < .001), M4 (χ2(1) = 39.02, p < .001), M10 (χ2(1) = 41.89, p < .001), and M11 (χ2(1)= 15.72, p < .001). Yet, it is of note that despite showing a significant improvement, M2 still exhibits 50.0% incorrect answers in the post-treatment questionnaire. The misconceptions with higher numbers of correct pre-treatment responses showing significant improvements after the treatment are M6 (χ2(1)= 15.43,
p < .001), M7 (χ2(1) = 6.75, p = .009), M8 (χ2(1) = 10.03, p = .002), M17 (χ2(1) = 5.88, p < .015), M18
(χ2(1)= 11.08, p < .001), M21 (χ2(1)= 8.10, p = .004), M22 (χ2(1)= 5.82, p = .016),and M23 (χ2(1)= 9.09, p= .003).
On the other hand, despite the decrease of the overall prevalence of misconceptions, the number of correct responses did not increase for all misconceptions from the pre-treatment questionnaire to the post-treatment questionnaire. Namely, the two misconceptions M9 and M19 exhibit more incorrect answers in the post- treatment questionnaire than in the pre-treatment questionnaire. However, for both no significant differences were found.
4.4 Results 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% M1* M2* M3* M4* M5 M6* M7* M8* M9 M10* M11 M12 M13 M14 M15 M16 M17 M18* M19 M20 M21 M22* M23 pre-correct/retention-correct pre-incorrect/retention-correct pre-incorrect/retention-incorrect pre-correct/retention-incorrect
Figure 4.17: The changes in correct and incorrect responses for each of the misconceptions from the pre-treatment question- naire to the retention questionnaire. Each colour represents the respective number of participants with “pre- treatment/retention-treatment” responses, e.g. “pre-incorrect/retention-correct” is the number of participants having responded incorrectly in the pre-treatment questionnaire and correctly in the retention questionnaire. The misconceptions with significant improvements are marked with a ‘*’.
Pre-Treatment to Retention Analysis
This section describes the results pertaining to the prevalence of the misconceptions in the retention ques- tionnaire.
Differences between SME-r and the other two SMEs. The misconceptions were more prevalent in the employees of SME-r than in the employees of the other two SMEs. A Wilcoxon rank-sum test showed a significant difference (W = 422.5, p = .001). An effect size of r = 0.351 indicates a medium effect. This difference between SME-r and the other two SMEs remains in the post-treatment questionnaire, albeit smaller. A Wilcoxon rank-sum test again indicates the difference to be significant (W = 999.5, p = .0.015). However, an effect size of r= 0.265 indicates a small effect.
Analysis of H3b. The portion of correct responses increased from 63.5% in the pre-treatment questionnaire to 92.4% in the retention questionnaire. A Wilcoxon signed rank test indicates that this difference is signif- icant (V = 0, p < .001). An effect size of r = 0.605 indicates a large effect. Therefore, the results seem to provide supporting evidence for H3b.
Figure 4.17 depicts the changes in the prevalence of the individual misconceptions. Most of the miscon- ceptions (14 out of 23) exhibit no incorrect responses in the retention questionnaire7: M1 (χ2(1) = 22.04,
p < .001), M2 (χ2(1) = 22.04, p < .001), M3 (χ2(1)= 21.04, p < .001), M4 (χ2(1)= 13.07, p < .001), M5,
M6 (χ2(1) = 5.14, p = .001), M7 (χ2(1) = 5.14, p = .023), M8 (χ2(1) = 12.07, p < .001), M13, M15, M16, M18 (χ2(1)= 5.14, p = .023), M20, and M21. Not all of these misconceptions show significant differences due to ceiling effects. However, in particular M1, M2, and M3 stand out in SME-r with a high prevalence in the pre-treatment questionnaire, but a perfect score in the retention questionnaire and therefore a very strong (significant) improvement. In addition to the 14 misconceptions achieving perfect scores, five misconceptions achieve a prevalence of less than 10 percent (i.e. less than ten percent incorrect responses) in the retention questionnaire: M9, M17, M19, M21, and M23. None of these achieve a significant difference. Notably, the difference for M19 is an increase in incorrect responses (albeit not significant), bringing it down from a per- fect score in the pre-treatment questionnaire. Unfortunately, there is another misconception which exhibits more incorrect responses in the retention questionnaire than in the pre-treatment questionnaire: M11 (again not significant). This will be further discussed in section 4.5. Of the remaining misconceptions only M10 exhibits a significant difference (χ2(1)= 12.07, p < .001).
4 Development and Evaluation of Effective Password Security Awareness Materials