Feedback Forms

The results from the student feedback forms provided student reflections of the learning experience. A total of 12 feedback forms were collected in the first study and 11 in the second. Figure 25 shows a comparison between the distributions of responses to each of the 8 questions in both studies. As can be seen from the graphs for the first class, average responses dominated with all questions averaging between 2.7 and 3.4 and with median of 3 across all 8 measures. The averages for lecture and prototype were 3 or more and tutorial and facilitation getting averages up to 2.75.

In the second class all responses were noticeably better. All medians were 4 and all averages were 3.4 or more. There was clearly higher appreciation for insightfulness (which

Figure 25: Student responses to the class survey after the use of CCO. The left chart shows how engaging students found the components of the experience and the right one shows how insightful it was. Note: Here µ denotes

is how informative the learning activity was). There was also a strong increase in the attractiveness of lecture and facilitation, with tutorial and toolkit being somewhat less appreciated. As can be seen in Table 25 this also resulted in insignificant difference when results from the two classes are compared for attractiveness of the tutorial and the toolkit. The answers to the open questions in both classes reconfirmed some of the observations in the lab study. When asked about what they found positive, students wrote that the toolkit is “very intuitive” (C1S3) and “easy to use” (C1S4). Others pointed out that it is “very

attractive and colourful” (C1S1) and that they liked the “design” (C1S12). In the second

class one student (C2S3) called the toolkit “amazing, especially when it marks an idea as

innovative”. They also wrote that it was positive that this was “practical exercise” (C1S2)

and that “there was a lot to think about, very engaging” (C1S1). Two students in the second class (C2S2), (C2S6) liked the toolkit’s structured approach. One of them summarised: “well structured, understandable, educating” (C2S6). In the second class two students (C2S8), (C2S10) noted that they appreciated the feedback they received.

When it came to discussing the downsides of the toolkit, again there was some agreement that it was “too vague” (C1S1), (C1S2), “somewhat confusing” (C1S6), “very hard to apply

the results in general way” (C1S8), “confusing questions” (C1S11), “too rigid” (C2S10), but

there was at least some appreciation of the benefits related to this ambiguity, one of the answers being clarified with “this made us think out of the box so it was good” (C1S1). One student (C2S8) criticised the duration of engagement of the toolkit saying “The process

seems too long for one sitting” (C2S8).

Again, similar to recruited participants, students found that it was difficult “to distinguish

between some of the cases such as environment and enclosure” (C1S1); “understanding

Construct Scale df p(T<=t) t Critical

Insightful Lecture 20 0.0013 1.7247 Tutorial 21 0.0057 1.7207 Toolkit 21 0.0161 1.7207 Facilitation 21 0.0001 1.7207 Attractive Lecture 20 0.0005 1.7247 Tutorial 20 0.0754 1.7247 Toolkit 21 0.1068 1.7207 Facilitation 21 0.0051 1.7207

and applying case-relevant causes/influences” (C1S8), to “come up with ideas” (C1S12);

and “language” (C1S10). Students did not provide much feedback to the other questions, except a few cases of reconfirming the same conclusions “the differences between some

(similar) categories” were difficult (C1S7), “don’t understand how scoring is done” (C1S4),

and something also encountered in the lab study: the scenario would have been presented better with “video and audio instead of text” (C1S12).

8.4 Discussion

The class studies demonstrated complexities related to real-world research. This included a combination of factors undermining student engagement with the task. It is typical for students that they are assessment-focused (Biggs and Tang, 2007) and thus not very engaged for activities that do not count towards their final grade. The proportion of non- engaging students in these classes was clearly larger than in the case of vLeader. However, these comparison could be possibly related to the general tendency towards more flexible engagement in the module of these studies.

Also, the inadequate expectations and the subsequent ambitious plan for the engagement of students led to an unrealistic agenda for the first tutorial. This led to a weakness in facilitation that undermined the entire use of the toolkit. Quantitative results from the first feedback survey clearly show that students did acknowledge the problems with facilitation during the tutorial.

Feedback clearly reflects the fact that the second class went better. In the survey responses appreciation for the lecture and facilitation outperformed appreciation for tutorial and toolkit. Even though it could be considered that this indicates that the toolkit was not as attractive as intended, it does also show overall appreciation for the learning process, which is related to the intended objective about engagement.

As can be seen from the results comments on the toolkit were very similar to the ones collected in the lab study. In both cases there were comments about the general attractiveness of the toolkit, but also the potential to improve it further with more multimedia. There was also the discussion of the vagueness of rays, with some learners appreciating the need for such vagueness. This theme merged into the theme about the difficulty to contextualise the general guidelines of CCO into causes and interventions for the specific problem being analysed. Learners were complaining of an unintelligible scoring mechanism and challenging professional terminology from crime prevention

(students were more familiar with information security jargon).

Despite the suboptimal delivery of both the toolkit and the way it was delivered to students (facilitation), students engaged in discussion and successful idea generation. Although both the CCO framework and the toolkit are complex and difficult to understand, from comparing the two classes it becomes apparent that better facilitation significantly improves the experience. One particular change that was adopted in facilitation was that students were driven to engage in more personal discussion about the problem, rather than discussing the use of the toolkit itself.

8.5 Conclusion

This chapter described the development and evaluation work conducted to develop an early version of a web-based prototype. In the class study insights were collected about a number of tacit skills related to facilitation that are critical to computer-supported learning.

Consistently in these studies, similar to Study 1 (using vLeader) reported in Chapter 6, some engaged with the game for longer periods, and others engaged minimally, regardless of financial incentives (in the case of Study 4 – first class study using the CCO toolkit). Engaging beyond the required minimum typically led to more interaction (and in the case of CCO data generation) than during the initial period.

Despite its existence for over a decade and positive reception among crime prevention practitioners across the world, feedback collected in these studies, challenged CCO itself. Participants had very specific comments regarding the visual representation of the theory and the terminology used. As a result, efforts have started towards re-phrasing and re- designing the CCO diagram and its terms, but this is beyond the scope of this thesis.

Using an incremental approach to develop a serious game, I succeeded in delivering a mature prototype that already gives early results – both in terms of engagement and learning. Frequent prototypes and usability feedback provided early signals on what needed changing in the game design and what could be kept. They also provided valuable insights into the underlying CCO theory and the process of teaching it. The user evaluations examined the types of engagement (as a contribution to learners’ motivation to learn) and learning developed with the prototype.