Method

In the study I conducted to pilot my assessment methodology use of vLeader was embedded within a masters-level degree course run at UCL. The class consisted of 60 students from different management degree programmes that were taking part in a class titled Leadership, Ethics and Communication. The first 5 weeks of the course were dedicated to the topic of leadership. Use of vLeader was embedded in that period. In addition to the game, the course had a number of other activities dedicated to the topic of leadership including lectures (reviewing different theoretical models of leadership), case studies (shedding light on practical complexities) and videos (demonstrating leadership behaviours in action).

As a result of the methodological review conducted in Chapter 4 for this study I used a modified methodology for the evaluation of learning effects in serious games. I chose to adapt the learning study (Pang and Marton, 2003) in order to evaluate a serious game. As a research methodology the learning study is an attempt to take advantage of, and combine, controlled experimental studies with an incremental improvement method called

lesson study. The learning study is used to evaluate by comparison of learning effects

between subsequent deployments of a course, introducing improvements over several iterations. In order to fit within a single university term I attempted to provide two comparable learning conditions within a single class.

The study used a between-subjects design. The independent variable was variation, which included the conditions variation (the experimental group) and no variation (the control

group). At the beginning of the study, the students were randomly assigned to either the

experimental condition or to a control group. I did not have the necessary level of control of the software to implement the separation between control and experimental conditions, so it was implemented through the paper-based materials (containing different written instructions for the two groups) for each scenario, called activity sheets. These are described in detail in Section 5.2.

5.1.1 Learning Environment

To put it in context, vLeader was analysed according to the evaluation framework of de Freitas and Oliver (2006) as was reflected in Table 3. Here it is considered in further detail how it is meant to be used and how this was adapted for the purposes of this study. This is done by reviewing the four dimensions of the framework: context, learner specification, pedagogic considerations and mode of representation.

The simulation game was developed by Simulearn Inc. and is commonly delivered as the main learning resource for commercial webinars, which defines the context according to de Freitas and Oliver. These webinars are typically a series of online conferences before and after learners play each scenario at home (a take on briefing-debriefing of the game experience). At a webinar a facilitator introduces features of vLeader and encourages learners to discuss their experiences with it. Learners are encouraged to play each scenario eight times. The game itself is described in Section 5.2.1. The webinars are accompanied by a Student’s Workbook which guides learners through the learning process for each of the sessions. Simulearn Inc have also occasionally deployed the software in university classes. In doing this, they have identified the need to break down their Student’s Workbook into smaller bits of information that would allow for better adaptation to the curriculum of the host organisation. However, this process is still at its start and was piloted with the materials prepared for this study (see Activity Sheets described in Section 5.2.2).

Using vLeader in commercial trainings, as opposed to university classes, implies that there’s very little common ground between learners across courses. Thus it is difficult to talk of general learner specification, as de Freitas and Oliver suggest. Yet, within a single webinar typically learners come from the same company which gives them certain common background. The actual simulation game being single-user, discussion and reflection are the activities that allow for group work and this is being utilised in the

webinars and university classes, including in this study.

With respect to pedagogic considerations, vLeader is a typical example of a simulation game promoting experiential and exploratory learning. It employs techniques like conflict resolution (a special type of problem-solving) and role-play with changing perspectives. Yet, as Kebritchi and Hirumi (2008) point out that it is typical for educational games, the authors of the game do not elaborate whether and how they derive the possible game activities from a pedagogic theory (Aldrich, 2003). Despite that, the authors manage to come up with their own theoretic model and a game design that allows parallels to other leadership and negotiation theories like Fischer and Ury’s model (1991), even if this has not yet been documented in corresponding learning materials. For the appropriation for the purposes of this study the game (detailed in Section 5.2.1) and class (see learning objectives in Section 5.2.2) goals were elaborated and related to one another.

In vLeader the authors have chosen to use a first-person 3D environment, supposedly to help players identify with the played role. They have also very cleverly adopted an interaction paradigm that is closed in itself (meaning that there’s only a limited number of possible interactions), but still allows for great expressive power that is relevant to leadership and negotiations. This is further elaborated in Section 5.2.1.

5.1.2 Procedure

The study consisted of three major stages: learning, assessment and post-study interviews. The learning stage was influenced by the available game scenarios, the experimental between-subjects design and the procedure for handling each of the scenarios. The learning stage occurred over 5 weeks. During this period, five classroom sessions, 180 minutes each, were used for teaching. In each of these sessions, only a part was dedicated to the study in order to introduce the game scenarios, to administer study materials, and to facilitate group discussions around the game playing experiences. The rest of the classroom sessions were used for delivering lecture materials on the topic of leadership. The game playing experiences took place individually in between these classroom sessions. As was the practice in Simulearn Inc’s courses, it was suggested to students to play each scenario at least 8 times. Before each class the corresponding learning materials were made available on the course learning support system (Moodle).

In Week 1, the students were introduced to the game and the study. Students were given 10 minutes to do the pre-test questionnaire. The course tutor then continued with the course

lecture. Towards the end of the 180 minutes, the first game scenario was introduced and the students were each handed a paper-based version of the activity sheets. Students were then encouraged to play Scenario 1 of the game at a time of their own choosing prior to the next classroom session. This introduction to the game during the first week was disrupted by a fire alarm and subsequent evacuation of the whole building. As a consequence the introductory session was less elaborate than originally intended, but the study had to continue, effectively relying on the online presentation and follow-up student inquiries. In Week 2, the 180-minute classroom session incorporated a 10 to 15 minutes group discussion that focused on the students’ experiences of Scenario 1 of the game and relevant leadership theory. Typical questions in these discussions were addressed to how each of the game characters behaved and whether students could relate these game experiences to real life examples. Towards the end of the session, I introduced students to Activity Sheet 2 (provided online) and Scenario 2. Again, the students were encouraged to play the game prior to the next 180-minute classroom session.

During the remaining weeks, a similar pattern was followed: the game scenario introduced the previous week was discussed in a short 10 to 15 minute session and, at the end of the class, the next game scenario and activity sheet were introduced. The initial discussion was held in relation to the relevant leadership theory presented in that particular class. Due to the restricted number of classroom sessions dedicated to leadership, it was necessary to introduce and encourage students to play scenarios 2 and 3 in one week. These two were chosen in particular because of the similarity between the two situational contexts. In the final session (Week 5), following a short discussion of the fifth and final game scenario, the post-trial written assessment test was administered.

5.1.3 Learning Assessment

As explained in detail in Chapter 4 written assessment tests are a widely used form of assessment in studies of learning technology (Anderson and Lawton, 1992). Free-form written assessment methods can be used to measure deep learning (Biggs and Collis, 1982). This study employed written assessment tests before and after exposure to vLeader as a measure of learning that took place.

For this study, a bespoke written assessment questionnaire was developed to measure understanding (RQ1). It consisted of 3 open-ended questions and 7 Likert-scale questions (see Section 5.2.3). Responses to open-ended questions were examined via content

analysis. In order to test application (RQ2), a bespoke role-play assessment was developed, putting students in a situation, similar to the ones they encounter in vLeader. Retention – the number of times that learners return to play the game – was intended to be used as a measure of engagement (RQ3).

After the learning and assessment parts of the study were completed, a series of in-depth semi-structured interviews were conducted. The aim of these interviews was to further explore students’ perceptions of the game and application of variation in particular. The planned questions are listed in Table 8. These were adapted to become relevant for the experience of each individual participants. For example, changes considered where students used vLeader and how this affected their experience, when a student did not engage with the game or the role-play, they were asked what made them not engage, instead of asking for comments on the experience.