Comparing the experiments

Although the experiments have a similar set-up, there are differences between them in the areas of (A) independent variables, (B) setting, (C) incentives and rewards, (D) utility, and (E) the measurements for which we look. Below, we discuss these differences. An overview is given in Table 4.1.

A. Independent variables

In all three experiments, we compared playing a serious game voluntarily (VG) with playing the game mandatorily (MG).

In addition, in the second and third experiment, we looked at the effects of participating in the experiment voluntarily (VP) versus mandatorily (MP).

All participants in Experiment 1 participated voluntarily (VP). Hence, in Experiment 1, all participants are in the group of voluntary participation (VP). In Experiments 2 and 3, teachers assigned participation to their students, resulting in a group of mandatory participants (MP) in addition to the voluntary participants (VP).

The independent variables for all three experiments are (1) Participation and (2) Gameplay.

B. Setting

All participants in Experiment 1 were recruited through social media. They volunteered to participate. In Experiment 2, recruitment was done through institutions for Higher Education, and in Experiment 3 through Flight Academies.

Some participants volunteered; for others, it was part of a school assignment. This resulted in an informal setting for Experiment 1 and more formal settings for Experiments 2 and 3.

4

C. Incentives and rewards

In Experiment 1, all participants who completed the experiment had a chance of winning a gift card. This can be considered to be an incentive to participate. In Experiment 2, some of the VP participants had a chance to win a gift card, while other VP participants received school credits. Receiving credits can also be considered to be a reward. The MP participants were neither offered an incentive nor did they receive a reward. In Experiment 3, we did not offer any incentive or reward to the participants.

D. Utility

When utility is lacking, motivation to learn is unlikely [203]. Based on the work by Vroom [203] and Clark, Dobbins and Ladd [34], we define utility as follows.

Definition 4.8 - Utility

Utility is the perceived usefulness of a training to reach a particular goal, for example in a job or career.

Participants have a higher motivation to learn and report a greater amount of learning when they identify the utility of the training for their job or career [34, 168]. Furthermore, a higher sense of utility has been linked to a higher motivation to transfer the knowledge and skills to the workplace [121, 170].

For Experiments 1 and 2, we did not focus on recruiting participants for whom the topic of the experiment was relevant. A small number of participants may have perceived utility in the topic, but in general, the topic was not a motivating factor. Experiment 3 was aimed at pilots and pilots-in-training. Therefore, the topic was more relevant to them, i.e., they had a higher perceived utility.

E. Measurements

All three experiments in the study serve to determine the effect of voluntary (VG) and mandatory (MG) gameplay on the game’s learning effect and the player’s game experience. In total, we look at seven measurements (see below and Table 4.1) to determine the learning effect and game experience. Two measurements are objectively measured, viz. Game score and Test score, the other measurements are more subjective as the participants give their personal opinion about them. The participant’s opinion about being obliged to play is used to measure the effect of mandatory play. After Experiment 1, we add a focus on the motivation to participate and the engagement in the game.

With regard to the learning effect (RQ 2a), we look specifically for Measurement 1: Game score, Measurement 2: Test score, and Measurement 3: Time spent playing the game.

With regard to the gameplay experience (RQ 2b), we look for Measurement 4: En- joyment, and Measurement 5: Obligation. If present in the experiment, we will also look for Measurement 6: Motivation, and Measurement 7: Engagement.

4 C. Incentives and rewards

In Experiment 1, all participants who completed the experiment had a chance of winning a gift card. This can be considered to be an incentive to participate. In Experiment 2, some of the VP participants had a chance to win a gift card, while other VP participants received school credits. Receiving credits can also be considered to be a reward. The MP participants were neither offered an incentive nor did they receive a reward. In Experiment 3, we did not offer any incentive or reward to the participants.

D. Utility

When utility is lacking, motivation to learn is unlikely [203]. Based on the work by Vroom [203] and Clark, Dobbins and Ladd [34], we define utility as follows.

Definition 4.8 - Utility

Utility is the perceived usefulness of a training to reach a particular goal, for example in a job or career.

Participants have a higher motivation to learn and report a greater amount of learning when they identify the utility of the training for their job or career [34, 168]. Furthermore, a higher sense of utility has been linked to a higher motivation to transfer the knowledge and skills to the workplace [121, 170].

For Experiments 1 and 2, we did not focus on recruiting participants for whom the topic of the experiment was relevant. A small number of participants may have perceived utility in the topic, but in general, the topic was not a motivating factor. Experiment 3 was aimed at pilots and pilots-in-training. Therefore, the topic was more relevant to them, i.e., they had a higher perceived utility.

E. Measurements

All three experiments in the study serve to determine the effect of voluntary (VG) and mandatory (MG) gameplay on the game’s learning effect and the player’s game experience. In total, we look at seven measurements (see below and Table 4.1) to determine the learning effect and game experience. Two measurements are objectively measured, viz. Game score and Test score, the other measurements are more subjective as the participants give their personal opinion about them. The participant’s opinion about being obliged to play is used to measure the effect of mandatory play. After Experiment 1, we add a focus on the motivation to participate and the engagement in the game.

With regard to the learning effect (RQ 2a), we look specifically for Measurement 1: Game score, Measurement 2: Test score, and Measurement 3: Time spent playing the game.

With regard to the gameplay experience (RQ 2b), we look for Measurement 4: En- joyment, and Measurement 5: Obligation. If present in the experiment, we will also look for Measurement 6: Motivation, and Measurement 7: Engagement.

Table 4.1: Comparing the experiments

Experiment 1 Experiment 2 Experiment 3

Independent variables Gameplay Participation, Gameplay Participation, Gameplay

Setting Informal Formal Formal

Incentives and rewards For all participants For some participants None

Utility None None High

Measurements 1. Game score + + + 2. Test score + + + 3. Time spent + + + 4. Enjoyment + + + 5. Obligation + + + 6. Motivation - + + 7. Engagement - - +