Experiment Design

The following sections will describe the tools used in this study with respect to Phase 2: Experiment Design. The ordering of the tools with respect to the experimental procedure are described in Figure 3-7.

Figure 3-7 Experiment Design Procedure

3.4.2.1

Pre/Post-Tests

As the efficacy and accuracy of the assessment measures will determine the accuracy and validity of the learning outcomes within the students, the need for standardised testing is apparent (Spӧrer et al. 2009). As such, certain test development organisations such as Australian Curriculum Assessment and Reporting Authority (ACARA) utilise standard tests such as the Statements of Learning for English developed by the Curriculum Corporation (ACARA 2017e). These statements attempt to standardise the expectations for curriculum and testing in Australian schools.

The purpose of this study’s pre/post-tests is to determine the state of the

participant’s knowledge in relation to the educational content. The tests are multiple-

choice questions and answers that are of similar structures to those of the Progressive Achievement Test (PAT-R) (ACER n.d.), or the Australian NAPLAN tests in literacy (ACARA 2017a), which are calibrated and normed measures for assessing the state of a person’s literacy capabilities.

Due to the standardised nature of Australian literacy education, NAPLAN testing is administered to all year 3, 5, 7, and 9 students in Australia on a yearly basis (ACARA 2017a). The standardised NAPLAN tests have created the space for many

Pre-Test •Administered at the start of the study Game •GEQ administered after 4 weeks GEQ •Administered at the end of the study Post-Test •Administered following the GEQ Student/Teacher Survey •Administered following the conclusion of the intervention

73

ancillary services, which allow educational providers to access NAPLAN aligned content. Education Services Australia, a joint state owned not-for-profit provides access for educational professionals standardised assessments aligned to the Australian Curriculum, with the ability to administer tests to students electronically. The content is provided to the Improve service by state and federal government educational bodies and aligns with the Australian Curriculum.

The student pre-test phase of the intervention involved surveying the current state of student knowledge with respect to reading comprehension performance. A series of 18 calibrated questions, aimed to challenge the students of different reading levels. This pre-test is described in greater detail in Appendix I. The purpose of the pre- test is to provide a baseline to student reading performance, through which a post-test can act as a second baseline of student performance, following the conclusion of the intervention period.

The pre and post-test quizzes draws texts, questions, and answers from the

Improve service and are split into three question sub-types (Australia 2017). The sub- types are author intent, inferring meaning, and identification of word meaning in context. These sub-types form an integral component of reading comprehension; the ACARA (2017b) version 7.5 lists these skills as (ACELY1698), (ACELY1702), and (ACELY1703) respectively. Both the pre-and post-tests are composed of 18 question and answer multiple-choice quizzes that resolve to four metrics: inference score, author meaning score, word meaning score, and the total reading comprehension score. The scores provide a metric of comparison for the four previously mentioned factors. The test can be viewed in full in Appendix I and Appendix II.

3.4.2.2Game Experience Questionnaire

While significance of learning outcomes can be measured quantitatively, it is important to determine how well the game system fulfilled student requirements. Serious games have the dual purpose of providing instruction, while also aiming to be entertaining. The entertainment value of the system is therefore as vital as the magnitude of change that each student may experience insomuch as continual participation with the system is well supported by meaningful play as described previously in Section 2.2.1. This study utilises the Game Experience Questionnaire (GEQ), designed by IJsselsteijn

74

et al. (2013) and provides a normalised pattern for describing participant experiences following a video game experience. The GEQ resolves to several normalised scores which indicate an overall picture of the following factors relating to video game experiences. The GEQ is divided into four separate iterations of the quiz: a core test, a social test, an in-game test, and a post-game test. This study makes use of the post-game test for two reasons. Firstly, the GEQ was administered twice over the course of the study, at the end of a play session. Secondly, fatigue with students was a primary concern, with respect to question and answer surveys. The post-game module provides a reduced question set, without compromising the validity of the measure. The reading age of this test was confirmed as approximately Grade 2 (age 7– 8), when assessed using an averaged reading age composite checker (readable.io 2017).

Questions GEQ Core Components

1. I was interested in the game's story

GEQ Core 3 Sensory and Imaginative Immersion

2. I felt successful GEQ Core 17 Competence 3. I felt bored GEQ Core 16 Negative Affect

4. I found it impressive GEQ Core 27 Sensory and Imaginative Immersion

5. I forgot everything around me

GEQ Core 13 Flow

6. I felt frustrated GEQ Core 29 Tension

7. I found it tiresome GEQ Core 9 Negative Affect 8. I felt irritable GEQ Core 24 Tension

9. I felt skilful GEQ Core 2 Competence 10. I felt completely absorbed GEQ Core 5 Flow

11. I felt content GEQ Core 1 Positive Affect 12. I felt challenged GEQ Core 26 Challenge

75 13. I had to put a lot of effort

into it

GEQ Core 33 Challenge

14. I felt good GEQ Core 13 Positive Affect

Table 7 GEQ Post-Game Survey

As displayed in Table 7, the GEQ Post-Game Module breaks game experience down into four categories of experience. The GEQ was administered twice during this study. On the first occasion, half-way through the intervention period at 4 weeks, and again at the conclusion of the study at 8 weeks. The purpose of multiple GEQ tests is to determine whether gaming experience changes over time, across cohort and within the same student.

3.4.2.3Student Feedback

The student feedback survey, as previously mention in aims to gather subjective, and demographic information on the students. The purpose of this feedback is to determine whether demographic subsets of the cohort have meaningful interactions with the system, in this manner, SRQ4 can be answered. Along with the demographic survey, opinions on the system were gathered, to provide information on the user experience of the system. The survey is described fully in Appendix III

3.5

Summary

This Chapter presented the core methodology used throughout this study. Tools for the design of a serious game intervention were described. The Behaviour Change Wheel methodology was detailed, along with the LM-GM framework for mapping game mechanics to learning mechanics. Tools for collecting data to answer the research questions were also presented, including a description of the pre/post-tests, and the Game Experience Questionnaire. The following Chapter will detail the way the serious game used in this study was developed, and the way it collects, stores, and processes the information needs presented in Section 3.2.

76

System Design

As identified in the preceding Chapters, attention and motivation are key components into the success or failure of serious game solutions for children. It is therefore imperative that both behaviours are fully examined in the context of a behavioural framework, such that a rigorous process of game design may be overlayed upon the said examinations and findings. In simpler terms, it is important to fully understand the problem, before a solution is designed for it.

4.1

Phase 1: System Design

Figure 4-1 System Design Process

As described in the previous chapter, Figure 4-1 indicates the tools that were employed throughout the design phase of this research project. The following sections will now describe the application of each of these tools.