Randomized Controlled Study II

For our new experiment, we made some changes in our intervention while maintaining the same experiment design. We increased the overall problems within the tutor. We also added extra-tutoring sessions. When students make multiple errors in selected problems, they are taken in special tutor-mode screen. Students go through tutorial on particular skill before they resume their activity in Monkey’s Revenge.

Figure 33 Screenshot of tutorial

We made some changes in our pre-test and post-test design. We created two sets of equivalent questions that were randomized among students. We also increased the number of questions in each set from 8 to 11.

4.1.5.1 Participants

A total of 252 middle school (12-14 year olds) students from four Northeastern schools of the United States participated in this study. Students were randomly assigned to the four groups, where the randomization was within each class.

Table 7 participants who logged in the tutor with pretest score

Tutor version N preTest % correct

mean (SD)

Monkey’s Revenge 53 53(23)

without visual feedback 46 52 (23)

without narrative 44 52 (24)

Basic tutor 47 52 (26)

RESULTS

We want to analyze the results on the paradigm whether our game-like intervention:

• Engaging (higher liking and satisfaction)?

• Effective (higher learning gain)?

• Efficient (no cognitive overload and time overload)

Similar to our previous study, we found a gradient across increasing levels of game- likeness where liking the tutor increases as the tutor becomes more game-like. The three groups with game-like elements are similar to each other and different from “Basic tutor”. We also asked if students liked specific elements such as story and graphics. The

Is game-like intervention Engaging (higher liking and satisfaction)?

Table 8 Survey Responses across tutors (mean, SD and 95% CI)

Tutor version N Like tutor Had fun Tutor helped Better than other programs Monkey’s Revenge 34 4.0 (0.9) (±0.3) 4.1 (1.2) (±0.4) 3.9 (.9) (±0.3) 3.9 (.9) (±0.3) without visual feedback 25 3.9 (1) (±0.4) 3.9 (1) (±0.4) 3.6 (1) (±0.4) 3.7 (1) (±0.4) without narrative 27 3.6 (1.3) (±0.5) 3.3 (1.3) (±0.5) 3.2 (1.3) (±0.5) 3.8 (1.3)(± 0.5) Basic tutor 28 3.0 (1.3) ( ±0.5) 3.0 (1.3) (±0.5) 3.1(1.3) (±0.5) 3.4 (1.3)(± 0.5)

We looked at student’s profile on who completed the study. We found that students with higher incoming knowledge were the ones who persisted in all groups. But this trend was more prominent as the tutor is less gamse-like. This suggests that the basic tutor was able to retain only high knowledge kids, losing the low knowledge kids. If we accept this hypothesis, we would expect to see higher overall retention in game-like condition, which is not the case now. A possible explanation would be that students in game-like condition could have run out of time rather than dropping. We cannot make any claim on the basis of data we have, but retention is a worthwhile variable we need to look at.

Table 9 Retention of students in Tutor groups

Students who logged in Students who completed study

Tutor version N preTest % correct

mean (SD) N

preTest % correct

mean (SD)

Monkey’s Revenge 53 53 (23) 34 66 (19)

without visual feedback 46 52 (23) 25 69 (20)

without narrative 44 52 (24) 27 70 (18)

Basic tutor 47 52 (26) 28 74 (18)

We also collected open feedback from the students to get a qualitative assessment of the tutor. Though we did not quantify the open feedback, we found the feedback to be favoring game-like aspects in general. The following is a sample of students’ open comment feedbacks:

“I liked how the monkey was brought into the story and how I got to give him a name. Also I liked how the story went with the coordinates and it wasn’t too difficult but helped me learn. Some of the problems were confusing though.”

“You made this exercise fun by putting in pictures, words and a story! These problems made me want to do more; I was always excited for what might happen next!”

“I liked the pictures, but some of the questions were pretty confusing. You could word the words a little better.”

“I think that the problems are challenging, but they could be harder. The storyline is great, same with the pictures. It would be great if the game was more interactive in a learning manner.”

“The monkey was very cute and usually I have a problem focusing but this helped me stay focused because I liked it a lot. Thank you!!! :)”

“This was very fun. I enjoyed playing it. I liked being able to customize my characters name, and it made it more fun to play. Also it made learning a little more interesting. The monkey was mean though.”

“I can’t do these problems. I didn’t like the pictures or scenario. I already have low self esteem.”

Is game-like intervention Effective (higher learning gain)?

This time again, we are again not able to make any conclusion on learning gain. Students in Monkey’s Revenge condition got significant learning gain. But the effect size is too small and variance too high. There could be a number of reasons for this. The study ran only for one class session of around 60 minutes. The tutor covered a range of skills rather than focusing on few measurable skills. A lot of students dropped post-test which shows their unwillingness and also lack of adequate time.

Table 10 Learning gain across tutor groups (mean and 95% CI)

Tutor N Pretest % correct

mean (SD)

Learning gain (in %) mean (SD) (95% CI)

Monkey’s Revenge 34 66 (19) 10 (27) (±9)

without visual feedback 25 69 (20) 5 (18) (±7)

without narrative 27 70 (18) 7 (29) (±8)

Basic tutor 28 74 (18) 3 (19) (±7)

Is game-like intervention Efficient (no cognitive load and time overload)?

One of our major objective is to minimize cognitive overload on students that could result from extraneous details. We did not have sophisticated rubric to measure cognitive overload. We are however observing students’ performance across tutors to make sure that cognitive overload is not causing lower performance. Students in Monkey’s Revenge condition are having more problems correct despite having slightly lower incoming knowledge. Though this does not guarantee that there is no cognitive overload (could have been offset by increased attention), cognitive overload is not prominent enough to cause hamper student performance.

Table 11 Student performance across tutors

Tutor N

Pretest %correct

mean (SD)

Problems correct in the tutor (max=27)

mean (SD) (95% CI)

Monkey’s Revenge 34 66 (19) 20.3 (3) (±1.1)

without visual feedback 25 69 (20) 19.8 (5) (±2)

without narrative 27 70 (18) 18.6 (3) (±1.2)

Basic tutor 28 74 (18) 18.5 (4) (±1.5)

To measure time overload, we calculated time spent on non-tutor activity (narrative sequences and general instruction). While total time spent on the study is similar for all tutor groups, students’ in Monkey’s Revenge spend 5 more minutes in non-tutor activity. Our effort for minimalist approach seems to have been effective.

Table 12 Time overload across tutor conditions

Tutor Total time

(in minutes)

Non-tutor time (in minutes)

Monkey’s Revenge 50 10

without visual feedback 47 13

without narrative 42 9