The validity of information gained using different methods

It is important to make sure that the information gathered is as valid as possible. Sometimes it may be hard to figure out whether or not the results are accurate or not. The reason for invalid results can be for example that there was something wrong with the evaluation arrangements like that the questions might have been hard to understand and to answer.

In the evaluation of Coup we noticed that data gathered from other sources apart from log files yielded quite similar observations. For example, many participants complained in the interview about the unresponsiveness of other players and long waiting periods. That also became evident from the questionnaire. Figure 23 shows that seven out of nine Coup players felt that there is too much waiting in the game.

Figure 23 Coup players felt that there were too much waiting in the game (N=9).

The questionnaire and the interviews showed that almost all of the players felt that they were alone in the game as can be seen from these player quotes.

…So I really couldn’t play properly that because you always had to get some approval from the emperor, and it could take a day to get some approval… (male, 24)

You just waited there that those others would try to do something… (male, 25)

I had some difficulties to maintain the interest because there were so few players and then quite a many of them were not that active. So my own game stood still when I couldn’t really do anything else when those others didn’t come into the game to give an approval or something… (male, 24)

However, log data showed otherwise. According to log data, players were in the game almost at the same time. If the analysis had been based solely on the log data it would have appeared that the players did have the possibility to interact with each other and not having to wait so long.

Based on the evaluation of Coup, questionnaires and interviews give a lot of valuable and usually correct information about the player experience. This can be said because the information that was gained using these methods was so consistent. On the other hand, it also showed that it is not sufficient to use solely log data in the evaluation.

In the evaluation of Coup, questionnaires and interviews gave consistent information. This was not the case in the evaluation of Insectopia. Insectopia is a quite simple game so players can easily start playing the game even without reading the manual first. The fact that most of the participants did not read the manual at all became very clear in our evaluation. The players could not understand or had missed entirely the insect feeding functionality of the game. About half of the interviewed participants did not realise at all that they were supposed to feed the insects. This comes out very clearly in the following two quotes from interviewed participants.

The feeding didn’t come out in any way in this game. I had no idea that I was suppose to feed them… (female, 24)

Hmm… how did the feeding actually work? I thought that it happened automatically when I did a search. Is there some specific place where you can feed them? (male, 25)

This information differed from the information that we got from the questionnaire. In the questionnaire one question was about how easy the information given on the screen was to understand. Almost 70 % of the participants replied that it was easy to understand (Figure 24). Still they had missed out totally one of the core functionalities of the game and we would not have noticed that without using also interviews.

Figure 24 The participants felt that the information that was given on the screen was easy to understand (N=9).

In the example game evaluations focus group discussions were used in the game concept phase which means that the actual games did not exist yet and the participants could not play the games for real. In the evaluation of the Garden of Earthly Delight [Koivisto and Eladhari, 2006] it was noticed that people who had playtested the game gave more positive feedback than those who participated only in the focus group discussions. This is probably because of the novel nature of pervasive games. Participants have difficulties in understanding this new way of playing so they look at it sceptically. So focus group discussions cannot predict what kind of reactions people will have towards the finished pervasive game.

The problem with focus groups is that one should be able to describe a concept without having anything concrete to show. This causes that the results are based on participants’ imagination and it can be far from the actual concept that was on designers’ mind. This happened also in one of the world’s most popular PC games The Sims. When Will Wright (the man behind The Sims) presented the idea to the focus group they hated the idea. It was also rejected by the product-selection committee [GIGnews, 2002]. As Will Wright said:

Most people when they imagine that concept in their minds are not going to imagine something very compelling. So it’s the designer’s task to make that into a compelling activity with interactions, the way it’s visualized, etc.

This shows that even though focus groups can give valuable information about the game concept it still does not necessarily predict what kind of a game it will be. Sometimes it is good to take the game concept further in the development process even if the focus group rejected the idea, but only if one believes in the game.

These experiences show that every method has its weaknesses and therefore none of them should be used alone. Interviews seem to be the most accurate method to use, so if it is necessary to choose only one method, then it should be an interview after a playtest period. Interviewing is the only method that gives an evaluator the possibility to ask clarifying questions and really find out what a participant thinks about the game. Also if the participant has difficulties understanding the question the evaluator can clarify it. The danger of misinterpretations is lower in interviews than for example in questionnaires.