Choosing data gathering methods

There are at least three different factors that affect which methods to choose: 1) the nature of the project, 2) the skills of the team, and 3) the culture and style of the company [Jones and Marsden, 2006, p. 113]. The nature of the project determines what methods are applicable. This includes the phase of the game development and the goals of the evaluation. The second factor, the team’s skills also have an effect on what data gathering methods should be used. For example, conducting a field study or doing an interview requires at least some sort of expertise. Otherwise it is possible that wrong conclusions are drawn or some valuable information is left unnoticed. Usually the problem with the non- expert evaluators is that they are not accurate enough, i.e., they do not understand the importance of little details [Montola et al., 2005]. The third thing that affects is company culture and style, which basically means that some companies may have some traditions in what methods to use and how. Jones and Marsden [2006] do not mention resources as one of the factors, but they are also one very important factor that affects what methods can be used. Usually this factor is the most important when lack of time and money forces to take shortcuts in evaluation.

Based on the example game evaluations it is recommendable that at least two different methods are used (usually one that gives qualitative data and the other that gives quantitative data). One evaluation method is not enough to give reliable information so different methods should be used to back up each other and prevent from misinterpretations. Determining what those two (or more) methods should be is more difficult. It appears that despite the expansion style(s) of the game, the methods varied. One thing that of course affected a lot was that the games were in different development phases when the evaluation was carried out.

Combination of log data, questionnaires, and interviews gave a lot of valid information and proved to be efficient to evaluate playability and player experience. Therefore this combination is recommendable to use if possible. Still in some situations, this combination can be too complex and time consuming to be used, especially in the middle of the game development process. The situation of course is different in every gameplay evaluation so methods should be selected to meet the circumstances and demands of that particular situation.

Interviewing was the most often used data gathering method among example game evaluations; it was used almost in every evaluation. It seems that interviewing fits all cases. The same was with questionnaires. There were no noticeable differences in their use between the expansion styles. One reason why these methods were so popular despite the game might have been the fact that they are probably the most well-known and used data gathering methods. They also fit very different situations depending on how they are used. By using open-ended questions it is possible to get a lot of qualitative data and also notice issues that would not be seen otherwise. Based on the experiences of the example game evaluations interviews and questionnaires are data gathering methods that should be used in every evaluation, especially if the evaluation is done for the final game product.

Games that expand the magic circle socially used most often combinations of interviews and focus group discussions. The exceptions for this were Insectopia (only interview), Botfighters (only interview), and Day of the Figurines which did not use either of these methods. The focus group method was used in many example game evaluations in the scenario phase, when the game idea was tested. It is quite understandable why the focus group method is so popular especially when evaluating socially expanded games, because it is not always obvious what is socially acceptable and what is not. For example Niemi et al. [2005] used this method to solve which of the four games (Yum Yum Sheep, Your Story, Spy Blob, Hot Potato) would be the most acceptable for the target audience. This evaluation showed how effective focus groups can be in detecting issues that include for example ethical concerns that people may have. Therefore focus group discussions fit well for evaluating the acceptance of socially expanding pervasive games in their early development phases.

The field observation method was used the most in those example games that expand the magic circle spatially. Another joint characteristic for these games is that most of them are event games, which means that they run only for a short period of time and need moderators. It may sound weird that games expanding the magic circle spatially use field observation, because their playing field is not fixed in the traditional sense. This could make field observation impossible. However, some of the spatially expanded example games happen in some pre-set area so using field observation is possible, although not in the traditional way. In some of the example game evaluations where field observations were collected (like in Savannah) the observations were recorded using video cameras attached to a couple of players. This way the evaluators could get information on what happened from the player point of view. Based on the Savannah evaluation such recording worked well, even though it is

technically more challenging than for example an interview. Still using this method could be a good idea especially when the game is targeted for small children.

A closer look at the example games showed that it is not easy to say which methods would be the most suitable for which pervasive game type. It seems that other issues are more important when selecting data gathering methods than the expansion style. One of the most important affecting factors was the development phase of the game. Also there have been so few pervasive game evaluations so far that the methods that are most suitable for them are not yet clear. More research is needed to find out what different data gathering methods really would be the best for which pervasive game type.