Data Collection and analysis

The CatchBob! platform allowed us to collect a wide set of data ranging from quantitative measures to player interviews and accounts of the game. This section describes this data and how we used it for different analyses.

Task performance

We measured task performance as the group travel distance (i.e. sum of the path length over all players in a group). Task performance is thus a group index measured in meters. We did not choose time as a performance variable since we did not want players to run on the campus with a Tablet PC and because finding a proper path was better suited to the discussion of a relevant strategy. We categorized the different group spatial behaviors that had been reflected in the paths each individual within a team chose. Looking closely at how players spread over the campus in the first phase as well as how they joined the first one who found “Bob” allowed us to gain insights about the group strategy as well as how the MLA interfaces impacted it. For each of the three parts of the game, players demonstrated various spatial behaviors. In the first phase, there were two possibilities of spreading over the campus as depicted on Figure 37. In the former, players maximized the exploration of the campus by going in three different directions. In the latter, players went all in the same direction, not grouped together but instead heading out together one behind the other.

Strategy A: a way to maximize the exploration

Strategy B: all in the same direction.

Partners’ trails recall

As explained in the procedure section, we asked players to draw their own path and the one of each of their partners. This enabled us to calculate the number of errors players made while drawing the path of their partners after the game. We used this “positions recall” measure as an indicator of mutual modeling accuracy.

Since players had to draw the paths of their partners, we could compare the path player A attributed to B with B's real paths and the same for A&C or B&C. This comparison, measured by the number of errors, represents the quality of A’s representation of B and C’s behavior in space.

Figure 38. (a) Drawing A made of B’s path; (b) Real path followed by B as extracted from the logfile.

Calculating the number of errors was made through the use of layers on printed material: the trails drawn on the paper-based questionnaire (Figure 38a) and the real paths generated after the game by the replay tool (Figure 38b). What we counted as an error was either a place where the partner has not been or the omission of a place where he/she went. Two criteria have been defined to describe what constituted an error: distance (if the line was longer than the maximum size of our campus corridor), presence of an obstacle (door/wall/glass). Retracing ones steps was not perceived as an error. Calculating the number of errors an individual made concerning his or her own path was also a way to evaluate one’s ability to do it accurately.

We calculated this position recall index for each individual and each group (the sum of the individual measures). An individual Mutual Modeling accuracy index is the sum of errors made by a player about his/her two partners’ paths. We calculated this MM-accuracy for each individual (M (A,B), M (B,A), M (A, C), M (B,C),…) and for each group (the sum of the individual measures). It is important to stress that subjects made very few mistakes when drawing their own path on the campus (85% made 0 errors). This is why we did not use the number of mistakes made about one’s path in the MM measure (to avoid dividing by 0).

Communication

Shared map annotations written on the Tablet PC were the only technological way to communicate within a group during the game; they could however have occasional

face-to-face meetings on the field. To help us understand the data, we developed our own coding scheme that we employed to transcribe the annotations. We coded the content of the messages (position, direction, strategy, proximity to the object, off- task) and their pragmatic status (announcement, order, question, acknowledgement, corrections): each map annotation then had two codes. Figure 39 shows examples of such messages with the corresponding categories. Inter-judge reliability of the coding system (performed on 25% of the sample) has proven to be good as shown by a Cohen’s Kappa (1968) of 0.89 for the content variable, a kappa of 0.86 for the pragmatics variable.

This allowed us to count the frequency of each category as well as the total number of messages exchanged by each player (and then each group).

Figure 39. Examples of messages with regard to the two coding schemes: messages content and messages pragmatics.

Other qualitative data

The other qualitative data we collected ranges from players’ interviews after the game to individual’s spatial trails. The discussion with players when confronted with the replay tool has been used to investigate how players coordinated over time, namely to describe the coordination problems and the solutions. It also enabled us to further the investigation of how MLA has been used, and its role in collaborative problem solving. We employed players’ paths to understand the general spatial behavior of individuals and groups, as they reflect the strategy they adopted. We categorized the different spatial strategies to investigate whether the presence of the awareness tool would eventually have an impact. It also enabled us to define roles for each of the players.