Methodological choices

3.2.1

Three experiments

In this thesis, we present three experiments that have been conducted in different technological contexts. The first study has been carried out using a 3D virtual reality environment and the two others using the physical world through a pervasive computing application. The former is a preliminary experiment that helped us shaping the second and third experiments, which have a broader scope. In each of them, we have different conditions based on the presence or the absence of an MLA interface. The studies are not comparable strictly speaking but we aimed rather at understanding the salient and common trends in both environments, and not at making a point-by- point comparison of the results.

In addition, in line with our choice to have a deductive approach, we carried out experiments in two contexts. In the 3D virtual environment, the experiment was classic and easily controlled (because of the fixed setting). However, the study in the pervasive computing context was trickier. Empirical studies of pervasive applications generate various epistemological problems, especially about the confrontation of different experimental conditions. Using laboratory experiments (Kjeldskov et al., 2004) to control all the variables appears to be the wrong approach since it is not possible to artificially recreate an ecological validity. There is indeed a lack of physical (e.g. people’s movement during mobility) or socio-cultural context (e.g. the role of place, as described in Harrison and Dourish, 1996).

Another common methodology is to draw on ethnographic methods. In this case, the problem is the lack of objective measures such as the performance of the device or finding solid evidence to make comparisons between experimental conditions. This is why our objectives as well as the need to have the most natural context led us to use a ‘field experiment’ approach (Goodman et al., 2004), derived from the notion of “quasi-experiment” developed by Cook and Campbell (1979). Field experiments are quantitative experimental evaluations conducted out in the field, drawing from aspects of both qualitative field studies and lab experiments. On the one hand, they involve

real users in an activity that is setup in the real world. On the other hand, we can control variables and have different experimental conditions.

3.2.2

Benefiting from quantitative and qualitative analyses

Although we used controlled and semi-controlled experiments, we did not limit ourselves to quantitative data. As we mentioned previously, our research also has an exploratory dimension; this is why we also collected qualitative sources of information such as group interviews or messages content. Concerning the operationalized research questions, we will not present here the data we used in each experiment. It will be detailed in the descriptions of each study.

Each kind of data is meant to address our research questions. Our questions are addressed through quantitative measures such as mutual modeling evaluations or group performance; these indexes are used to compare situations in which players have a MLA interface to “control” situations (without MLA). Ethnographic methods will also be used to intensify our investigation of how MLA is interpreted and used among group participants. This research is therefore grounded in a quantitative dominant paradigm, with which we also used qualitative techniques (Creswell, 1994). Our aim is to gain a greater understanding of MLA influences on collaboration from the use of both qualitative and quantitative research methods. This is why we have chosen a developmental combination, in which quantitative analyses are used sequentially to test our hypotheses and qualitative techniques are used to illustrate them.

In terms of analytic orientation, given that these studies address the cognitive processes at stake in collaboration, we are interested in observable behavior (from an external observer) as well as the point of view of the participants. This is why we employed interviews and self-confrontation techniques in the second experiment to complement the observer’s analysis with the actors’ perceptions of what happened during collaboration.

3.2.3

Multi-user games as a collaborative platform

The tasks we used to assess the influence of MLA interfaces on collaborative processes were bound to a specific type of joint activity. First and foremost, we only studied small groups of participants, consisting of 2 persons in experiment 1 and 3 persons in experiments 2 and 3. In addition, these groups were engaged in a decentralized collaboration: the activity they carried out did not require any central command hierarchy. They were all participating in the task environment with the same information and there was no centralized control structure. The roles participants took during the collaboration thus emerged from the group dynamic, through conversations during the activity (or before in the case of experiment 2-3 in which there was a planning phase). Conversely, they were not given any normative principles to achieve the task: the way they completed the mission they had to undertake was entirely and not imposed by experimenters.

In order to conduct both experiments, we chose to use collaborative games, as proposed by Chalmers and Juhlin (2005). The use of such kind of platform has already been discussed in the human computer interaction field for a long time (Donchin, 1995; Holmquist, 1997). Several scholars have stressed the interest of using virtual environments like video games as research tool for psychological investigation (see

Slangen de Kort, 2001 for a general review about this topic) by citing three major reasons.

First, computer games are motivating and fun, and successful experimentation is easily achieved. Maintaining one’s undivided attention in video games is certainly easier than in other experimental environments. The use of a game metaphor has the advantage that it allows the presentation of complex problem solving tasks in an enjoyable environment, thus maintaining a high level of motivation amongst subjects. Besides, recent developments in augmented reality (Nilsen et al., 2004) have highlighted the motivational value of using game in HCI. Second, a game, especially a mobile computing one, involves participants in a real context (the physical world) with a certain ecological validity. A game in public space indeed creates a certain kind of complexity with passers-by or real-world features. Another useful aspect is the fact that they attract “participation by individuals across many demographic boundaries such as, age, gender, ethnicity, educational status and even species” (Kowalski, 1997). We thus expected participants to have a higher level of involvement in a game than in another kind of complex task.

However, these statements only hold for subjects that find such games enjoyable, those with little interest in games can fail to engage with the game, finding both the task and the interface difficult and confusing. Therefore, we chose to design simple games to avoid failures and misunderstandings.

3.2.4

Unit of analysis

As described in the research scope, we are interested in group cognitive processes. Research about collaboration has to deal with the “unit of analysis” problem: should we analyze social interactions at the individual or at the group level? This problem is of importance when one wants to carry out quantitative studies of collaborative interactions, as in this research. Kenny (1998, 1996) discussed those issues by highlighting how the non-independence of observations could be problematic. If the individual is used as unit of analysis, the assumptions of independence are likely to be violated because persons within groups may influence one another (Kenny and Judd, 1986). Alternatively, if groups (e.g. couple, team, organization) are used, the power of the statistical tests is likely to be reduced because there are fewer degrees of freedom than there are in an analysis that uses a person as the unit of analysis. This is the reason why Kenny (1996) promoted multi-level analyses since it allows more flexibility (explanatory and outcome variables can be of any type) and also allows for group and time heterogeneity to be included in the model.

Kenny (1998), however, also points to another simpler method to measure the non- independence of the data using the intraclass correlation7. This index can be viewed as the amount of variance in the persons’ scores that is due to the group, controlling for

7 _{This method can only be applied to nested data: when groups are assigned to levels of the independent}

variable such that every member of a given group has the same score on A with some groups at one level of A and other groups at other levels of A.

the effects of the variable. When the intraclass correlation is not large and total sample size and the group size are small, power is very low. If there is non-independence, then the group must be used as the unit of analysis and if there is independence, the individual may be the unit of analysis.

Therefore, when we analyzed social interactions in groups, we computed intraclass correlations to determine the level of analysis (Kenny, 1998). However, for certain variables, such as the group performance, the team was the unit of analyses because the activity was joint and could not be carried out alone. Moreover, group measures were either measured at the group level (e.g. time to solve the game) or resulted from the aggregation of measures at the individual level (e.g. sum of player’s scores).