Validity and reliability of the study

This is a case-study based enquiry in that it “investigates a contemporary phenomenon within its real-life context” (Yin, 2003, p.13). A multiple case study approach was chosen with the unit of analysis (Yin, 2003; 22-23) for each case being a single activity, or short sequence of learning activities, undertaken by a fixed cohort of students. These cases were treated separately to enable comparisons to be made between cases, both by repeating the same learning activity with different groups, and by attempting different learning activities. It was also hoped to analyse how the experiences of learners differed for those that had been engaged in the mediated environments for a short time and those who had more exposure. Merriam (1998; 40) states that “the inclusion of multiple case studies is ... a common strategy for enhancing the external validity or generalizability of your findings”. Treating each individual learning activity as a separate case also enabled interim analyses and findings to be made during the study. These analyses fed back into the design of the qualitative and quantitative tools during the study, changed the focus of the study and prompted the parallel investigation of students’ resistance.

The multiple case studies offered some variety, there were three different subject disciplines amongst the case studies (drama, counselling and technology) and the students had different degrees of experience of virtual worlds in an educational context (none, three to four hours, eight to ten hours). The observations of students in three case studies with no previous experience of the virtual world enabled literal replication to be conducted (Yin, 2003; 47). A framework proposing the idea of progressive presence was developed after the first case study, which was then used to predict the experiences of students that had spent more time in virtual worlds (i.e. a theoretical replication [Yin, 2003; 47]). A major revision of the framework incorporated the concept of virtual body image and virtual body schema, and this was used to incorporate the findings of the fourth case study. Therefore, although the cases could not be planned or selected to fulfil the replication approach for multiple case studies (Yin, 2003; 50) in effect they did conform to the replication model

109 to some extent. A larger number of cases with students with longer term experience of the virtual worlds would have been required to fully conform to this approach, but could not take place due to insufficient availability of cases with more experienced students.

Of the five case studies, the Red, Magenta and Blue studies were a mandatory part of the course, so there was no selection of participants. However, for these sessions, involvement in the evaluation was optional, due to the ethical constraints on the research study. This may have led to a response bias in the data. In one of these three cases (Red) the response rate for the questionnaire was high (14 responses out of 15 participants), so has more reliability. Of the two remaining cases, participation was optional; these were set up as extra-curricular activities, so the sessions ran with self-selected participants.

Internal validity

The study is interpretivist throughout. Most of the data come from the students’ perceptions of their experience, and are reliant on their ability to articulate those experiences and to do so accurately. The remainder are observations of the sessions.

Elements intended to improve the degree of internal validity of the study include the development of a conceptual framework to systematise the process of gathering and analysing the data. This enables points of correspondence with the literature and with other data to be identified and organised. The use of conceptual frameworks has the disadvantage, however, of being self- fulfilling, in that the data that fit within it will be recognised, and that which does not will be excluded (Smyth, 2004). The internal validity of the study is also reinforced by conducting both a qualitative and a quantitative element. This provides triangulation, particularly on the role of presence in the experience of learning (Merriam, 1998; 204).

The process of developing categories of resistance through a grounded approach also has limitations in its validity in that “the category scheme does not tell the whole story” (Merriam, 1998; 188). The validity of these categories is given more validity through linking the categories together in “a meaningful way” (Merriam, 1998; 188) and by identifying the equivalents to these categories in the literature. The models developed offer an interpretation of factors that may influence learning in these environments; but these models need to be further tested in the future and there may well be

110 alternative explanations to the observations. More cases and more data are required to provide greater validity.

Reliability

Reliability of the findings is demonstrated by employing a replication approach across the case studies (Yin, 2003; 46). Similar questions were asked both in the qualitative and the quantitative studies in each study and the data gathering methods were repeated as much as possible. The case studies differed in the subject disciplines involved, the universities at which they were conducted, and whether the students had self-selected themselves to be part of the study. These differences meant that the numerical breakdown of different responses varied (for example the proportion of students who were dissatisfied with the learning experience), but the nature and range of the experiences were repeated across the case studies.

External validity

External validity, or generalisability, is the degree to which the findings of a study can be generalised to other situations (Merriam, 1998; 207). The conceptual framework could be applied to a range of different activities using mediated environments, since the main categories draw on two models (Activity Theory and Communities of Practice) that are themselves applied in a range of situations, although details may change in different environments, for example the role of identity may not be as strong in environments that are distal rather than simulacral.

Originally the aim of the study was to examine experiences across a variety of mediated environments. Due to a lack of available opportunities for case studies, a case study employing telepresence was only available for the pilot study, and for the main study only cases employing a single virtual worlds platform (Second Life) were available. This limits the demonstration of external validity across different platforms and hence, although the conceptual framework is applicable to mediated environments, the conclusions made are only applied to virtual worlds.

The experiences of students when first exposed to Second Life remained consistent across the cases, and this indicates that these are generalisable to other cases involving the initial sessions of students in Second Life. The functionality of most virtual worlds have, by definition, the same

111 essential elements (navigation, creation of an avatar, communication, etc.) and the experiences of using Second Life in this study are therefore likely to be similar in the use of other virtual worlds too. The experiences of those students with longer term involvement with a virtual world may have not been shown to be generalisable, since these were not replicated within the study. Different subject disciplines or different student cohorts may lead to different results. The general applicability of the final model developed is therefore conjectural when the effect of longer term involvement in virtual worlds is considered.

3.3 Ethical considerations

3.3.1 Risk to students

In webconferencing, the technology links one site to another in a bounded and managed environment. In virtual worlds the learning takes place in a larger social networking environment in which a range of cultures and communities, with a range of behaviours, interact. Bringing students into these worlds therefore runs the risk of exposing students to these behaviours, and also in exposing members of that world to students, some of whom may not be aware of the rules and conventions of that space. Although the learning activities took place largely within educational areas, part of the ethics of teaching within these environments is to inform students of the potential for being exposed to people other than the students on their course, and for their responsibilities with regard to the people they may meet there. The participants in virtual worlds assume different sets of social rules depending on whether they see the world as a social one with real relationships and a sense of embodiment, or whether they see it as simply a game-space with no consequences (Ryan and Childs, forthcoming). This can lead to experiences of griefing and abuse, perhaps without the abusers being aware that their behaviour may be seen as such.

Other ethical issues are caused by the illusory nature of virtual worlds. Pasquinelli (2010; 201) states that exposing people to the illusion of non-mediation (i.e. virtual presence) and the illusion of being in the virtual world (i.e. embodiment) is ethically problematic in that one is exposing participants to something illusory and therefore intrinsically deceptive.

112 Grimes et al (2010; 79) also list the “psychological importance and emotional attachment to virtual objects” as an important ethical consideration, for example the identity a student may have built around their avatar, and its appearance and social connections, may have become important to them. There therefore needs to be provision for the student to retain their avatar and any objects acquired for their avatar at the end of the learning activity.

These factors, however, only emerged during the study as elements to be considered and were not included in the original ethical considerations. The area in which this research did require ethical consideration was in regard to the privacy of the participants in the research. In the ethical guidelines published by BERA (2006) the section on privacy states:

The confidential and anonymous treatment of participants’ data is considered the norm for the conduct of research. Researchers must recognize the participants’ entitlement to privacy and accord them their rights to confidentiality and anonymity (BERA, 2006, 8)

Grimes et al (2010; 79) add to this by drawing attention to the “Importance of privacy, confidentiality, anonymity and reputation of users and their avatars”, i.e. that a research subject’s online identity requires the same due consideration as their offline one.

However, ethical behaviour also requires that any data be properly attributed to their source, for example, the Association for Institutional Researchers Guidelines (2002) state:

The institutional researcher shall … follow scholarly norms in the attribution of ideas, methods, and expression and in the sources of data. (Association for Institutional Research, 2002, 2)

These two statements are not in conflict in the real world, since a conversation, or an interview, is not usually public and the content is not considered to have been published. However, within a chat room or discussion board, the divide between personal conversation and public document is blurred, or non-existent, and this holds true for virtual worlds due to the “inconsistencies in the classification of public and private spaces in virtual worlds” (Grimes, et al, 2010; 79).

The decision was to anonymise all the data, and to inform the students that this would be the case. Students were given a consent form to sign, identifying that the data would be used in the form of short quotes from transcripts for the PhD, for evaluation of the Theatron project (where this was applicable) and in any other publications and presentations. Names were removed from all

113 quotes and replaced by a letter plus the anonymised name of the institution. This anonymisation was also carried out for students’ personal avatar names. An ethical approval form was submitted to the Warwick Institute of Education as part of the upgrade process.

3.3.2 Risk to learning

The deployment of virtual worlds in higher education is subject to many barriers, due to lack of reliability of the platform itself, the low specification and lack of availability of computer equipment in many universities and the time taken for students to become accustomed to using the interface. This means that the risk associated with conducting learning in these environments is sufficient that the lecturers participating in this study did not want students’ degrees to be adversely affected if the technology could not be implemented effectively or if the environment proved to be one that was not effective for learning. To minimise these risks, sessions were either conducted as extra-curricular activities (the Yellow and Green case studies) or were very few sessions thereby reducing the risk to the overall learning goals if the virtual worlds sessions were not effective (the Blue, Magenta and Red case studies). Pasquinelli (2010; 209) identifies a further risk to learning, which is that the learning activities may not be treated as serious academic activities because they are taking place within a virtual environment, with a consequence of undermining the students’ ability to learn.