Validity, Reliability and Reflections on the Data

The accomplishment of validity and reliability is essential and crucial to any research project. Its importance is what Kirk and Miller (1986) and Silverman (2001) point out “the issues of reliability and validity are important, because in them the objectivity and credibility of (social science) research is at stake” (cited in Peräkylä 2004:283).

However, it should also be noted that the questions of validity and reliability take different forms in different qualitative methods (Peräkylä 2004:284).

According to Peräkylä (2004), CA’s unique aim to investigate the talk in interaction in its own right has made the issue of validity in CA distinct from other qualitative methods. In CA research, the accomplishment of validity is closely link to its

fundamental research perspective, that is, an emic or participant’s perspective. The close link between the issue of validity and the emic perspective is explained by Seedhouse (2004, p.255):

“However, the crucial point in developing an emic perspective is that it is the participants' perspective, rather than that of the analyst’s. Conversation analysts know what the participants' perspective is, because the participants document their social actions to each other in the details of the interaction by normative reference to the interactional organisations. We as analysts can access the emic perspective in the details of the interaction and by reference to those same organisations. Clearly, the details of the interaction themselves provide the only justification for claiming to be able to develop an emic

38

demonstrated by the interactional detail without destroying the emic perspective and hence the whole validity of the enterprise".

In this thesis, the validity is demonstrated through providing a turn by turn analysis of how the participants make sense of each other’s talk to co-construct a word search sequence, that is, the participants themselves make the claim that they are involved in such an activity through a “next-turn proof procedure” (c.f. Peräkylä 2004). The validity is then tested by the reader’s own analysis. Having discussed the issue of validity in CA, I am now moving to another important research issue: reliability. A key question in reliability is that “whether the results of a study are repeatable or replicable” (Bryman 2001, cited Seedhouse 2004, p.254). Seedhouse (ibid.) provides a clear explanation of how this is achieved in CA studies. He states:

“Many research methodologies do not require presentation of the primary data on which a study is based in publications about that study, and hence the reliability of major sections of the analyses of that data is not available for scrutiny. By contrast, it is standard practice for CA studies to include the transcripts of the data they employ and increasingly to make audio and video files available electronically via the Web. Furthermore, the conversation analyst makes the process of analysis transparent for readers. This enables readers to analyse the data themselves, to test the analytical procedures which the author has followed and the validity of his/her analysis and claims. In this way,

conversation analyses are rendered repeatable and replicable to readers. Also, it is standard practice for CA practitioners to take their data and analyses to data workshops and to send their work to a number of other practitioners for comment before sending them for publication”(pp.254-255).

Based on Seedhouse’s claim, the reliability of this thesis has been satisfied to some degree. First of all, I have made the transcripts of the data and their analysis accessible for other researchers to challenge. Secondly, I presented my data in three different data sessions in 2008 and 2011 in MARG (Micro Analysis Research Group) at Newcastle University. In each session, the presented recordings and transcripts were reviewed and examined by other researchers.

The accessibility of the data and the transparency of analysis certainly is CA’s strength in terms of its validity and reliability; however it does not warrant the reliability of observations. Peräkylä (2004) points out that it is important for the researchers to pay more attention to the technical quality of recordings and the accuracy of transcripts while producing them.

39

Efforts were made in this thesis in trying to ensure the quality of recordings and transcripts. The quality of the recordings was good in terms of the clearness of the sound and images; however, it was a great pity that most of the time the teachers were not in the shot. There were two main reasons. First of all, one teacher did not want to be filmed so in the recording he was always out of the shot. This was beyond my control. Another reason was that as I had only one camera and with only one camera it was impossible to include all of the participants so I made the decision to focus on the students. As I started to transcribe the recordings and analyse the data, it soon became clear that the limited access to the teachers’ embodied behaviours had made it difficult to make some analytic claims. Thus the current data is “imperfect” in that not all transcripts of the word search sequences include all the relevant and necessary actions of the teachers. Thus it is clear that when recording teacher-fronted classroom

interactions, it is necessary to set up at least two cameras in order to catch the relevant non-verbal information from both the teacher and the learners. If I had to do this study again, I would certainly place two cameras to avoid missing relevant non-verbal information. It is worth noting that although using multiple cameras can increase the adequacy of the recordings, the researchers need to be aware of their obtrusiveness in the classroom. It seems that it remains a challenge for researchers to maintain the accuracy of the recordings and to avoid their intrusiveness at the same time.

The issue of “observer’s paradox” (Labov 1972) in this study requires some reflection. In some classrooms I was permitted to stay but I tried to act as unobtrusively as

possible. For example, I chose a less intrusive place to sit in and I did not make any attempt to move the camera to follow the movement of the participants while recording. On the occasions when I was not permitted to stay in the room, I set up the camera and left. Even though efforts were made to reduce the intrusiveness, the presence of the researcher and /or camera still made some influence on the participants. For example, in one of the recordings, the participants were making a joke about who is going to be the host of today’s show when they saw the camera. Fortunately, these instances were rather short. It seemed that the participants may have been initially aware that they were being recorded, but they forgot about it soon after they started to engage themselves in the lessons. However, due to there not being any prolonged observation of these classes before the recording was made it is difficult to determine whether the participants changed their usual behaviour in the classroom or not.

40

3.9 Conclusion

This chapter has detailed the methodological framework chosen for this study, i.e. Conversation Analysis. It has shown that how CA provides a useful set of analytic tools to address the questions being investigated in this study. Overall information about the data collection tools and procedures has also been presented. Additionally, transcript procedures, including some transcription conventions used for Chinese and non-verbal information have been introduced. Finally, various research issues for the study

including ethics and permission for the data collection as well as validity and reliability have been considered. The following chapter, Analysis, will present detailed analyses of the examples selected from the word search collection. It will demonstrate the result of the application of the methodology and also reflect the theoretical stance taken in this study.

41