Reliability, Validity and Generalisability in CA Research

As has become apparent from the discussions in this, as well as the preceding, chapters, a CA approach is a radical departure from most other forms of social scientific research. This section will attempt to positioning CA in relation to other

methodologies by explicitly addressing the reliability, validity and generalisability of the CA research process. Surprisingly, few researchers have attempted to directly engage in these matters, and so these discussions will be drawn almost exclusively from Seedhouse (2005).

3.4.1 Reliability

Seedhouse (2005) suggests that the primary issue with regards to reliability in CA research is pertaining to the recordings of the interactional episodes under analysis. How the recordings are selected, and their quality are crucial in ensuring a reliable study. Although no recording can capture everything which occurs, it is expected that recordings capture as much as possible. For example, cameras should, at the very least, capture all of the participants involved in any given interaction. If audio-only recordings are made of face-to-face encounters, then many (visual) aspects of such encounters will be unavailable for analysts to consider, and the end analysis may not be wholly reliable. This issue will be revisited with regards to the present study in Section 4.4.

The repeatability and replicability of analytic findings are central to the

reliability of CA studies. If analysis is found to be solid, and other researchers achieve similar observations and findings, then reliability can be seen to be good.

Since audio/video recordings are seldom made available in conjunction with published CA research, the transcripts included in published reports are the only thing which can be scrutinised in order to test the quality of analyses. Naturally, this places a lot of importance on the quality of the transcript. Although transcripts are merely a representation of the data, and not the data itself, they should ideally provide as much information as possible from the recordings which they represent. Of course, this is not possible to test without access to the original recordings.

However, CA studies typically present a fine-detailed, thorough analysis, which can be pored over and scrutinised by other analysts, in order to check for logical and empirically-supported claims. In this sense then, the source of analysis, as well as the analytic process, is made available for the testing of reliability (Seedhouse 2005).

3.4.2 Validity

Seedhouse (2005) discusses four kinds of validity in relation to CA research: (1) internal validity, (2) ecological and (4) construct validity. Each of these will be briefly described presently.19

Internal validity relates to the credibility of the findings. That is, do the analytic claims fit the data upon which they are based? In order to ensure this, CA researchers strive to maintain the ‘radically emic perspective’ outlined in Section 3.2.2. In trying to make observations based upon participants’ demonstrable orientations and

understandings, internal validity is easily testable. The demonstrable relevance or, and procedural consequentiality of, talk-extrinsic contexts is also important here. Again, analysts are strict in ensuring that they only invoke social contexts or categories if they participants being analysed can be seen to demonstrably invoke those categories . Further, this must be seen to be consequential to the subsequent actions of that

participant and/or his interlocutors (Schegloff 1991). Again, this form of validity can be tested by examination of the data by other analysts.

Ecological validity refers to whether analytic findings are applicable to the ‘real’ everyday, social world. This form of validity is normally concerned with social research which is conducted in experimental and/or laboratory-based settings, which may not be transferable. However, as has been mentioned, one of the key tenets of CA research is that it takes its data from naturally-occurring situations; that is, from settings and encounters which would have proceeded even if a camera was not recording. As such, CA research can be considered as ecologically valid. It should also be noted that on occasions when CA research takes its data from experimental settings, such data ought to be understood as just that. This in itself can produce interesting findings about how artificial settings are socially organised.

Finally, construct validity with regards to CA research adopts a complicated position. While in other forms of, particularly etic, social research, construct validity is concerned with the categories created and applied by the researcher, from an emic perspective, the ‘constructs’ refer to those of the participants being analysed. That is, as has been mentioned, the constructs which the participants demonstrably orient to in their social conduct are also the relevant constructs for the CA researcher. Again, this

19 _{External validity is also included in this list by Seedhouse, but is addressed here under the heading of}

is something which is testable by other researchers, by checking the empirical evidence of recorded social interaction.

3.4.3 Generalisability

Generalisability is concerned with the extent to which analytic observations can be generalised or applied to other settings beyond that of the research. Generalisability can often come in the form of quantification of social phenomena; this is a trend which has been criticised in the past. For example, Zimmerman and West (1975) conducted analyses of cross-gender ‘interruptions’ and conducted additional quantitative analyses to suggest that men interrupted women more frequently. This was heavily criticised by Schegloff, who suggested that such quantification ignores the individual differences which occur with every single episode analysed, and so undermines the entire CA research project (1987). Despite such criticisms, the quantification and generalisation of analytic observations still occur in some recent CA studies (see, e.g. Fox et al 2009; Stivers and Rossano 2010) and are still being levelled with the same criticisms by the same individual (see Schegloff 2009, 2010, respectively). One of the main criticisms levelled at the quantification of CA analyses is that it by necessity requires the labelling of social actions. This goes against the emic perspective of CA research, as the sequential environment in which those actions occur can no longer be considered (e.g. Schegloff 1993).

As Seedhouse (2005) points out, however, this is not to say that CA research can not talk of macro social issues beyond the micro-level details of social interaction. Indeed, those individual cases of social interaction are locally organised and

understood by participants according to their general, normative expectations of the social world. In examining individual cases then, analysts can unpack what these general expectations and orientations are.

In the following section, some of these issues, particularly the lattermost, will be reconsidered in light of how they are formulated as criticisms of CA. Some defences against these criticisms will be made, although some of the limitations of the