Reliability and validity

The reliability and validity of data is an important consideration, especially in a mixed method approach with a case study research design. Analysing different data sets can be done by triangulation: moving between the data sets and gaining knowledge from them, then bringing it together to supplement each other; thus, gaining a multi-dimensional perspective to address a

research question (Feilzer, 2010:12). According to Yin (2013), when a case study’s different methods are designed to collect “overlapping data”, triangulation is possible, and the converging findings may increase the “validity”. However, Richardson and St. Pierre (2008:478) propose that the image for validity should not be the triangle (as in triangulation), because of its two-dimensionality, but rather the crystal (crystallisation), a multi-dimensional shape with many angles. The process employed in this study may be seen as crystallisation. According to Denzin and Lincoln (2008:7), through the crystallisation process, the writer tells the same story from different

perspectives. The process of crystallisation “combines multiple forms of analysis” into a “coherent text”, which also admits its partiality and researcher biases (Ellingson, 2011:4). It can provide “deep, thickly described” interpretations about the occurrence or group studied, and it may include more than one genre/medium, like a narrative, report, or video (Ibid.:10). The current study fits the description, as different data sets were analysed differently, but were combined again to form a thorough

description of the cases. It included videos, reports, and transcribed texts from the videos, which takes on the form of a dialogue between teacher and learner. Furthermore, Ellingson (2011:14, 15) divides crystallisation into integrated or dendritic: integrated crystallisation presents the multiple genres or texts into one “coherent” depiction, like a book, which can be “woven” (the different genres are blended), or “patched” (the different genres are juxtaposed to each other). Dendritic does not combine the different texts or genres into a “single text” (Ellingson, 2011:14, 15). The current study follows integrated, woven crystallisation, as the different texts derived from the different methods of collection, were combined into a single report. The separate texts from which it was derived, are presented as appendices.

There are other measures that can be taken to improve chances of validity and reliability in a study. Quantitative data is seen as having a reliability advantage above qualitative data. Standardised questionnaires, large samples, and the distance between the researcher and the participants strengthen the validity (Lewis, 2009:8). The quantitative aspect of the current study used standardised

questionnaires, and there was a considerable distance between the researcher and the learners, as they were not the researcher’s own learners. The sample was however not large (four teachers; five learners each), which should not have impacted the reliability significantly, as learners are grouped according to their teachers, and being treated as units within a multiple case study. In this study, the mixed method approach may thus be seen as strengthening the reliability, as qualitative findings can be backed by the quantitative data. Furthermore, this mixed method research has an “integrated design” (as opposed to segregated), as both the qualitative and quantitative data attempt to answer the same research question, and is synthesised after being analysed (Sandelowski et al., 2006:34).

In qualitative data collection, certain measures can be taken too to ensure optimal validity. The procedures of collecting and analysing data should preferably be derived from that which has been successfully applied in previous similar studies (Shenton, 2004:64). Observations, being video recorded (as in the current study) is a viable procedure for collecting qualitative data, and employing coding for the analysis has been done in many qualitative studies. Within this general guideline, specific measures should also be taken to strengthen validity. The researcher must, for example, make sure that participants’ words are directly reported, and not a shortened version transcribed by the researcher (Lewis, 2009:9). Video recordings are a way of validating “descriptive data”, although the researcher still has to make sure that a thorough description of the environment is given (Ibid.). In the

current study, the video recordings are thus contributing towards the validity of events described. The video recordings, as opposed to just audio recordings, helped capturing the environment and setting, instead of just words and sounds. Another threat to validity can be the researcher’s interpretation, which should be avoided by asking open-ended questions in the case of interviews (Lewis, 2009:10). In the current study however, events were recorded as they occurred naturally, without any imposition from the researcher. The researcher first transcribed events just as they were, before attempting to interpret them.

Central to the analysis of data, is the theory being used. It is important not to “force” the data into the theory that is used, and that discrepant data be acknowledged and reported, even if it cannot be accounted for by the theory (Lewis, 2009:10). According to Yin (2013:327), findings in a case study may also be connected to the theory by explaining the shortcomings in the literature. In the current study, the quantitative section is less problematic in this sense, as the questionnaires are standardized, and therefore based on a specific theory (SDT), and designed to yield outcomes specific to that theory. The qualitative side is, however, open to outcomes outside of the theories employed. These outcomes were acknowledged, but not discussed or interpreted in-depth; rather opened up for further investigation. The process of crystallisation recognises that one’s knowledge of a topic is never complete, but still provides an insightful account (Ellingson, 2011:12).

Researchers likely possess bias of some sort, which should be reported (Lewis, 2009:10). A

significant cause for bias in this study could be the researcher’s knowledge of some of the instruments that were observed. This could cause the researcher to be more critical towards lessons of these instruments that are familiar to the researcher. The researcher also knew all the teachers personally before commencement of the study, which could cause interpretations of events based on prior knowledge of the person. This familiarity is not necessarily a disadvantage, as Shenton (2004:65) states the early acquaintance with the participant “culture” as a strengthening factor for credibility of findings. In the current study, the familiarity between researcher and teachers could possibly put them more at ease during the lesson observations. The possible bias it could cause was overcome by viewing each lesson as an event occurring at a specific time, and only judged according to what happened at that specific time, backed up only by the written accounts (SDT and enjoyment) from the learners. Participating learners were not personally selected by the researcher, which partially

supports the credibility measure of selecting a sample randomly, which may reduce possible researcher biases (Shenton, 2004:65). The learners were recruited by their teachers according to criteria given by the researcher, and those who fitted the criteria’s partaking depended on their and their parents’ willingness to participate.

A researcher must also be aware of the effect that his/her presence has on the participants; a term called “reactivity” (Lewis, 2009:10). In the current study, this effect could be seen as minimal, as data was collected by means of observations, and not interviews or any other form of participation by the researcher. Within specific video recordings, there are also some clues regarding the impact of the researcher’s presence. It can be seen in how often participants look into the camera, or talk directly to the researcher. All of these clues were included in the transcription of the recordings, and could thus provide possible explanations for some discrepancies between the learners’ written accounts, and what was seen in the observation.