CHAPTER 4: DESIGN AND METHODS
4.8 Trustworthiness and Credibility
4.8.1 Reflexivity
An investigation which is based on the assumption that language is social practice and which uses CDA must be subject to reflexive scrutiny of the position of the researcher. The theoretical perspective underpinning this study is that no text is neutral but all are ideologically invested constructions of the world. Texts, including interview data and this thesis, are products of social conditions including relations and are also resources for the process of interpretation (Fairclough, 1989). According to Fairclough (1989), “the only access an analyst has to [the discourse processes] is through her capacity to engage in the discourse processes she is investigating” (p. 167). That is, I must draw on my interpretive procedures or member resources such as assumptions, beliefs, knowledge of language, in order to explain how the participants in the study draw upon theirs. Reflexivity regarding my agency in both the production and interpretation of these texts is therefore necessary and accounted for in the suggested findings. Lee and Otsuji (2009) claim that this is one area where
CDA, as a relatively young analytic method, needs further work. They suggest that CDA, as an etic exercise produced from the outside (A. Luke, 2002; Pennycook, 2001b) needs to consider “which readings, whose readings, and by what warrant, come to constitute authoritative accounts” (Lee & Otsuji, 2009, p. 73).
Reflexivity in the process of production of the data
In many ways, as a teacher educator and known advocate for critical literacy through my publications and workshop facilitation, I constitute part of the official discourse about critical literacy in high schools, as explained in Chapter 1. Reflection on my own investments and professional partiality and on how this might influence data collection and data analysis is therefore necessary. My own relations with research participants influenced the manner in which the interview data are constructed and articulated. The ways in which the teacher’s classroom practice is enacted is also affected by my agency as researcher, and how the participants construct me as such.
To illustrate, the data collected (e.g., teacher talk and classroom practice) are viewed as “episodes” of the particular discourses being identified and analysed in this study. However, teacher talk with a researcher/interviewer is not a regular “episode” in this field of practice and is undertaken for the express purpose of collecting data to analyse. As a result, while syllabuses and unobserved critical classroom practice can be considered typical events and therefore representative texts produced in this field, the research interview is not considered a normal, daily episode in the process of EAL/D teaching. Similarly, the classroom practice that was observed is also not representative of the normal routine. This means that the interviews, in particular, and the observed lessons position me as privileged as I have access to information that might not be shared in the normal course of the day. In addition, this raises issues of the relative status and expertise shared by the researcher and the researched and issues of surveillance.
All of the above required specific reflexive scrutiny on my part, for example, whether the language I used in questioning and responding co-opted or differentiated me in relation to the teachers’ own discourses. Marshall and Rossman (2011) note how easy it is for the inadequately self-reflective researcher to be appropriated by and become complicit in the dominant discourses that serve more powerful members of the community in question. Acknowledging this possibility and identifying
moments where such outcomes may appear in the data (and the analysis) were fundamental to the internal coherence and ethical integrity of this study. An example of this is outlined below when I discuss reciprocity.
Special consideration in terms of identifying the teachers’ discursive relationship to the networks of social conventions surrounding them was also necessary. According to Fairclough (1989), these networks of social conventions – or the orders of discourse – are structured in particular ways in social institutions such as schools. They include which types of practice are included and excluded and how these practices are related to each other (Fairclough, 1989, 2003). The interview texts as data, therefore, might indicate certain discourses embodying particular ideologies that are then constrained by interdependent orders of discourse and the social order at the institutional level. While this may indicate, one way or another, the way in which the schools structure discourses relating to EAL/D pedagogy, my own investments and agency, as indicated above, may impact on the way the participants divulged or concealed their own discursive position in the interviews. It is also important to note that the teachers may or may not be the mouthpieces of the orders of discourse operating in the school. Equally, detailed explication of the member resources (assumptions, beliefs, knowledge) I am relying on to make interpretation and the procedures by which I employ/deploy these member resources are necessary for the critical analyst. Ultimately, as Creswell (2008) reminds us, reflexivity in case study research calls researchers to account for the fact that “their interpretation is only one possibility, and that their report does not privilege authority over other interpretations that readers and participants and other researchers may have” (p. 485).
Credibility was also sought through an awareness of democratising the research relationship. Attempts at reciprocity or “a mutual negotiation of meaning and power” (Lather, 1986, p. 272) were used as a reflexive strategy and a means to deconstruct the author’s authority (Pillow, 2006). It was enacted in a number of ways in a mutual dynamic – from me to the participants and also from participants to me. I attempted to establish reciprocity by:
1. Approaching two sites where I was confident the teachers would demonstrate respected practice, indicating a high level of regard for the two departments and their teachers.
2. Assuring the four teachers in writing that I was not comparing their knowledge and practice to any external sources, including my own. The study draws on a view that “truth” is multiple and varied and takes form differently in different contexts with varying social practices.
3. Assuring one participant, Riva, who was seeking my “approval” for what she was saying, that I wanted to hear her views/ideas/conceptions and not those of policy or textbooks and it wasn’t my place to approve or disapprove of what she was saying. I wasn’t representing the Department of Education or Queensland Studies Authority but was interested in her experience of the way critical literacy has been defined and implemented in her teaching context.
4. Actively seeking member checking (or “participant validation”, Simons, 2009) of transcripts on a number of occasions. See Appendix D and below for further discussion of this process.
5. Publishing an article during the research process about the pedagogies the teachers used and emailing the article to them for comment.
It is expected that “the researcher” will attempt to establish reciprocity in qualitative research that takes a critical or post-structural stance (Carspecken, 1996; Thomas, 1993). What is less commonly documented is “the researched” making democratising moves to equalise the relationship. Two participants attempted to establish reciprocity with me in a number of interesting ways. Firstly, Riva endeavoured to equalise the relationship from the beginning. She took me to a student study cubicle at the back of the library and interviewed me before agreeing to be a participant. She set up the terms of her engagement, as she had been “burnt” before by research that misrepresented her and was determined not to let this happen again. She wanted to know why I was doing this, would she have a right of reply, an opportunity to read a report. Secondly, Riva also took control of the initial interview (March 17, 2010). She sidestepped the opening question of the interview and proceeded with what she wanted to tell me. In this way, she exercised agency6,
6 Fairclough (2003) argues that social agents are socially constrained but not socially
casting me in the novice position, not the “researcher who knows best”. She wanted to “fill me in” on the context and background and how it had been for her, which ended up being a large portion of the interview (over 5 minutes of her talking). Somewhat nervously, as a novice researcher, I let her go and didn’t stop her to get back to my interview question schedule. I was fascinated by the information she was providing and by the way she managed the direction of large parts of the interview. The extract below provides evidence of Riva’s reciprocity move in the initial interview (see Appendix E for transcription style).
(At the very beginning of interview 1, lines 1-14)
JA: Thanks, Riva. So first of all can you just tell me a little bit about how the ESL syllabus is taught at this school?
Riva: (5.0) Can I start by contextualising why it’s taught here at this school and all of that?
JA: Yes, yes, anything you want to tell me.
Riva: We have quite a cohort of ESL students, it varies but in Year 11 and 12 it tends to be around the 25%, sometimes it’s more and sometimes it’s less, but it’s a significant cohort of ESL students and this school always acknowledged that they needed ESL support to do that. The (1.0) English Head of Department some years ago, when the 2003 or 2002 English syllabus came in, took the students into consideration when she prepared the work program and the ESL teachers were involved in that. Um, NEVERTHELESS.... [laughs and pauses for emphasis]
JA: [soft laugh]
Riva: The school found that that syllabus was overly demanding in the critical....
(The interview continued with me nodding and agreeing and not asking another question until 5.03 mins into the interview).
In addition, in Margot’s first lesson, she was speaking to me “off camera” (literally behind the camera, standing beside me) as an aside while I was video recording the class. Her talk with me was informal and chatty, as a colleague, not as a researcher; insider-talk about the teaching methodology she was using. She used a hushed voice as if the camera microphone might not pick it up. I didn’t want to say
too much for fear of disrupting the class, so I nodded and “mmmed”, and waited until she was ready to resume the role of teacher. Margot included me in her classroom talk as a colleague, repositioning me not as a distant researcher but someone who understood her decision to do “chalk and talk” with a class of 28 students in a small room on a hot Friday afternoon. [This lesson was at the outset of the research process and before we had really established rapport. Prior to data collection, I had met Margot on a few occasions at professional development sessions and I knew she had a reputation as being a respected ESL teacher but we had not spoken at length and had no professional collaboration until now.] To me, this suggests Margot had embraced that I was doing research “with” and not “on” her (Pillow, 2006. p. 179).
The above examples of participant-afforded reciprocity were at times uncomfortable and required me to be watchful for and sensitive to such cues in the data collection process, and to adapt moment by moment as these moves played out in the interaction often in unexpected ways. However, as Pillow notes (2006), reflexive strategies on the part of the researcher may in fact reinscribe the power relations they attempt to address, for example, announcing our politics and ceaselessly interrogating them (Denzin, 1997) or the inclusion of longwinded, self- confession tales about our position as researchers serve to privilege us textually. I see my methodological tool of reciprocity as a way of further understanding the “complicit relationship” (Pillow, 2006, p. 192) reflexivity has with researcher power.
Member checking, as a crucial technique for strengthening credibility (Guba & Lincoln, 1989, 1994) and prominent in democratic case study that has political intent (Simons, 2009), is often overlooked as it is time consuming in an already time- consuming activity. I use the term “respondent validation” (Simons, 2009, p. 131) to refer to the process of participant checking of the accuracy and adequacy of my representations and interpretations of their experiences and statements.
Validation of raw interview transcripts (direct from transcription with minimal mark up by me) occurred after all interview data transcription had taken place (three interviews per teacher). Teachers were invited to comment on the transcriptions before detailed analysis was undertaken and were given over 12 months to do so. See Appendix D for a sample of teacher replies (via email) regarding the validation process. Teacher responses were used to ensure I represented their knowledge and
practices as accurately as possible, given my own partialities and the partial nature of what is knowable (Lather, 1992; Pillow, 2006).
Respondent validation is problematic too and may in fact mask a researcher’s power over a participant. As Trinh (1991) points out, the validation share of power is often given to participants, and not taken. Reflexivity, then, occurs from unequal power relations. As a result, one participant’s checking in this study is worthy of note. Riva, a busy senior teacher, took the time to read all three interview transcripts and return them to me with “track changes”. She asked me to delete certain things, which I did before analysis, and she also clarified certain words that were inaudible to the transcription service and to me. She also wrote comments in the margins of the transcripts which enabled further conversations with me via email. I call these “conversations in and at the margins” of data collection and they play a significant role in democratising the relationship between the researcher and the researched (Carspecken, 1996; Pillow, 2006), affording the participant additional opportunities to exercise power. In this way, a literal dialogic relationship with this participant, consistent with critical case study method (Carpsecken, 1996; Cresswell, 2008) was achieved, to a degree, adding an additional element of validity to this study.
Credibility was also enhanced through the stimulated recall procedure outlined in section 4.4.5. Participants were invited to review the video footage of their own teaching of a lesson of their choice, via the stimulated recall interview enabling them to comment on and clarify the contents before analysis began. They were also invited to discuss the findings. This validation process was used to ensure I represented the participants’ perspectives as accurately as possible under the circumstances (Guba & Lincoln, 1989). This is particularly necessary in a critical case study that claims to have had literal dialogue with participants and to be representing those participants for emancipatory purposes, or the process of separation from constraining modes of thinking about or acting with EAL/D learners that limit perception of and action toward realising alternative possibilities (Thomas, 1993).