Validity and Reliability

According to McMillan & Schumacher (2006, p324), validity is “the degree to which the interpretations have mutual meanings between the participants and the researcher” and thus they agree on the description or composition and the meaning of events. Rule & John (2011, p107-108) view validity (which they refer to as “trustworthiness”) as “the extent to which a case study recorded the fullness and essence of the case reality” and is dependant on “thick descriptions” of the case. However, validity threats exist - alternative explanations or “rival hypotheses” which are ways “you might be wrong” (Maxwell, 2005, p106). In order to minimise the impact of these threats, strategies have been included in the design to try identify and “rule out these threats” to my interpretations and explanations (Maxwell, 2005, pp106-107).

A multi-method approach that included observations, document analysis, questionnaires, interviews, transect walks, auto-photographs and focus groups (Hartley, 2004; McMillan &

Schumacher, 2006) was adopted for the study in an attempt to increase its validity. I also took notes during certain methods to record data.

However, such explanations for validity (such as multi-method approach, member-checking) are general, theoretical and abstract and lack sufficient justification to rule out alternative explanations to conclusions that will be presented. Such explanations also do not consider validity threats of researcher bias and reactivity (Maxwell, 2005).

Researcher bias, also referred to as the subjectivity of the researcher, is responsible for two threats to the validity of qualitative conclusions, namely “the selection of data that fit the researcher’s existing theory or preconceptions and the selection of data that ‘stand out’ to the researcher” (Maxwell, 2005, p108); that is the lens through which the researcher views the data and that influences her conclusions. It is impossible to eliminate this lens. It is therefore important that the researcher makes known her values and expectations and how these influence the conduct and conclusions of the study (Maxwell, 2005).

Reactivity, also referred to as reflexivity in some texts (McMillan & Schumacher, 2006), refers to the influence that the researcher has on data collection (Maxwell, 2005). This influence cannot be eliminated and the goal in a qualitative study is to understand this influence and use it productively (Maxwell, 2005). Reactivity appears to be more of a validity threat in interviews rather than in observation studies as “what the informant says is always influenced by the interviewer and the interview situation” (Maxwell, 2005, p109). In conducting the study and writing conclusions, I had to attempt to understand how I was influencing what the informant said and how this affected the validity of the inferences I could draw from the interview (Maxwell, 2005). A strategy that I employed during data collection that assisted with this understanding was to consult different peer debriefers (or ‘critical friends’) at different points with whom I discussed my preliminary analysis and next strategies and this helped me understand my own position and role in the inquiry (McMillan & Schumacher, 2006)

As researcher, I have attempted to account for these validity threats by describing my role as researcher as instrument, including my philosophical assumptions towards education and research, at the beginning of this chapter. In conducting research and writing conclusions, I also considered tests for validity and the existence of potential threats to conclusions as suggested by Maxwell (2005).

Specific threats of the research was that I was the only field-worker and that data collection took place over two different periods with a long interval between them, phases 1 to 3 (in 2008), and then phase 6 (in 2012). I attempted to rule out these threats through employing various methods and procedures, namely intensive, long-term involvement in the site over the two periods, collecting “rich” data, respondent validation, intervention, searching for discrepant evidence and negative cases, triangulation, quasi-statistics and comparison (Maxwell, 2005, pp109-113).

I observed a variety of situations and events so as to collect data that was “more direct and less dependent on inference” (Maxwell, 2005, p110). Repeated observations and my sustained presence assisted with ruling out premature conclusions and allowed me to “develop and test alternative hypotheses” (Maxwell, 2005, p110). My presence assisted with informants being more open in interviews and candidly sharing their views and insights with me that challenged any conclusions that I may have developed.

I was able to collect “rich” data through my long-term involvement and intensive interviews, which provided “a full and revealing picture of what is going on” (Maxwell, 2005, p110). I had verbatim transcripts of interviews and detailed, descriptive notes of specific events I observed that assisted with the validity of the data I collected ((Maxwell, 2005).

Comments made by members in interviews, transect walks and focus groups were checked by members, a strategy referred to as member-checking or respondent validation (Maxwell, 2005;

McMillan & Schumacher, 2006; Rule & John, 2011). This assisted me in “ruling out the possibility of misinterpreting the meaning of what participants say and do and the perspective they have on what is going on” (Maxwell, 2005, p111) and helped me to identify my biases and any misunderstandings. This feedback helped me gain evidence regarding the validity of my account and the conclusions I drew from the data (Maxwell, 2005).

At times, informal interventions took place where I shared my observations and interpretation of events with the director of the site, while being cognisant that these disclosures could influence his behaviour. I could then trace any changes in behaviour back to these disclosures and the process of that change, which provided corroborating evidence of the disclosures (Maxwell, 2005).

I looked for discrepant evidence and negative cases that indicated a variance or contradiction to the emerging pattern of meanings (McMillan & Schumacher, 2006) and attempted to understand

if the data could be accounted for or if there were defects in such accounts (Maxwell, 2005). In difficult cases, I reported “the discrepant evidence and allow readers to evaluate this and draw their own conclusions” (Maxwell, 2005, p112).

Triangulation (referred to by Rule & John (2001, p109) as “crystallisation”), which is the collecting of information from a range of individuals and settings using a variety of methods to identify patterns in the data (Maxwell, 2005; McMillan & Schumacher, 2006), was used.

However, using a variety of methods does not necessarily rule out validity threats as the methods that are used may be subject to the same vulnerability (Maxwell, 2005); for example, document analysis, interviews and questionnaires may all be subject to self-report bias. This is a real threat for my research as I am the sole researcher and could be blinded by my bias. An awareness of this bias is perhaps a caution for myself when interpreting data and a caution for the reader to be aware of the potential biases that may exist based on the lens through which I view my research. It is also a reminder to be constantly self-disclosing when writing up data.

Quasi-statistics, which refers to “the use of simple numerical results that can be readily derived from the data” (Maxwell, 2005, p113) were used to assess the amount of evidence in my data concerning a particular conclusion or threat. Various counts and numerical results were used for questionnaires and are reported on.

Participants who were employed over the entire research period were able to provide comparisons of the site at an earlier time (phase 6 as compared to phases 1 to 3) (Maxwell, 2005). Their insights were gauged through questionnaires and interviews and assisted in identifying “crucial factors and the effect that these have” (Maxwell, 2005, p113).

Reliability refers to the ability of the study being replicated in order to produce the same results (McMillan & Schumacher, 2006); “when the state of what is being described is consistent, the answers are consistent as well” (Nunnally, in Bickman & Rog, 1998, p344). Rule & John (2011) view reliability as methodological rigour so that the research community can accept the findings and case accounts with confidence. As human subjects are not static, replication of the study should not produce the same results. However, the design of the study can be tested through being replicated. The reliability of this study was checked through an ‘audit trail’ (Rule

& John, 2011, p108) where the data collection method and its purpose were referred to prior to and after use of the method (refer to Appendix H).

There is a story to tell here and as anyone who has been on a journey, I would like to share the tales of my travels with any willing audience. But how should I tell this story - how do I write it up?