Issues of validity and reliability

‘Bias’ is a key issue in establishing the trustworthiness of any scientific research (Robson, 1993). As discussed above, however, all research is value-laden to some extent. Based on this premise of no research being completely unbiased, Scott and Garner (2013) find that in qualitative research the issue is not “whether or not the

94

study is biased, rather the concerns on how to deal with bias in the research” (Scott & Garner, 2013, p. 68). This includes issues such as the degree of bias, clear consciousness on the bias, and moreover, the potential impact of bias in data collection and analysis.

As a strategy for these concerns, the researcher should pursue “transparency: and “falsifiability” in the research (Scott & Garner, 2013, p. 69). Transparency means the researcher needs to clarify their point of view, and take steps to ensure they record information accurately. This measurement should be pursued through all steps of data collection and analysis. Falsifiability means the researcher should be prepared for “inconvenient facts,” where they may encounter results that may diverge from their initial hypotheses. This is particularly necessary during analysis, and requires flexibility in the research design and openness in the researcher’s mind.

Five sets of approaches were used in this research to enhance transparency and to test ‘falsifiability’ during data collection and analysis. These included (i) prolonged engagement and persistent observation in the field; (ii) triangulation; (iii) construct validity and systematised reflexivity; (iv) face validity; and (v) catalytic validity.

(i) Prolonged engagement and persistent observation in the field

Prolonged engagement and persistent observation in the field is required to establish the trustworthiness of data analysis. This technique is advised by Lincoln and Guba (1985) and used by many qualitative researchers until now, including (such as Ely, 1991; Golafshani, 2003; Loh, 2013). Data were collected using semi-structured interviews with individual participants in focus groups, life

95

narratives of Hatakeyama and two other participants, and participant observations. In addition to the fieldwork, communication with participants was continued through phone calls, emails and letters, and used to report back to the participants about collected data and to facilitate negotiation. During the research period, data was collected in Mombetsu, Sapporo and Tokyo during four visits: 1-28 February 2012; 10-15 February 2013; 13-23 December 2013; and 22 December 2015 - 8 January 2016.

(ii) Triangulation

Triangulation is an essential technique that ensures the trustworthiness of data and its interpretations. It is a technique used to analyse data by integrating multiple data sources, methods and theoretical schemes, or obtaining information relevant to a topic or issue from several participants (Robson, 1993). Using multiple sources of data through multiple data collection and interpretation techniques help to minimise researcher bias. In this study, triangulation was used in three ways. The first was conducted by crossing between data and theory to find the relevance and the gap. The second was the analysis of data obtained through different data collection techniques, such as combining semi-structured interviews, life narratives and participant observations, to understand differences in the reactions of the same participant in different settings (e.g. change in Hatakeyama’s words and behaviour during a group in a meeting room and during individual meeting in his life context). The third was the analysis of data by involving multiple participants. This helped particularly in the interpretation of words and hidden meanings in Hatakeyama’s behaviour.

96

(iii) Construct validity and systematised reflexivity

Construct validity and systematized reflexivity is a constant dialogue between data and theory (Lather, 1986, p. 270). It is necessary to understand how data based on the experience of participants, relates to or challenges an a priori theory. Construct validity was integrated into data collection and analysis in Stage 3, 4 and 5 of Part I and II. The progress of the study on my educational practice at MSN, especially with Hatakeyama, was systematically analysed. In particular, how the critical EE/ESD related to the process of emancipation and empowerment of the local community, particularly the marginalised stakeholders like Hatakeyama.

(iv) Face validity

Face validity is referred to as ‘member checks’, and refers to people who are not necessarily experts in what the researchers are studying (Lather, 1986). It recycles descriptions, emerging analysis, and conclusions back to the participants for their opinions and advice. Member checks were done in Stage 3, 4 and 5 of both Parts I and II in this study. All transcribed data and interpretations were circulated to key MSN members for the verification. Analyses were also checked by researchers in Japan working in the area of local community-based ESD. Data collection and analyses were conducted during 2013 to 2014 at the University of Tokyo. Extensive advice on the analysis process was received from one of the supervisors and fellow higher degree students. The member checks were also conducted by the researchers at Tokyo Gakugei University, and Tokyo University of Technology and Agriculture, who examined the relevance and effectiveness of the analyses in the broader local community context.

97

(v) Catalytic validity

Catalytic validity refers to the extent to which the research process reorients, focuses and energizes participants toward knowing themselves and their situation to transform it (Lather, 1986). Catalytic validity is referred to a process of consicentisation. It was an essential part of this study, which aimed to construct the emancipatory empowerment of participants through the critical ethnography research design. Catalytic validity was confirmed in Stage 5 with Hatakeyama on how he found the relevance and effectiveness of the conceptual framework that was theorised through the research.