A convergent evaluation methodology

Both kaupapa Māori research and action research practitioners stress the importance of evaluating the outcomes of the research from the perspective of the participants (Lincoln & Guba 1985; Bishop 1996; Harmsworth 2001,2005; Cram 2016; Bradbury et al. 2017; Carlson, Moewaka Barnes & McCreanor 2017). As such, methodologies for evaluating the trustworthiness and rigour of collaborative research projects from both kaupapa Māori and Western perspectives were

investigated. Five evaluation criteria offered by Russell Bishop (1996, 1998, 1999, 2005) and further developed by others practice (Cunningham 2000; Te Maro 2010; Haar 2011a,b) for kaupapa Māori research and four offered by Lincoln and Guba (1985) for action research were identified as an appropriate convergent evaluation methodology. The next section outlines the evaluation criteria of the two approaches.

3.5.1

Kaupapa Māori evaluation

The power dynamics inherent in research relationships are highlighted in a kaupapa Māori research framework developed by Russell Bishop (Bishop 1996,1998,1999,2005). He identified five power issues in examining the extent to which ethical research adheres to a kaupapa Māori framework. This is sometimes referred to as the IBRLA framework (Initiation, Benefits, Representation, Legitimisation,

research? (iii) Who is represented in the research? (iv) Who legitimises the analysis and

interpretation of the information? and (v) Who is accountable in the research? Addressing such issues of process and power in kaupapa Māori research is as vitality important to the results, as what is discovered or developed.

The IBRLA framework has subsequently been built upon with additional questions developed according to the context of the research (Te Maro 2010; Haar 2011b,a; Macfarlane & Macfarlane 2018). Haar (2011a, 2011b) highlights implications for undertaking research in Māori communities (initially identified by Irwin (1994)), these being: that research should be (i) “culturally safe”; (ii) have scientific rigour; (iii) be undertaken with a Māori worldview; (iv) have a goal of empowerment; (v) have a whānau focus; (vi) be aligned with the Treaty of Waitangi, and (vii) be undertaken by a researcher with empathy for Māori (Haar 2011b). Cunningham (2000) identified five goals or outcomes of kaupapa Māori relevant to addressing issues of process and power in kaupapa Māori research, these being (i) Māori epistemology; (ii) the Treaty of Waitangi; (iii) Māori development; (iv) capitalising on the investment in Māori medium education; and (v) internationalisation.

Taking Bishop’s (1996, 1998, 1999, 2005) framework as a starting point and further populating it with imperatives framed as reflective questions identified in the works cited (Cunningham 2000; Te Maro 2010; Haar 2011b,a) an evaluation methodology is presented to evaluate the success of the research from a kaupapa Māori perspective (Table 2).

Table 2: Evaluation methodology of research success from a kaupapa Māori perspective

Criteria Related questions

Initiation _{Who has conceptualised and initiated the research? ‡ #}

How did Māori participate in the conceptualisation and initiation process? # Who has designed the research? †

Do the research participants have power to be decision-makers? ‡ How was the agreement to proceed with the research achieved? #

Benefits Does the research support Māori and Māori endeavours and aspirations? ‡ Will the research serve to empower Māori? §

Will the research lead to Māori development? *

Will the research capitalise on the investment in Māori medium education and build research capacity? *

Source: * = Cunningham (2000), † = Bishop (2005), ‡ = Te Maro (2010), § = Haar (2011a, 2011b)

3.5.2

Action research evaluation

There is no single set of criteria used to evaluate the rigour of qualitative research, but rather criteria for 'good' research vary across epistemological standpoints, research traditions and methodologies (Cohen & Crabtree 2006). Lincoln and Guba (1985) argue for alternative criteria for judging the quality of qualitative research which betters reflect its underlying assumptions. They favour the concept of ‘trustworthiness’ for evaluating the worth of qualitative research. They identify four criteria, these being credibility, transferability, dependibility and confirmability. If these are met, then they contend the research can be considered to be trustworthy. Following Bishop’s example for kaupapa Māori, a set of reflective questions has been developed across criteria from which to evaluate the trustworthiness of research from an action research perspective (Table 3).

Will the research outcomes have international Indigenous applications? *

Representation Whose voice, interests, needs and concerns are heard? ‡ Does the research have a whānau60 _{focus? §}

Who undertakes the work? †

Legitimisation Is the data legitimate, valid and robust from a Māori worldview / based on Māori epistemology? * ‡ §

Is the research aligned to Te Tiriti o te Waitangi / the Treaty of Waitangi? * § Does the research meet the test of scientific rigour? §

Accountability Who shares in the knowledge? Who distributes it? Who controls the knowledge? ‡ Is the research process “culturally safe”? §

Table 3: Evaluation methodology to evaluate the trustworthiness of research from an action research perspective

Criteria Related questions

Credibility61 _{Does the research team have confidence in the 'truth' of the findings? “The} participants are the only ones who can legitimately judge the credibility of the results” (Trochim 2006). Has the research been member checked? (Lincoln & Guba 1985). Member checking is seen as a most crucial technique for establishing credibility (Cohen & Crabtree 2006). This is when the authenticity of the data, analytic categories, interpretations and conclusions are checked with participants from whom the data were obtained. Has the research been peer debriefed? (Cresswell 2014). Peer debriefers review and ask questions about the study so that the account resonates with people other than the researchers.

Transferability Can it be demonstrated that the findings have applicability in other contexts? Qualitative inquiries are often specific, and the findings applicable to a particular time, setting, situation, or a small group of individuals. Cohen and Crabtree (2006) contend that by describing a phenomenon in sufficient detail (thick descriptions) one can begin to evaluate the extent to which the conclusions drawn are transferable to other situations and populations.

Dependability Can it be shown that the findings are consistent and could be repeated? Has an external audit trail been carried out? “External audits involve having a researcher not involved in the research process examine both the process and the product of the research study. The purpose is toevaluate the accuracy and evaluate whether or not the findings, interpretations and conclusions are supported by the

data”(Cohen & Crabtree 2006).

Confirmability62 _{Has the research involved reflexivity to attend to researcher bias? Can the}

research be triangulated? Can it be shown that there is a degree of neutrality to the research, with the findings of the study shaped by the respondents and not researcher bias, motivation, or interest? (Cohen & Crabtree 2006).

61 _{Credibility is the parallel concept of ‘internal validity’ in quantitative research} 62 _{Confirmability is the parallel concept of ‘objectivity’ in quantitative research}