Rigor: Reliability and validity

Design-based research is conducted in the messy and complex classroom context and is indeed specific to its given context. For this reason, it may become difficult to assert causality. Furthermore, DBR does not have any predefined ‘method’ with which to proclaim rigor. Nevertheless, rigor is as important in DBR as any other method of research. Being rigorous in design-based research involves having concern for validity and reliability. Briefly, “validity concerns whether we really measure what we intend to measure. Reliability is about independence of the researcher” (Bakker & van Eerde 2015: 24). To overcome these issues, “reliability of findings and measures can be

103

promoted through triangulation from multiple data sources, repetition of analyses across cycles of enactment, and use (or creation) of standardized measures or instruments” (Design Based Research Collective 2003: 7). Validity and reliability, as relevant to DBR, are discussed as follows.

Validity can be divided into internal and external validity. The issues of internal validity “refers to the quality of the data and the soundness of the reasoning that has led to the conclusions” (Bakker & van Eerde 2015: 24). To this end, transcripts can provide a meaningful context, and can be compared against other collected data such as field notes and interview data.

External validity refers to the extent to which results are generalizable, or how a designed intervention may be useful to practitioners in other educational contexts. Addressing external reliability means “framing issues as instances of something more general [and] present the results (instruction theory, HLT, educational activities) in such a way that others can adjust them to their local contingencies” (Bakker & van Eerde 2015: 25). A further issue is that of transferability, which is the need to describe how what is better understood through one iteration can be applied in future iterations and, depending on the scope of the research, in other contexts.

Likewise, reliability can be divided into internal and external reliability. Internal reliability refers to “the degree of how independently of the researcher the data are collected and analyzed” (Bakker & van Eerde 2015: 25). This may be addressed using the following methods:

 Discussing data with colleagues.

 Noting the agreements and disagreements when data is discussed among researcher and colleagues.

 When data is coded, it should be ensured that the sampled data be large enough. Finally, external reliability refers to how “the conclusions of the study should depend on the subjects and conditions, and not on the researcher” (Bakker & van Eerde 2015: 26). This is usually described as virtual replicability, in that it is incumbent on the

104

researcher to describe research so that it is transparent and trackable to the reader and failures, as well as success are noted.

Schwandt et al. (2007), however, asserts the limitations of applying traditional criteria to judge rigor in the controlled laboratory context to judge inquiry that is based in real world settings. Yet, the move away from the use of traditional criteria to judge rigor have often been criticized as a threat to rigor itself. Nevertheless, it is necessary to admit both the impossibility of applying such criteria and the need to devise a new set of criteria that allow for the maintenance of maximum rigor. The following draws on Guba and Lincoln’s (2007) attempt to address this problem in two ways. They first offer a trustworthiness criterion which “‘analogs’ to “scientific” understandings of conventional notions of internal validity (credibility), external validity (transferability), reliability (dependability), and objectivity (neutrality)” (Schwandt et al. 2007: 12). In tandem, Guba and Lincoln also offer four authenticity criteria: fairness, ontological authenticity, educative authenticity, and catalytic authenticity. Both trustworthiness and authenticity will be will be outlined here respectively.

As noted, trustworthiness may be viewed as analog to the traditional concept of rigor. Within the framework of trustworthiness, Guba and Lincoln (2007) first state the need to address issues of credibility, which may be done in a variety of ways, i.e., through a. engaging with phonemena and respondants in a prolonged way b. continued and careful observation of salient elements of inquiry c. triangulating a variety of data d. engaging in discussion of the inquiry with disinterested peers e. looking for cases that are negative to emerging insights, and f. constant checking of information from all stakeholders.

Second, regarding transferability, Guba and Lincoln recommend developing a narrative with the use of thick descriptive data that may be cross checked by others. Finally, inextricably linked are the criteria of dependability and reliability. As research is externally audited, the examination of research results represents judgment of dependability. Likewise, when the product of the research is examined judgment of comfirmability may be made.

105

Alongside a framework of trustworthiness, Guba and Lincoln assert the need for a further criterion of authenticity to be to be included in naturalistic research conducted in real world settings. Within the framework of authenticity, criteria of fairness, ontological authenticity, educative authenticity, and catalytic authenticity will be respectively discussed as follows.

The criterion of fairness represents the attempt to achieve balance among the pluralistic values of an inquiry. To achieve fairness, a. the different values and their underlying system need to be outlined, particularly when conflict arises within the research. Subsequently, b. there is a need to negotiate “recommendations and subsequent action, carried out with stakeholding groups” (Guba & Lincoln 2007: 21), done so in open and equal manner. c. The need for fairness also means fully informed consent is necessary between all parties to the evaluation of the research. Consent must also be renegotiated as necessary as the research develops. Finally, d. as outlined in the criteria for trustworthiness, a “member-check process” of all interested parties is needed to ensure fairness is present and is being achieved.

The second criteria regarding authenticity is ontological authentication. In this regard Guba and Lincoln (2007) assert the need for consciousness raising among both participants and researchers. In the present research this is done, for example through the making of guidelines for talk. These are decided by students and agreed upon by myself, in the role of teacher (and researcher), achieved through a dialectical process.

The third criteria for authenticity is educative authenticity. This entails a process of mutual appreciation among stakeholders of the views of others and their underlying value systems. Guba and Lincoln further note the need for “gatekeepers who can act to increase the sophistication of their respective constituencies”, in the case of this research, for example, teachers and students.

Catalytic authentication is the final criteria for authenticity. This refers to the need for research to enable action through dissemination (Guba & Lincoln 2007). This strongly parallels the aim of design-based research to bridge the gap between research and practice (McKenney & Reeves 2013). A further point outlined by Guba and Lincoln

106

is the need for tactical authenticity, which works to safeguard the effectiveness of the action that is taken.