Validity and reliability

Quality and trustworthiness of research are often associated with the measures of validity and reliability.

Validity has been defined as the appropriateness, correctness, meaningfulness and usefulness of the specific inferences researchers make based on data they collect

(Fraenkel & Wallen, 2008). According to Hoadley (2004: 204) validity of a study refers to the “likelihood that our interpretation of the results accurately reflects the truth of the theory and hypotheses under examination”. Reliability points to the “degree to which a measurement can be replicated” (Hoadley, 2004: 204).

Reliability implies that repeated measurements of the same phenomenon are able to produce consistent results.

Qualitative and quantitative studies tended to rely on different sets of criteria for establishing validity and reliability of their research (Johnson & Christensen, 2004: Cohen et al., 2007).

In quantitative research quality and trustworthiness concerns are primarily related to the following four types of validity (Cook & Campbell, 1979):

i) Internal validity or causal validity: the validity with which it is inferred that the relationship between two variables is causal;

ii) External validity or generalisation: the extent to which results of a study can be generalised to and across populations of persons, settings, outcomes and

treatment variations;

iii) Statistical conclusion validity: the validity of which it can be inferred that two variables are related and the strength of that relationship; and

iv) Construct validity: the extent to which theoretical construct is accurately represented in a particular study;

On the other hand, qualitative researchers are often not concerned with exploring causal relationships between variables, and their notion of validity of research outcomes tends to rely on a different set of criteria (Guba & Lincoln, 1989). Qualitative researchers prefer to use terms such as plausible, credible, trustworthy, and defensible to describe

98

their research outcomes. However issues of validity are important to qualitative

researchers as well. Maxwell (1996) identified three types of validity that are applicable to qualitative research:

i) Descriptive validity: refers to validity of settings and events;

ii) Interpretive validity: refers to the validity of the statements about the meanings or perspectives held by participants; and

iii) Theoretical or explanatory validity: refers to the validity of claims about causal processes and relationships;

Qualitative researchers need to be aware of the threats to the credibility of their research due to the influence of “researcher bias” which results from ‘allowing one’s personal view and perspectives to affect data interpretation and how the research is conducted (Johnson & Christensen, 2004: 249). However, most of qualitative researchers are not concerned about the subjectivity associated with their research Denzin and Lincoln (2000), and often they use “reflexivity” as a means to achieve credibility. Johnson and Christensen (2004: 249) define “reflexivity” as “a self-

awareness and critical self-reflection by the researcher on his or her potential biases and predispositions as these may affect research process and outcomes”. Qualitative

research methods employ a range of techniques including triangulation, peer review, member checking, and participants’ feedback to enhance their trustworthiness of research outcomes.

DBR treats the notion of quality and trustworthiness differently from purely qualitative and quantitative research. Van Den Akker et al., (2006: 85) argued that DBR “typically triangulates multiple sources, and kinds of data to connect intended and unintended outcomes to process of enactment”. They also point out that the reliability of findings and measures can be promoted through triangulation from multiple data sources, repetition of analysis across stages/cycles of enactment, and use (or creation) of standardised measures or instruments.

DBR is not concerned with a broad generalisability of research outcomes, and as such ignores the issues related to external validity. Hoadley (2004: 205) argues that

“universality is rare in educational phenomenon and because methods take tentative steps by first examining individual context, design based researchers generalise their

99

findings only tentatively” because, researchers are involved in the process of intervention as participant observers and because they play an active role in

manipulating the environment they study. Hoadley (2004) points out that it becomes imperative for them to describe and monitor ways in which results may be influenced by their own agenda. For this matter design based researchers like others from different approaches not only document their perspective or starting point, but also document any plausibly relevant interventional strategies used by participants and researchers.

In this study the researcher has taken particular care to present the design and

intervention as a narrative that describes the practices of participants and the researcher, and the context within which this intervention is located. This study has involved a number of participants and care was taken to document the practice in an on-going manner for example, the iterative activities at the three stages of the study, i.e the preliminary analysis, prototyping and implementation of the intervention in the field. Similarly, participants’ feedback, review, (e.g. participants who were involved in the try-out stage provided feedback of their classroom practices) and presentation of the research to seminar(s) and conference(s) helped the researcher to document the practice in detail and at the same time the researcher was able to clarify the personal perspective and the possible effect it could have on outcomes in a reflexive way. The information kept in the researcher’s log book throughout the field work enhanced reflection on both the researcher’s and teachers’ practices.

Another technique for ensuring rigour in DBR is its reliance on multiple methods and multiple sources of data (Cobb et al., 2003). This effect rests on the premise that the weakness in each single data source, method, evaluator, and theory or data type is compensated by counterbalancing the strength of another (Miles & Huberman, 1994; Patton, 1990). This study employed triangulation design in order to corroborate findings from qualitative and quantitative methods employed in the answering of the research questions. The study adopted the convergence model of triangulation Creswell & Plano Clark (2007) whereby, the researcher collected and analysed qualitative and quantitative data from different sources separately for each research question. Thereafter, the two datasets were merged or converged (by comparing and

100

contrasting the different results) during the interpretation of the findings so that a complete analysis could be developed from both datasets.

The study also used a quasi- experimental design during field implementation of PD programme and evaluation (Table 5.2) to gain a better understanding of the impact of the intervention on student learning outcomes. The quasi-experimental design followed the criteria of internal validity and causal validity but it was not the concern of this research to address generalisability because the goal was to gain an understanding of the intervention as it unfolded in a particular setting, and to develop tentative theories applicable to the particular context. This helped to maximise ecological validity, i.e. the method, materials, and setting of the study must approximate the real life situation under investigation (Van Den Akker et al., 2006). Table 5.3 illustrates experimental threats to internal validity and how this study tried to eliminate/reduce them.

Table 5.3 Experimental threats to internal validity

Threats Authors’ explanations Elimination/reduction of the

threats in the present study

History Specific events which occur

between the first and second measurement that could cause the observed outcomes (Shadish, Cook & Campbell, 2002).

The experimental and control school teachers had the same headmasters in the same school but teaching different classes. Students were taught by the same teachers in their respective classes.

Mortality Loss of respondents to treatment or to measurement can produce artificial effects (Shadish, Cook & Campbell, 2002).

In the present research sample neither teachers nor students dropped out.

Instrumentation The changes in the instrument, observers, or scorers which may produce changes in outcomes (Robson, 2002)

The participants (students) and the instruments, i.e. Appendices C3 was the same during pre-testing and post- testing.

Maturation Naturally occurring changes over time that could be confused with a treatment effect (Shadish, Cook & Campbell, 2002).

Students in the control group had a similar experience as experimental students (e.g. use the same syllabus and class levels and teachers had the same education level, i.e. Diploma)

Selection There may be preliminary

differences between the control and experimental groups before involvement in the study (Robson, 2002)

Students were matched based on their respective class levels such as: Form I, II, III and IV. There might be other events which the research would have had no control.

101

Other techniques employed to maximise the validity and reliability are:

i) Using the assistant researcher during the evaluation stage to assist in classroom observations, i.e the researcher and assistant researcher independently observed the teachers’ classroom practices guided by the classroom observation checklist (Appendix C4). This helped to improve internal reliability of findings from classroom observations;

ii) Piloting of the data collection instruments during the prototyping stage of the study in order to ensure the validity of the instruments in collecting the intended information during the field implementation and evaluation stage (Chapter 6, section 6.5).