Reliability, reflexivity, and triangulation

In this study, there were a number of steps taken to ensure reliability through triangulation of data. However, as with any study, some steps were not possible due to circumstances relevant to particular places and institutional restrictions. Clarke (1990) explains that evaluators work under many

restrictions, including research budget, time limit, minimising disruption of programme activities, as well as critical and political factors to take into consideration. To compensate any possible limitations that may arise from doing research under such restrictions, steps were taken towards triangulation of the data in an attempt to increase the reliability and validity of the study.

Clarke (1999) discusses the four types of triangulation by Denzin (1970) which are data, investigator, theory and methodological triangulation. Data

triangulation involves the creation of multiple data sets collected by employing one or more methods at different times. In this study, two methods were used, interviews and document analysis, and multiple data sets, as each assignment constituted a data set of interview data and documentary data. Data sets were collected in different contexts (different universities, different countries,

different departments and different levels of study) as explained earlier. Another type of triangulation, theory triangulation, “entails making use of a number of alternative or competing theories in examining the data” (p.86). An argument could be presented here that FUAD, the driving conceptual

framework for this evaluation, is a synthesis of two theoretical concepts and a practical strategy, namely, flexible pedagogies, ubiquitous learning and device neutral assignments. Therefore, the theory in this theory-driven approach to

evaluation is based on two alternative theories and one practical strategy.

The third type of triangulation attempted in this study is methodological triangulation, particularly a ‘within-method’ approach. This approach entails “applying the same method on different occasions by using multiple

techniques within a given method” (pp.86-87). The interviews were conducted in a number of ways; face-to-face, telephone, video conference, in the office during work hours, twice at home and twice in coffee-shops. In some cases, email exchanges took place after the interview when data analysis was shared for confirmation and for the purpose of suggesting improvements. However, the investigator type of triangulation was possible only partially, as this is a doctoral study and is the sole effort of the researcher. Other researchers could not be invited to evaluate the same assignments. Having said that, two

researchers were invited to evaluate the FUAD framework itself. A professor in educational technology and an assistant professor in education reviewed the FUAD conceptual framework and gave positive feedback on it, along with some suggestions to clarify a few points. This adds to the reliability of the evaluation tool itself.

Moreover, special attention was paid to validity, reliability and objectivity, as Gray (2004) points that these are areas of concern with evaluation research. Gray explains that the researcher needs to approach the evaluation with an open mind and without any preconceived ideas due to the possibility of

interference by the researcher’s own values. Therefore, it was important that I should bracket my own values and evaluate the process based on the

reflection and checking, and through sharing analysis with interviewees as mentioned above.

Other considerations relate to the lecturers’ (interviewees’) different conceptions of learning and/or designs as well as their epistemological

positions compared to that of the researcher. Such differences in position were noted in the interview and throughout the analysis. It was framed as the

underpinning philosophy for certain design elements that limited or enhanced the assignment design.

Gray (2004) adds the problems of how honest the participants may be and how self-reflective as this may influence the data. He continues to point out the question of how sufficient the number of indicators is. However, in this

research, the interviewees were approached informally. They were not

contacted on behalf of any institution or administration. It was made clear that what they would share would be for the purpose of this doctoral research with no consequences on the lecturer’s own career. In addition, the interviewees were made aware that the information they provided would inform and add to the formation of the FUAD framework and could possibly lead to future joint research, if they were interested. However, interviewees did not have prior knowledge of the FUAD framework, so their information could not have been tailored to fit the framework. They were only informed that this research aimed at investigating how lecturers design assignments and learning experiences.

4.3. Summary

The chapter details the pragmatic design adopted for this theory-driven evaluation research. A non-probability, purposive sample was used to identify lecturers who had designed TEL experiences. Data collection was in the form of sixteen semi-structured interviews to discuss assignment designs from a variety of universities in different countries. Assignment documents were also analysed. Data from both sources was coded thematically. Finally, the chapter concludes with a discussion of research limitations.