Quantitative methods (questionnaire and observation) 1 Reliability.

Reliability refers to the extent to which studies can be replicated. According to Robson (2002), the term reliability refers to the consistency of the results obtained from a research study. That is, within a certain limit of experimental or random error, if the same methods are used with the same sample, then the results should be the same (Cohen, et al, 2007).Robson also indicated that a high reliability of response is obtainable by providing all respondents with exactly the same set of questions. For the questionnaire study, precise phrasing of questions was used based on data from the focus groups and these

180 were written as simply as possible to avoid making unwarranted assumptions about the respondents. Thus, all respondents were provided with the same set of questions, which were distributed to each of the schools around the same time.

As for observations, it would be ideal to have another evaluator present in the classroom to increase inter-rater reliability; however, this was impractical due to limitation of classroom size and to reduce the effect of observer reactivity. A video recording can provide many of the benefits of a second observer. The video can also be used to conduct a follow-up rating on a portion of randomly selected observation to check the consistency of the observer and to increase inter-rater reliability.

8.3.2 Validity.

Robson (2002) suggests that validity refers to the extent to which a research study actually investigates what the researcher claims to investigate and is inherently difficult to establish within a single statistical measure. If a questionnaire is perfectly valid, it must measure in such a way that inferences drawn from the questionnaire are entirely accurate. However, Cohen et al (2005) argue that it is impossible for research to be 100 per cent valid and that a measure of standard error which is inbuilt has to be acknowledged. In quantitative data, validity might be improved through careful sampling, appropriate instrumentation and appropriate statistical treatment of the data. The sampling technique I used in selecting the sample, and the time expended in developing the questionnaire, have been outlined in chapter 10. The following steps were taken to increase validity for the study.

181

8.3.3 External validity.

External validity emphasizes the generalization of the research findings. The external validity of this study is enhanced in the following ways. Firstly, purposive sampling allows me to select the cases that represent the features I was interested in (Silverman, 2010). For example, when arranging for questionnaires to be distributed to the schools, schools that differed in socio-economic catchment area are selected rather than selecting them randomly. This ensured that knowledge obtained is from a range of teachers and students from schools that differ in location and in the economic background of pupils’ families.

This study also enhances external validity through a combination of quantitative and qualitative methods. More importantly, the combination of qualitative and quantitative data has the potential to achieve triangulation, which is one of the important ways to enhance external validity. By utilising mixed methods, the level of convergence among different sets of data generated by diverse methods can be checked. Bryman (1988) argues that combining quantitative and qualitative approaches can enhance the generalization of the researcher’s findings if:

‘they can be shown to provide mutual confirmation’ (p.131).

This thesis examines the effect of historical, cultural and social factors on Korean pedagogy using both qualitative interpretation and description and quantitative statistical techniques.

Further, through a process of ‘overlaying’ data from multiple methods, we may obtain a ‘true’ understanding of the situation under study. Triangulation also involves the

182 use of a wide range of informants. Data from the focus groups with students provides viewpoints and experiences which are verified against data from the teachers, ultimately providing corroborating information. Further, the inclusion of both teacher and student experiences, not only reflects internal consistency but also the credibility of the voices that are heard. By doing so, it is possible to achieve an acceptable level of consistency in findings, and thus increase the external validity of the overall research.

8.3.4 Construct validity.

According to Yin (2003), construct validity refers to establishing correct operational measures for the concepts. Assessing validity in this sense aims to assess whether the researcher has found the most appropriate indicators. I recognize that the variables used in the quantitative models might not truly measure what is intended to be measure due to bias and subjectivity when interpreting data and making personal choices in developing the questionnaire and observation schedule. To reduce the threats to construct validity in the quantitative study, the words and categories from the focus groups are used to

construct the quantitative questionnaire and further to establish the observation schedule.

8.3.5 Face validity.

According to Cohen et al (2007), face validity refers to researchers’ subjective assessments of the presentation and relevance of the measuring instrument as to whether the items in the instrument appear to be relevant and reasonable. Face validity is

increased through an informal pre-check (see Section 10.4.1). Comments related to the wording, clarity, ambiguity and other related issues in response to the items on the

183 questionnaire were made by three colleagues, and were taken into consideration. Further, using phraseology from the focus group data helped in choice of words when developing the questionnaire, thus increasing face validity.

I acknowledge that the questionnaires were not pre-tested again after the re-wording and clarification of the questions by a small sample of teachers and students. This would have been preferable, but was prevented by time constraints.