Reliability and Validity Issues in this Study 35

In the following paragraphs I discuss issues of reliability and validity. First of all, reliability “refers to the consistency of a measure of a concept” (Bryman, 2012: 169). It has three prominent factors: stability, internal reliability and inter-observer

consistency. Specifically, stability asks whether a measure is so stable over time that we can be confident of receiving highly correlated findings from both a test and a retest on the same respondents (Bryman, 2012). In my study, the measures are Chinese values and impact of civic education activities. The meanings of Chinese values and the opinions to civic education activities may change randomly according to different student’s understandings and experience. So, the measures of this study are not very stable. Highly correlated results from different tests on the same group of students are uncertain; and the stability of this study may be in doubt. Internal

other words, with a multi-indicator measure, an internal reliability means that respondents give affiliated and related scores on all indicators. As for this study, the indicators of each measure seem related and coherent with each other. For instance, when students evaluating the statements concerning Teaching/Learning Methods in civic education practices, their score on each statement tend to be related to their scores on other statements – students preferring freedom of expression will

correspondingly like the teachers who encourage them to freely express views and dislike the ones who force them to accept specific opinions. In this sense, the internal reliability of this study is upheld. Thirdly, inter-observer consistency is damaged “when a great deal of subjective judgment is involved in such activities as the recording of observations or the translation of data into categories and where more than one 'observer' is involved in such activities” (Bryman, 2012: 169). Since I am the single “observer” or data collector, this actually cannot be used in my study. The reliability can be accordingly hampered. So, as a means to remedy this shortcoming, I have tried to describe as thoroughly as possible my data collection and analysis process. In all, this study has highly concerned and appreciated the three prominent factors of reliability. But, as this study is small-scaled with limited resources, this desirable reliability test is not possible in practice.

Cook and Campbell's validity system has been generally accepted with the

quantitative tradition (Kleven, 2007). There are three types of validity in this system: construct validity, external validity and internal validity. First of all, construct validity is about “to what extent are the constructs of theoretical interest successfully

operationalized in the research” (Judd, Smith & Kidder, 1991, in Kleven, 2007: 224). In other words, the issues of construct validity are about to “draw inferences from observed indicators to abstract constructs” (Kleven, 2007: 223), or about how well the concept is represented by the indicators (Kleven, 2007). Most constructs of interests in educational research are not directly visible or measurable (Kleven, 2007), as is also the case with the theoretical construct in my study -- they are abstract and invisible. So, in order to avoid systematic measurement errors, namely the construct

underrepresentation and construct irrelevance, this study has paid much attention to conceptualizing theoretical constructs of civic education. Scholarly literature and the researched university's own civic education practices have both been mulled over in the process of conceptualization. The study has been trying to go back and forth between its abstract theories and its measurable indicators. In this way, civic

education is expected to obtain a specific and explicit definition, on the basis of which this study's theoretical construct can become operationalized and measurable. In all, making inferences from students' observable judgment to abstract impacts of civic education, the construct validity is considered very important by this quantitative study. As the process of operationalization and measurement is consistently

theory-laden, this study should have put a damper on problems of construct validity.10 Secondly, quantitative methodology views internal validity as “validity of inferences from an observed covariation to a causal interpretation” (Kleven, 2007: 227). As earlier mentioned, due to a reliance on questionnaire instrument, this design has in a way hindered “the connection between research and everyday life” (Bryman, 2012: 179). By simply inviting students to fill in a questionnaire, it is still uncertain if they have the expected knowledge to answer a question or if they consider the researched topic similarly important in their everyday lives. Meanwhile, being restrained by fixed questions, it might be difficult for students to express their own opinions about the specific topic. Its ecological validity11 may be jeopardized; and this study will have a lack of rational assessment of alternative possibilities and interpretations. Accordingly, covariance between variables gives no basis for causality in this quantitative research; and it results in causal description instead of causal explanation as from qualitative researches. So, its internal validity may be weak. Probably a follow-up qualitative research will pave way for more accurate and deeper inferences (Kleven, 2007). Thirdly, external validity means the “validity of inferences from the context of the study to a wider context or to other contexts” (Kleven, 2007: 229). This is the issue of

10. See Chapter Three about the theoretical perspectives and the conceptualization of civic education in this study as detailed references to issues of construct validity.

11. Ecological validity asks whether the results of a study can be applied to different settings. More accurately, it is concerned more with settings than population; and it focuses on the validity of the results in other places (Nolan & Behi, 1995).

generalization or transferability, having direct connection with how a representative sample is created. But as for my research design with a convenient sampling, it does not have external validity and I cannot generalize the findings to a larger population. However, this study can still be seen as a contribution in adding to the pool of knowledge within this field.