Validity and Reliability

Validity and reliability are the two most important criteria for assuring the quality of a research study. “Social scientists rely on assessments of reliability and validity to evaluate the quality of their measurement methods or tools” (Wagner et al., 2012, p. 80). Thus, “reliability and validity are central issues in all measurement” (Neuman, 2012, p. 121).

An important consideration in relation to the concept validity is what it measures. “Validity asks: Do the instruments measure the phenomenon that they are supposed to?” (Hesse-Biber, 2010, p. 85). Smith defines validity “as the degree to which the researcher has measured what he has set out to measure” (1991, p. 106). According to Babbie, “validity refers to the extent to which an empirical measure adequately reflects the real meaning of the concept under consideration” (1990, p. 133). In terms of measurement procedures, “validity is the ability of an instrument to measure what it is designed to measure” (Kumar, 1999, p. 137).

There are two types of validity: internal validity and external validity (Creswell, 2009; De Vaus, 2001; Seliger, 1989). “Internal validity is the extent to which the structure of a research design enables us to draw unambiguous conclusions from our results”, whereas “external validity refers to the extent to which results from a study can be generalized beyond the particular study”. This distinction is important, as De Vaus notes, “a study may have good internal validity but its value is limited if the findings only apply to the people in that particular investigation” (2001, p. 28).

Reliability is also crucial to the quality of research. “Reliability estimates the consistency of your measurement” (Wagner et al., 2012, p. 80). Neuman (2012), in a similar vein, explains, “reliability means dependability or consistency. It suggests that repeated, stable outcomes are the same under identical or similar conditions” (p. 121). “Reliability asks: If I use the same measure today and repeat it again on the same population shortly thereafter, will I obtain the same result?” (Hesse-Biber, 2010, p. 85). According to Kumar, “reliability is the degree of accuracy or precision in the

measurements made by a research instrument. The lower the degree of ‘error’ in an instrument, the higher is the reliability” (1999, p. 140).

Although they are separate concepts, there is a close relationship between reliability and validity. The instruments we use for our research must be both reliable and valid. It is not useful to measure something consistently if we are not measuring the thing we intended to (Wagner et al., 2012). In relation to this point, Bryman (2008) also states that “although reliability and validity are analytically distinguishable, they are related because validity presumes reliability. This means that, if your measure is not reliable, it cannot be valid” (p. 153).

It is pointed out that “perfect reliability and validity are virtually impossible to achieve. Rather, they are ideals for which all researchers strive”, and “all social researchers want reliable and valid measures because it helps to establish the truthfulness, credibility, and believability of findings” (Neuman, 2012, p. 121). Therefore, in order to assure the quality of this research study, great efforts were made by the researcher during the research process to increase the validity and reliability.

Before data collection, the questionnaires were developed over a prolonged period of time. As mentioned in section 4.6.1.1, a pre-pilot phase and pilot phase were conducted to gather feedback and changes, revise and modify the questionnaire to enhance the reliability and validity of data collected. In the interviews, questionnaires, and class observations, anonymity and confidentiality were assured to make all participants feel confident and comfortable to answer questions openly. In order to increase the validity of the survey questionnaire, the characteristics of the sample population were the same as those of the target population (second year, non-major English students) at HUTECH. In the interviews with teachers and managers, every effort was made to create a friendly and comfortable atmosphere so as to encourage the interviewees to express their thoughts and opinions freely. During class observations, the researcher assured and encouraged both teachers and students to behave naturally, as in their normal lessons.

During class observations, note-taking was employed, and the researcher also sat at the back of the class, so as to disturb the class as little as possible, and not to distract the teachers or students from their teaching and learning activities.

In the data analysis procedure, steps were taken to ensure the validity and reliability of data analyzed. The recorded interviews were transcribed by the researcher carefully to ensure accuracy. When coding data, attention was given to the smallest details, and care taken to ensure consistency. Intercoder agreement was applied, which allowed the researcher to ask another person to cross-check the codes (Creswell, 2009). The coders agreed on most of the codes used. In addition, peer debriefing was also used “to enhance the accuracy of the account. This process involves locating a person (a peer debriefer) who reviews and asks questions about the qualitative study so that the account will resonate with people other than the researcher” (Creswell, 2009, p. 192).

This awareness of the importance of validity and reliability informed the research design and ensured that the researcher was aware of the need to recognize personal biases and impressions when analyzing and interpreting data. The researcher reported exactly and objectively the results of data analysis in the findings chapters. All the recommendations were made based on the results of the study, and related to the research questions of the thesis. The following section introduces the concept of insider research and the researcher has used a change of voice, switching from third person to first person, to highlight the issues of a researcher being personally connected to the research site.

A very important factor in this research is my insider status as a researcher

conducting this study. Researchers must be aware of their own position within the research and how it can influence the objectivity and subjectivity of the data. In my own research, my dual position as both a known teacher within HUTECH and a researcher presents potential conflict which can best be acknowledged by stating my role as an insider

researcher. Being an insider research carries both advantages and disadvantages. Insiders have a wealth of knowledge which outsiders do not have access to (Pham, 1999). On example of how an insider researcher can be advantaged is in the response of interviewees who may feel more comfortable and talk more openly with the researcher if they know the researcher well (Tierney, 1994). This personal knowledge of the participants and the researcher can also work negatively in that information could be withheld or modified because the researcher is known. It is important to understand that there is no neutral position and the researcher cannot control all the possibilities and needs to work with the tensions that arise within each context. Insider research, may have a potential to increase validity due to the researcher’s profound knowledge of the research’s background and context, which provides the richness, fidelity, honesty and authenticity of the acquired information (Peacock, 1997). Therefore, it is important for researchers to be aware of their status in a research study and the various impacts this can have on the validity of the study (Siao-Cing, 2012). In this study, my position as an insider is based on my four years working at HUTECH, and my knowledge of HUTECH’s education system, policy and operation. This knowledge was an advantage for the researcher when conducting the research (selecting appropriate participants, choosing appropriate research techniques, etc.). My insider status gave me a number of advantages, especially in the data collection phase. As a known teacher at HUTECH, I had no difficulty when asking for the approval from HUTECH’s leader to conduct the research at HUTECH, which an outsider might have had. I also have friendly social contacts with many teaching staff and administration staff at HUTECH. Therefore, I did not have difficulty when inviting them to participate in the research and asking for teachers’ permissions to access their classrooms to conduct observations. Most staff members reacted positively to my requests. In addition, some of my colleagues are also enthusiastic and supportive, by spending time reading my pilot

questionnaire and giving me feedback, which contributed to the success of my main questionnaire.

My awareness of my own insider status prompted more careful thought about possible bias, subjectivity and the inevitable blind spots that my knowledge about HUTECH may have caused me to overlook. As a teacher at HUTECH, I might have personal prejudices supporting the teachers’ interpretations. My data report might reflect my own impressions and feelings, which might show the support for the teachers. This awareness reminded me to be more careful during all phases of the research, especially in data analysis and interpretations, to avoid prejudice. In writing the report, I clarified the research process in details as much as possible. As mentioned earlier, I also had a person play role as a peer debriefer who read my data report and ask questions to clarify the findings and interpretations.