The issues of validity and reliability refer to the audit of the research. Validity is the degree to which research outcomes accurately reflect the settings of the study (Babbie, 1998). Validity is used to ensure that the methods represent the factors that are identified through the literature, and later emphasised in the research objectives (De Vaus, 2001). Greenfield (2002) considered any research to be valid when it measures what it is supposed to measure. Reliability is demonstrating the process of a study – such as data collection procedures, keeping records and repetitions with the same results (Yin, 2014). In other words, Gay et al. (2009) described the degree to which a test consistently measures the attribute it is measuring (p. 158). Four tests are commonly taken to establish the quality of empirical social science research. Yin (2014) stated that, among these tests, only internal validity is not relevant to case study research as it seeks to establish a causal relationship. The others are described below:
94
Construct validity: identifying correct operational measures for the concept being studied.
External validity: defining the domain to which a study’s findings can be generalised. Reliability: demonstrating that the operations of a study – such as data collection
procedures – can be repeated, with the same results (p. 46).
The testing of the construct and external validity, and reliability of this research are discussed next. Yin (2014) and Cresswell (2012) explained that construct validity could be increased through multiple sources of evidence, establishing a chain of evidence and then having the draft case study report reviewed by key informants. In the context of this study, the multiple sources of evidence include the pilot study, semi-structured interviews and the course documents which are used to form the chain of evidence. This process forms the construct validity.
Similar to Yin’s explanation, Trochim (2000) described external validity as the degree to which the outcome of the research study holds for other persons in other places and at other times. The issue, previously discussed in Chapter 3.4.2, is to produce generalisable results in a case study, either by conceptualising or by developing hypotheses. As this study conceptualises generic skills assessment within AL environments in Malaysian HE with the aim of improving the standardisation and reliability of generic skills assessments in HE institutions, this provides the basis for external validity.
Finally, the goal of reliability is to minimise the errors and biases in the research. Reliability in social science can be divided into two types: internal reliability and external reliability. This study referred to Drew et al. (1996) proposals to maintain reliability in qualitative research. The proposal includes the sets of steps as follows:
For internal reliability:
1. Use low inference descriptors.
2. Use multiple researchers when possible.
3. Establish a careful audit trail (record procedures on collecting and handling the data that can be followed by another researcher until the conclusion has been made). 4. Use mechanical recording devices where possible (and with permission).
5. Use participants as researchers or informants to check the accuracy or congruence of perceptions.
95 For external reliability:
1. Clearly specify the researcher’s status or position so that readers know exactly what point of view drives the data collection.
2. Clearly state who the informants are (or what role they play in the natural context) and how and why they are selected (while maintaining confidentiality).
3. Carefully delineate the context or setting of boundaries and characteristics so that the reader can make judgements about similar circumstances and settings.
4. Define the analytic constructs that guide the study (describing specific conceptual frameworks used in the design and deductive analysis).
5. Specify the data collection and analysis procedures meticulously.
Based on these steps, this study plans to maintain all points in the internal and external validity, as suggested. Consequently, the reliability of this research is achieved.
3.5.1 Triangulation
A massive quantity of data is produced as multiple methods of collection are adopted. This is to ensure that the issue of generic skills assessment within the AL environment is not explored through one lens, but rather a variety of lenses which allows for multiple facets of the phenomenon to be revealed and understood. Therefore, it is necessary to have a systematic organisation of the data to avoid data overload or perhaps data overload or confusion, which may cause the researcher to lose sight of the original research objectives. To avoid both circumstances, Table 3-3 shows how multiple methods and instruments may apply to this study. This process makes it easier to triangulate the data. Brewerton et al. (2001) described triangulation as the use of different research sources to collect data from alternative sources within the same study to assess the validity of the findings (p. 200). Triangulation allows researchers to create an appropriate framework for the phenomenon being studied especially when it involves studying the structure, development and changes in the setting. In the context of this study, the literature review, semi-structured interviews and course documents data helps to overcome issues of reliability and validity (Brewerton et al., 2001).
Table 3-3Data collection methods and their purposes (Source: author)
No. Method(s) Objective
1. Literature review and semi-
structured interviews
To critique the existing methods of assessing generic skills in the context of Engineering Education within an AL environment based on the literature.
96
2. Course documents and semi-
structured interviews
To collect data from two case study institutions in a range of different forms that would then allow integration with the literature.
To develop a framework to guide the assessment of generic skills in an AL environment.