RELIABILITY AND VALIDITY

As mentioned earlier, a scientific methodology presents the basic structure for scientific research design, which comprises explicit rules and criteria to the entire research process. As a research design seems to present certain logic, the research quality also can be judged following certain logical assessment. In assessing the quality of scientific research design, reliability and validity have been the most utilized quality criteria. To reduce the possibility of getting the result wrong in research, researchers should pay particular attention to the reliability and validity of research design (Saunders et al., 2003).

Reliability concerns the findings of the research, which refers to whether a study is replicable by anyone else and the results obtained will be same. Validity relates to the entire research process, in which the research findings accurately represent what is happening in the situation, there are several different forms of validity assessment: internal validity, external validity, and construct validity (Collis & Hussey, 2003).

Under the phenomenological paradigm, there is not too much clear criterion of reliability, and it may be interpreted in a different way. Consequently, the reliability under the phenomenological paradigm is low (Collis & Hussey 2003). However, a phenomenologist may establish other criteria or procedures to ensure the reliability of their findings. Meanwhile, the phenomenological paradigm aims at capturing the essence of a phenomena and extracting rich data from its explanation and analysis (Collis & Hussey 2003), and such complete understanding of the fact and meaning of a phenomenon results in high validity.

For case study, Yin (1991) identifies several tactics for dealing with these research tests, which has described by Yin as an important innovation. Table 3.2 presents the four tests and the tactics for dealing with them in case study. The techniques illustrated in Table 3. 2 can be adopted to be the guidance of improving the quality of

63 the cases study design. In this case study, focus was mainly on construct validity, external validity, and reliability. The exploratory nature of the study prevents the regarding to the internal validity that concerns the establishment of the cause-and-effect relationships.

tests Case-study tactic Phase of research in

which tactic occurs

Construct validity Use multiple sources of

evidence;

Establish chain of

evidence;

Have key informants

review draft case study report

Data collection Data collection

Composition

Internal validity∗∗∗∗ Do pattern matching

Do explanation-building Do time –series analysis

Data collection Data collection Data collection

External validity Use replication logic in

multiple –case study

Research design

Reliability Use case study protocol

Develop case study data base Data collection Data collection ∗ ∗ ∗

∗ Internal validity is used for explanatory or causal studies only, and not for descriptive or exploratory studies.

Table 3.2 Case study Tactics for Four Design tests Source: Yin (1991, p.40).

3.6.1 Construct Validity

According to Riege (2003), construct validity relates to the establishing appropriate operational measures for theoretical concepts being studied. Case study research is recognized to be more subjective than other qualitative research methodologies because researchers in case study research usually have close and direct personal

64 contact with the targeted organizations or people. Therefore, to enhance construct validity, researchers need to avoid subjective judgements during the process of research design and data collection.

In the present study, construct validity is enhanced by using multiple sources of evidence in the data collection phase; triangulation is achieved through the combined use of questionnaires, and interview records and archive information. Also, a chain of evidence was established during the data collection, which includes the digital records and transcriptions of all the interviews, questionnaire feedback, and all downloaded archive documents.

3.6.2 External Validity

External validity “is concerned with the extrapolation of particular research findings beyond the immediate form of inquiry to the general” (Riege, 2003, p.81). In other words, external validity is used to evaluate if the findings of a study can be generalized. In contrast to quantitative research aiming at statistical generalization, case studies tend to rely on analytical generalization, this refers to the particular findings of a case study can be generalized or expanded to theories rather than generalized to a population (Yin, 2003).

In the current case study, multiple case study design was chosen as the technique for increasing external validity. All cases were selected following literal and theoretical replication logic. The studied 3PL companies were selected from the list of Chinese top 100 logistic companies which were within the same industry and involved in similar business activities. The cases were chosen also on the basis of the criteria made by the researcher. The above scope and boundaries defined in research design stage ensure the achievement of reasonable analytical generalizations of the study.

3.6.3 Reliability

65 procedures of a study can be repeated by other investigators and the same results can be achieved. Generally speaking, in a case study, even though the researchers follow each step precisely, the results may be still different. In other words, the data related to real-life events collected by different researchers may result in different outcomes (Riege, 2003). However, Yin (1991) presents two useful tactics that can be adopted for researchers to increase the reliability of their studies: using case study protocol or developing case study data base.

To enhance the reliability of present case study, a case study database has been developed before data collection. As discussed earlier, the database was classified into different categories, and related information and data such as questionnaire feedbacks, interview records, and other retrieved archive information, was organized and documented in the database.

In addition to direct collected data, some information about the steps of conducting the study was also included in the database, such as research progress reports, feedback from research supervisor, and other information related to the research process. According to Yin (2003), the general way of approaching the reliability problem is to make as many steps as possible and to conduct research “as if someone were always looking over your shoulder” (p. 38). The introduction of case study database allows other researchers to review not only the research report, but also the raw data and explicit research procedures.