Step three: Quality assessment

Quality assessment is the process of evaluating the reliability, validity and trustworthiness of the research according to the research criteria. As stated in Chapter Two, quality criteria for conceptual research have not been defined clearly, which makes it difficult to evaluate how ‘good’ or ‘bad’ any particular piece of conceptual research is. In respect to Study One, however, the quality issues of conceptual research were discussed, which ensures its value. Meanwhile conceptual research does not sit in contradiction to quantitative or qualitative research, as indicated in Chapter Three. Since qualitative data and qualitative analysis method is used in Study Two, the evaluation of qualitative research can be adopted.

4.4.3.1 Reliability and validity of the study

Common criteria for measuring research quality emphasise the reliability and validity of the research design. Reliability is used to evaluate the extent to which the data collection methods and analysis process ensures the repeatability and consistency of the findings, whilst validity refers to how well a test measures what it is purported to measure or how truthful the findings are (Saunders et al., 2009). Reliability and validity were first considered systematically within the quantitative research tradition but have subsequently come to be evaluated within qualitative research (Seale, 1999). The establishment of validity and reliability means somewhat different things between quantitative and qualitative research, however (LeCompte and Goetz, 1982). This variety is reflected in modified terms (catalytic validity, transgressive validity, interrogated validity, etc.) that capture the difficulties in creating commonly agreed quality criteria for qualitative research due to its emphasis on creativity, exploration, conceptual flexibility and a certain freedom of spirit (Seale, 1999). Reliability in qualitative research refers to legitimate means of data collection and analysis that enable the production of rich and meaningful descriptions of phenomena, rather than to

obtain exactly the same results, as in quantitative research: different results do not mean that qualitative research is unreliable but that different studies have uncovered slightly different aspects or interpretations of the same phenomenon (Collingridge and Gantt, 2008).

The reliability and validity of Study Two is assured by the distinctive characteristic of both the systematic review and content analysis. The coding units in Study Two were developed based on previous studies involving similar coding objects (sustainable tourism principles and Chinese cultural values). In order to ensure the reliability and validity of Study Two, at least two coders participated in both the coding of the principles of sustainable tourism (the author is named coder 1 and another Chinese doctoral student in tourism is named coder 2) and Confucian and Taoist values (coder 1, coder 2 and a Chinese who is majoring in traditional Chinese literature, named coder 3).

4.4.3.2 Trustworthiness of the study

Seale (2007) argues that the philosophical position adopted does not influence the quality of research. Evaluating the quality of research from different paradigms is a difficult task, however (Kuhn, 1996). Consequently, there is a need for different terms to judge the quality of qualitative research. As Healy and Perry (2000) asserted, the quality of research in different paradigms should be evaluated by their own paradigm’s terms. Reliability and validity are criticised to be terms more appropriate for quantitative research (in the positivist paradigm) and different terms have thus been created for qualitative research (in the post- positivist, interpretivist or constructivist paradigms). The most distinct and commonly agreed such term is ‘trustworthiness’ proposed by Lincoln and Guba (1985). They developed four criteria of trustworthiness that paralleled the quantitative criteria terms. The first criterion is

credibility, a term corresponding with internal validity (truth value), which is concerned with

how truthful the findings are. Secondly, transferability is conceptualised to replace external

validity (applicability) to assess how applicable the research findings are to another context or

group. Dependability is paralleled to reliability as a judgement of the consistence and reproducibility of the results. Conformability is a criterion designed to replace objectivity (neutrality) to evaluate whether the findings are neutral or a production of the researcher’s biases and prejudices.

Trustworthiness requires that research is able to “demonstrate truth value, provide the basis for applying it, and allow for external judgements to be made about the consistency of

its procedures and the neutrality of its findings or decisions” (Erlandson et al., 1993, p.29). Both) and the following researchers (Seale, 1999; Harrison, MacGibbon and Morton, 2001; Morrow, 2005; Rolfe, 2006) suggest how to ensure trustworthiness in qualitative research with regards to the four criteria. Particularly, Decrop (1999, 2004) focuses on trustworthiness in qualitative tourism research and how to achieve it. Credibility can be enhanced through prolonged engagement, persistent observation and referential adequacy (Decrop, 2004). This study does not refer to these issues and cannot adopt these techniques, however. Member checking is considered to be the most effective way to enhance credibility (Lincoln and Guba, 1985; Shenton, 2004; Decrop, 2004). Since Study Two does not involve participants (interviewees for example) no informants can be invited to check the accuracy of the transcripts of dialogues, but member checking can be is used in other ways (e.g. peer checking) such as member checking of coding (at least two coders) and data interpretation.

Transferability can be gained by using purposive or theoretical sampling (Decrop, 2004). As illustrated in the last section, Study Two uses purposive sampling (typical case sampling and critical case sampling) and various samples are selected to ensure the broadest range of information. The researcher is required to provide sufficient contextual information about the study to enable readers to make a transfer (Shenton, 2004). The detailed illustrations of rationale and research settings in Chapter One provide an explication of contexts and the researcher’s orientation. Dependability can be increased by having a research plan, through prolonged engagement and the use of an ‘auditor’ (Decrop, 2004). In this study, apart from a research plan, the supervisors act as auditors, providing a second opinion in respect to data interpretation or the research process. On the other hand, the research design and its implementation are explained in detail in the sections Philosophical Position of the Study (2.3.2), Searching for Possible Research Methods for Study Two (2.4), Method used in this chapter (3.3) and Research Process of Study Two (4.4). These efforts help to ensure that similar results would be obtained if the study were to be repeated. Auditors can also be used to enhance conformability. Supervisors and peers invited within different cultures and genders minimise the effect of the researcher’s biases and prejudices.

Apart from the above individual techniques, the most comprehensive way to achieve trustworthiness is triangulation (Shenton, 2004; Decrop, 1999, 2004). This is also adopted in Study Two. Triangulation is the combination in one study of two or more data sources,

study refers to the variety of data sources (books and journal articles in English and Chinese). Method triangulation involves the use of multiple methods (systematic review, content analysis and conceptual research approaches) to enhance credibility and dependability. The involvement of supervisors and peers in this study represents the adoption of investigator triangulation. Theoretical triangulation pertains to the use of multiple perspectives to interpret the data (environmental perspective, social-cultural perspective and economical perspective in interpreting sustainable tourism; philosophical perspective, social-cultural perspective and psychological perspective to interpret cultural values). This can also be considered as interdisciplinary triangulation.