Study selection phase 3: Quality assessment

Review results can be biased by poor internal and external validities of eligible studies (such as unsuitable interventions and inadequate follow-up time for internal validities, unreliable measurement techniques and inappropriate statistical analysis for external validities [54, 85-87]). Based on suggestions of the Cochrane Collaboration and other experts [83, 88], assessment of quality/validity was conducted by means of a quality assessment checklist. This was in order to minimize possible biases caused by poor validities of the eligible studies

2.5.1 Developing the quality assessment checklist

A number of checklists are available with each of them focusing on specific study designs (such as the checklist for randomized controlled trial and the checklist for observational studies) [54, 89]. In the context of the current review, the quality assessment checklist was constructed according to the following aspects (the checklist can be found in Appendix 2.5).

a. Modifying two existing checklists

The quality assessment checklist was mainly based on Drummond’s and BMJ study quality checklist [90, 91], as both of these two checklists were capable of assessing the quality of economic evaluation studies. Since economic evaluation studies were not the only type of eligible studies for the current review, the checking questions derived from the aforementioned checklists were further divided into two categories/phases according to their characteristics.

In the first phase, the checking questions focused on the quality of the reported costing method of all the eligible studies. The emphasis was placed on what and how the eligible studies measured the cost (for example: Were the costing measurement methods described comprehensively?). In the second phase, the checking questions focused on the quality of the economic evaluation studies (for example: were all the important and relevant costs and outcomes identified?). This was because the questions were important for economic evaluation studies but inadequate for judging the quality of non-economic evaluation studies.

2.5 Study selection 3 (quality assessment)

- 23-

Considering the above, after the two-phase quality assessment checklist was built, all the eligibility studies were assessed by means of the first-phase checklist, while economic evaluation studies were further assessed by means of the second-phase checklist.

b. Two types of evaluation study

Economic evaluation studies can be further divided into two categories: trial-based studies (using a single source of evidence) and model-based studies (using multiple literature sources) [83]. Generally, most reviewers show preference in designing separate quality checklists for different study bases [54]. However, in some occasions, joint checklists can be also considered [92]. In the current review, a joint checklist was used, as there was a large number of common checking questions. The checking questions were divided into two, only in the study design quality domain. In the part designed for trial-based studies, the checking questions focused on the quality of the study structure (such as inclusion / exclusion criteria, or data collecting method). In the part designed for model-based studies, the checking questions focused on the quality of the literature review (such as the method of deriving input parameters for the model, or the representativeness of the model in terms of clinical practice)[84].

c. Internal and external validity

Internal and external validities are both important for quality assessment. In the current review, more emphasis was put on internal validity, as latter was a prerequisite for external validity [54, 84]. Internal validity focused on whether results of a study were

‘correct’ for the circumstances being studied. In contrast, external validity focused on whether results of a study were applicable to other circumstances (outside the study) [93].

Therefore, if the results of a study were invalid (internal validity), then the checking questions of its external validity became redundant.

d. Simple checklists rather than quality scales

There are two approaches for assessing the study quality. One is using the summary scores, and the other is examining the influence of the quality components by simple checklists [58, 94]. Although the use of summary quality scores can provide a useful overall quality assessment, it was decided that the simple checklist approach should be used (instead of the summary score) in the current review. This was for the following two reasons. Firstly, different measurement scales may lead to discordant results. Secondly, summary score was difficult to interpret, as associations between score results and quality

2.5 Study selection 3 (quality assessment)

- 24-

were unclear [58, 94]. In order to avoid the aforementioned issues and to ensure the transparency of the study selection process [89], it was decided that the simple checklist approach should be preferred.

2.5.2 Quality assessment process

After the quality assessment checklist was built, it was applied to the full texts in order the quality (validity) of the eligible studies to be assessed, while it was, also, judged by two un-blinded & independent reviewers. Because of time and workforce restrictions, only one reviewer assessed the qualities of all the eligible studies, while another independent reviewer only double-checked the results of twenty random eligible studies. This was in order to reduce possible bias caused by reviewer’s preference. Both of the reviewers were un-blinded, as blinded review was not necessary here [54, 58, 83]. It is worth noting that the quality assessment results of the two independent reviewers were consistent. Based on this, it is claimed that the selection results were robust.