Data appraisal

Once a comprehensive literature search were conducted and articles and/or studies have been retrieved and assessed for inclusion and exclusion eligibility (as indicated in the systematic review protocol), the next stage of the systematic review process was to critically appraise the evidence found.

Critical appraisal, also called a critique of literature, is an organized, systematic approach to evaluate a research study or a group of research studies or other pieces of evidence found using a set of established critical appraisal criteria to objectively determine the quality of evidence provided by literature to determine its applicability to research, education or practice (LoBiondo-Wood and Haber, 2010:57). Critical appraisal should aim to assess the validity of the selected studies, determine the reasons why, other than chance, there is a difference between study results and

139

provide sufficient information for the reader to determine if the systematic review is applicable to their clinical practice. Furthermore, the purpose of critical appraisal is to include those studies that are of a high quality and to exclude the studies that are of a poor quality in the systematic review. The appraisal achieves this by evaluating critical components of the study design (Evans, 2001:54). Critical appraisal is very important as it assesses the relevance and quality of the paper to the review question (Aveyard, 2010:92).

Once the papers found were categorised according to the hierarchy of evidence levels, the papers were re-read to become familiar with them and to make absolutely sure regarding the type of study design used in each study. The rationale for doing this was to ensure that the correct critical appraisal was chosen for the type of paper to be appraised.

In order to facilitate the process of critical appraisal, various critical appraisal tools are available, for instance, the appraisal tools designed by the Public Health Research Unit at the University of Oxford (available at

www.phru.nhs.uk/casp/casp.htm), which is known as the Critical Appraisal Skills Programme (CASP). Other critical appraisal tools include the Pedro Scale, the JADAD scoring system, RevMan (used by the Cochrane Systematic Review Group), RAPid (available via www.joannabriggs.edu.au/services/rapid), Catmaker accessed via http://cebmh.warner.ox.ac.uk/cebmh/education-critical-apraisal.htm and the SUMARI software packages (Pearson et al, 2007:98). Alternatively, critical appraisal tools can be self-developed according to the characteristics of the studies included in the review.

For the purpose of this research study, the SUMARI (System for Unified Management, Assessment and Review of Information) Version 4.0 software package, developed by the Joanna Briggs Institute, was used (www.joannabriggs.edu.au). Because the research study is based on the JBI theoretical framework, it was decided to use SUMARI as it assists reviewers in all components of the systematic review process, from systematic review protocol

140

development to critical appraisal, data extraction and synthesis of both quantitative and qualitative evidence.

SUMARI or the JBI-SUMARI is a software suite designed to assist health and other researchers, scientists and practitioners to conduct systematic reviews. SUMARI has a management component (the Comprehensive Review Management System: JBI- CReMS) that manages the systematic review process, including protocol development, facilitation of critical appraisal, data extraction, data synthesis and report development processes. CReMS can be used as a stand-alone program to manage and archive a review, or alongside the four analytical modules developed to articulate with it. Whenever a protocol is entered and confirmed in CReMS, the review details and the details of studies and/or papers selected for retrieval are automatically uploaded into the analytical modules selected by the reviewer. Similarly, as studies and papers are appraised, extracted from and synthesised, the results are uploaded to CReMS to generate an ongoing, editable report builder.

CReMS links to the four analytical modules of JBI-SUMARI namely:

 Joanna Briggs Institute Qualitative Assessment and Review Instrument (JBI- QARI). The module (accessible through CReMS) is designed to facilitate critical appraisal, data extraction and meta-aggregation of the findings of qualitative studies;

 Joanna Briggs Institute Meta Analysis of Statistics Assessment and Review Instrument (JBI-MAStARI). The module (accessible through CReMS) is designed to conduct the meta-analysis of the results of comparable cohort, time series, descriptive studies, case reports or review papers using a number of statistical approaches.

 Joanna Briggs Institute Narrative, Opinion and Text Assessment and Review Instrument (JBI-NOTARI). The module (accessible through CReMS) is designed to facilitate critical appraisal, data extraction and synthesis of expert opinion texts and of reports.

 Joanna Briggs Institute Analysis of Cost, Technology and Utilisation Assessment and Review Instrument (JBI-ACTUARI). The module (accessible through CReMS) is designed to facilitate critical appraisal, data extraction and synthesis of

141

economic data (SUMARI suite accessed via www.joannabriggs.edu.au). A schematic outline of SUMARI version 4.0 is given in Figure 3.3.

Figure 3.3 Outline of SUMARI suite (www.joannabriggs.edu.au).

Once all the papers found were categorised, the systematic reviews were registered on the JBI SUMARI software suite. According to the evidence found, the critical appraisal tools in the MAStARI and the NOTARI modules were used to appraise the evidence. The JBI MAStARi was designed to enable reviewers to systematically review and combine the results of quantitative healthcare research. The critical appraisal tools in the JBI NOTARI module in the software package were used to appraise the Level VII evidence found, including opinion or expert, conference, committee and literature review papers.

However, despite the fact that the SUMARI software suite is quite comprehensive, it did not allow for the critical appraisal of clinical guidelines found. Therefore, the Appraisal of Guidelines Research and Evaluation (AGREE) appraisal tool, that was developed as part of an international collaboration of researchers and policy makers, was used. The instrument provides a standard framework for the development and implementation of clinical practice guidelines. A checklist of 23 items across six different quality domains provides a useful tool for the generation and evaluation of

142

guidelines. The AGREE instrument is generic and can be applied to all types of clinical practice guidelines (Schmidt and Brown, 2009:325). In this research study, the AGREE instrument was used to critically appraise the clinical practice guidelines or best practice guidelines found (see Annexure M).

Due to the diverse levels of evidence found in the study, different critical appraisal tools were used. Each tool is designed to assess the validity of study design and methods used and reported. Critical appraisal of Level I evidence addressed aspects related to the rigour of the searching process and reporting of data. Critical appraisal of randomised controlled trials (Level II) and quasi-experimental studies (Level III) assess if patients were randomised to study groups, the method of allocation, whether the allocation to treatment groups was concealed from the allocator, if the groups included in the study have been comparable at entry, whether the outcomes have been measured in the same manner for all participants, if the statistical analysis used was appropriate and if the rates of patient follow up have been reported as adequate. Furthermore, it is important to assess the degree to which possible bias has been limited. Source of bias in RCT designs includes selection, performance, attrition and detection biases.

Selection bias usually arises out of an inadequate randomisation of participants. In

order to minimise the chance of selection bias, strategies, for instance, blinding until treatment allocation can be used. Blinding in this context means ensuring that neither the researchers nor the research participants are aware of who is allocated to the control or the experimental groups. Randomization should ensure that every participant has an equal chance of being in any of the study groups. In performing critical appraisal, the appraiser, therefore, needs to establish how well this was achieved (Jones and Evans, 2000:70; Pearson et al, 2007:74).

Performance bias occurs if additional therapeutic interventions are provided

preferentially to one of the comparison groups (Egger et al, 2001:90). Performance bias is avoided by concealment of the treatment group and might be achieved through blinding both those participants and carers that might affect the study result. This can be referred to as double blinding (Pearson et al, 2007:76).

143

Detection bias refers to the tendency to look more carefully for an outcome in one

of the comparison groups. Detection bias arises if the knowledge of the patient assignment influences the process of outcome assessment. Detection bias is thus avoided by the blinding of those assessing outcomes, including patients, clinician investigators, radiologists and review committees (DiCenso et al, 2005:548).

Protocol deviations and loss of follow-up may lead to the exclusion of patients after they have been allocated to treatment groups, which may introduce attrition bias. Possible protocol deviations include the violation of eligibility criteria and non- adherence to prescribed treatments. Loss of follow-up refers to patients becoming unavailable for examinations at some stage during the study period, either because of the patient‟s refusal to participate further, or because the clinician decided to stop the intervention or because the patient cannot be contacted for further follow-up (Egger et al, 2001:90). The critical appraisal tool makes provision for the assessment of the different typesof bias that can be found. A copy of the critical appraisal tool used for experimental studies is illustrated in Annexure N.

In reviewing Level III evidence, which includes cohort studies, case control, surveys, prospective, observational and descriptive studies, the comparison made between groups, the outcomes measured, the time period for conducting the study, the attrition rate and whether an appropriate statistical analysis was used in the studies needed to be assessed. These were done using the critical appraisal tools as reflected in Annexures O and P.

No evidence was found at Levels V and VI; therefore, the critical appraisal will not be discussed. In assessing Level VII evidence, the source of the opinion had to be clearly identified as well as the background to the source of the opinion or author of the literature papers. The critical appraisal tool assess aspects for instance the material on which the opinion or paper is based, whether the opinion or paper is supported by peers. See Annexure Q for a copy of the critical appraisal tool used.The SUMARI software package made provision for the different types of critical appraisal tools to be used as illustrated in Table 3.4.

144

Table 3.4 Evidence hierarchy and software packages used for critical appraisal

Level Type of evidence Software modules

Level I Systematic review or meta-analysis of randomised control trials

Clinical practice guidelines based on systematic reviews

Joanna Briggs Institute Meta Analysis of Statistics Assessment and Review Instrument (JBI-MAStARI)

Appraisal of Guidelines Research and Evaluation (AGREE) instrument

[Note: this appraisal tool is not part of SUMARI as discussed before]

Level II A well-designed randomised controlled trial

JBI-MAStARI software

Level III Controlled trial without randomization (quasi-experimental study)

JBI-MAStARI software

Level IV Single non-experimental study

 Cohort, correlation, case control

 Descriptive, survey and/or observational

 Case reports

JBI-MAStARI software

Level V Systematic reviews of descriptive and qualitative studies

Joanna Briggs Institute Qualitative Assessment and Review Instrument (JBI-QARI). Not used as no evidence related to the topic found

Level VI Single descriptive or qualitative study JBI-QARI. Not used as no evidence related to the topic found

Level VII Opinion of experts and/or reports or expert committees, conference papers Literature review papers or best practice information sheets or guidelines (added)

Joanna Briggs Institute Narrative, Opinion and Text Assessment and Review Instrument

(JBI-NOTARI) software

Once the study design and method of each paper to be critically appraised were assessed, two reviewers independently performed the critical appraisal using the SUMARI software packages. Consensus was reached and the primary reviewer, namely the researcher, concluded the assessment process in order to move onto the next phase, namely data extraction.

145