Definition

Egger et al., (2001a) combined several definitions to create the following:

A review that has been prepared using a systematic approach to minimising biases and random errors which is documented in a materials and methods section. A systematic review may, or may not, include a meta-analysis: a statistical analysis of the results from

independent studies, which generally aims to produce a single estimate of a treatment effect.

Egger, Davey Smith and O’Rourke (2001) p. 5.

A systematic review should contain a clearly focused question (Main, 2003) and be conducted to an explicit and reproducible methodology (Greenhalgh, 1997) which strives to avoid the subjectivity and selection bias of traditional, or narrative, reviews (Main, 2003). The use of meta-analyses allows a single answer to a specific question to be produced from the multiple component studies (Main, 2003). Systematic reviews therefore are used to reduce or resolve uncertainty about evidence (Pettigrew, 2003).

3.2.3 BACKGROUND.

The increasing body of published evidence makes it difficult for clinicians and researchers to remain up to date (Main, 2003) and for effective policies and guidelines to be developed by health service providers and decision makers (Mulrow, 1994). Previous research by the author of this thesis has found that physiotherapists in particular encounter problems locating and accessing literature (Beeston, et al., 2001) with trials published in “almost every journal you can think of” many of which are not included in common databases such as Medline or Embase (Knipschild, 1994). Literature reviews are used to improve accessibility to research

findings. Traditionally narrative or journalistic reviews of the evidence have occurred, lacking a standardised and objective approach to the location and analysis of evidence and open to the personal biases of authors to present evidence supporting their own beliefs/theories

(Greenhalgh, 1997). The consequences of this include confusion regarding choice of

treatment, or poor/harmful practice being wrongfully continued (Chambers, 2001 p. xiv). The need for an objective, meaningful appraisal of the literature prior to undertaking research seems apparent. Systematic reviews can save resources and prevent unnecessary research occurring in areas where conclusive evidence exists (Mulrow, 1994). Reviews have been increasingly popular in medicine from the 1980s; especially after 1992 and the introduction of the Cochrane Collaboration (Montori et al., 2003), an international organisation dedicated to the preparation and maintaining of systematic reviews regarding the effectiveness of health care interventions (Chalmers, 1993). The result has been that within a relatively short period of time systematic reviews have become “one of the cornerstones of evidence-based

medicine” (Moher, 2008). As already mentioned, systematic may or may not include meta-analyses. Where meta-analyses are appropriate and possible for a review these are considered beneficial since they improve power¹ and precision², answer questions not addressed by individual studies and address controversies arising from conflicting studies Deeks et al., (2006). However, sometimes the wide scope of a review, or the diversity within component trials, mean that it becomes meaningless and potentially misleading to compare component studies (Egger, Davey Smith and O’Rourke, 2001a p.4-5; Main, 2003). In physiotherapy, it is often impossible to acquire data in a format permitting meta-analyses (Main, 2003) or

sufficient high quality studies to achieve clinically relevant systematic reviews.

1 Power is “ the chance of detecting a real difference as statistically significant if it exists” (Deeks et al., 2006).

Combining multiple studies may mean there is a greater chance of detecting an effect.

2 Precision refers to the degree of variability of an observed sample statistic (Sim and Reid, 1999).

The use of systematic reviews, and meta-analyses in particular, remain controversial with some statisticians and clinicians not accepting either the mathematics or the findings:

confusion caused by conflicting reviews upon the same topic and of reviews of small trials being subsequently contradicted by larger definitive trials has contributed to this dissent (Egger et al., 2001c p. 43). These discrepancies may be due in part to poor methodologic quality in smaller trials (Kjaergard et al., 2001) plus the results of a review may be unable to predict the results of large trials for up to a third of the time (Main, 2003).

It has been suggested that, for public health and social interventions, the “stainless steel” law of evaluation exists (Pettigrew, 2003), namely, “the more rigorous the review, the less evidence there will be that the intervention is effective” (Pettigrew, 2003). Furthermore, systematic reviews are able to answer single questions regarding specific interventions; they are less likely to provide clear answers to questions regarding complex interventions

(Pettigrew, 2003) and it might be argued that this could be true for many areas of physiotherapy which incorporate complex interventions.

Despite being more explicit and objective than narrative reviews, systematic reviews may still be susceptible to bias (Main, 2003). An overview of sources of bias during the location and selection of trials has been provided by Egger and Davey Smith (1998) and summarised in the figure below. Such bias frequently results in an overestimation of a treatment. A further source of potential bias is reporting bias; there may rapid or delayed publication, or selective reporting, of a trial depending on the nature and direction of results (Eggar et al., 2001c p. 51-62).

Overall, despite the limitations of the approach, systematic reviews are advocated as the best available approach which currently exists to summarise and synthesise data (Main, 2003) and are considered a major advance in the objective review of evidence (Crombie and

McQuay, 1998). Please see the table on the next page for a summary of the advantages and limitations of this approach.

Inclusion Bias

The criteria for including studies may be influenced by knowing the results of potential studies; therefore the review

might selectively include/exclude studies i.e. Selection Bias

Citation Bias Database Bias

Can occur from using reference lists In less developed countries studies from trials because significant results are with significant results are more likely cited more frequently (regardless of size to be published in a journal indexed

and quality) in a literature database

English Language Bias Bias in Provision of Data

Meta-analyses published in English When information additional to the trial

language journals are often based on report is required this may be difficult to trials published in English (which are obtain since the investigators may be

more likely to be significant) unwilling to make their data available

Possible Sources of BIAS

Affecting a Systematic Review

Multiple Publication Bias Publication Bias

Multiple publications from single studies are studies with significant results are more more likely to be studies with significant results. likely to get published therefore there is a The inclusion of duplicate data can overestimate risk of meta-analysis showing spurious

treatment effects beneficial treatment effects

Need to Examine for Absence / Presence of Bias by

Sensitivity Analyses Funnel Plots Figure 2. Summary of the Sources of Bias when locating and selecting studies for inclusion into a Systematic Review (Summarised using Eggar and Davey Smith, 1998).

Advantages of Systematic Reviews

Limitations of Systematic Reviews

Vast amounts of literature can be summarised and presented to health care professionals and service decision makers

Promotes the use of focused questions

Review methodology is reproducible and explicit Explicit methodologies may limit bias in locating and selecting studies

Increased reliability and accuracy of findings (i.e.

effect sizes and risks)

Meta-analysis can increase the precision of results Increased statistical power, useful for areas with low event rates or smaller effect sizes

The time taken to implement research findings into practice may be reduced

Can establish the generalisability and heterogeneity of results

Can generate new hypotheses for future research from main findings or sub group findings May identify areas of inadequate research

Can prevent obsolete research being undertaken in an already adequately explored area

Efficient use of existing data

Physiotherapy studies with large numbers, and multi-centre trials, are still rare. Large trial results are often not yet available thus systematic reviews may be able to inform practice more quickly.

Poor/ differences in methodological decision making can still lead to erroneous and/or conflicting results.

There is a lack of agreement regarding the assessment of study quality; many measures are inappropriate for physiotherapy studies.

Quality affects results. Inadequate randomization, allocation concealment and double blinding effect treatment estimates.

There is a lack of consensus regarding the appropriateness of analysis strategies used There are often insufficient physiotherapy studies to enable successful systematic reviews to occur Physiotherapy papers often do not report the information required for successful meta-analysis Physiotherapy studies may be too diverse to allow meaningful meta-analysis to occur Combining data may disguise / oversimplify distinctions between studies

Bias, such as selection bias, publication bias and language bias, can still be introduced

The results of a systematic review may not predict trial results for up to a third of the time

Systematic reviews are often criticised as being unable to give specific guidance; often because the studies provide few outcome evaluations There may be a “stainless steel” law in operation, that is, the more rigorous the review, the less evidence there is that the intervention is effective.

Reviews may overlook important clinical details in the component papers, thereby reducing their validity

Table 1. Summary of the advantages and limitations of systematic reviews for Physiotherapy (Summarised using Mulrow, 1994; Greenhalgh, 1997; Hopayian, 2001; Main, 2003; Pettigrew, 2003).

A systematic review of the literature evaluating the effectiveness of post operative physiotherapy exercise following elective joint replacement appears worthwhile prior to undertaking the proposed trial for the following reasons:

1. To systematically determine the extent of previous research; checking the value of the question and whether it has already been adequately addressed (Mulrow, 1994;

Knipschild, 1994).

2. To inform the development of the proposed trial intervention using previous studies (Knipschild, 1994), assisting the establishment of its theoretical basis (MRC, 2000) and lessening the chance of mistakes (Knipschild, 1994).

3. To inform the choice of appropriate outcomes for the trial.

4. To assist other clinicians seeking easily accessible knowledge (Greenhalgh, 1997; Main, 2003) in this common area of practice.

5. To provide, if possible, information regarding treatment effect size and its consistency (Mulrow, 1994).

6. If the level of heterogeneity permits, to increase the power to detect treatment effects (Pettigrew, 2003).

7. To determine the level of generalisability of the findings; since the diversity of multiple studies provides an interpretive context wider than in any single study (Mulrow, 1994).

8. To identify and seek to explain inconsistencies and conflicts in the research/data (Mulrow, 1994).

Issues needing consideration to produce a high quality review are summarised in Table 1 and highlighted, where relevant, in the protocol.