Figure 6. Date of last search for evidence for identified interventions.
For some interventions and topics, we have identified a large num- ber of overlapping reviews, and determining the most comprehen- sive and up-to-date review was complex. We urge researchers to take action to avoid publication of overlapping or similar reviews by searching for reviews and protocols before initiating a review, by publishing review protocols and by clearly highlighting when a new publication supersedes previous publications. Registration and publication of Cochrane reviews is designed to avoid the challenges associated with overlapping reviews, and the Cochrane Stroke Review Group takes steps to ensure that no overlap occurs between Cochrane reviews. When a Cochrane review is out-of- date, researchers interested in an updated review on that topic or intervention are encouraged to contact the Cochrane Stroke Re- view Group to discuss collaboration on updating the review, rather than preparing an alternative journal publication.
Applicability of evidence
The aim of this overview was to synthesise best evidence on upper limb rehabilitation interventions into a single, accessible, compre- hensive document, thus supporting clinicians and policy makers in clinical decision making for stroke rehabilitation. However, the aim was not to bring together all evidence required to make an individual treatment decision about an individual patient within a specific setting. This overview serves to signpost clinicians and policy makers toward relevant systematic reviews to support clin- ical decisions. It is the nature of stroke rehabilitation research and clinical practice that the application of evidence to an individual patient or healthcare setting will depend on the specific details of that patient or setting, and that clinical decisions require expert clinical reasoning and judgement if available evidence is to be in- terpreted and applied effectively. Before any evidence is applied, we therefore recommend that clinicians and policy makers are
guided to the appropriate review, and that they consider carefully the details of the trials synthesised within that review, specifically reflecting on the relevance of the participant population, trial set- ting and context, interventions delivered and outcomes assessed in relation to the clinical decision to be made. We believe that, given the large volume of overlapping evidence and the variable quality of this evidence, this overview can serve to efficiently guide clini- cians and policy makers to the most appropriate review evidence. Within this overview, in addition to variations among participants, interventions, setting and context, we specifically found that the dose of interventions, outcomes and comparisons were central to assessment of the potential applicability of evidence. Further discussion related to the impact of these on the applicability of evidence is provided inAppendix 5.
Quality of the evidence
Assessment of quality of included reviews
We assessed the quality of included reviews using a modified ver- sion of the AMSTAR tool to derive answers to the original AM- STAR questions (Table 1). Despite a number of challenges associ- ated with development and use of the mAMSTAR and AMSTAR tools (seeAppendix 3for further discussion and details), we be- lieve that our use of mAMSTAR questions has provided substan- tial benefit, and that our clear reporting of agreed upon responses (inFigure 4) enhances the transparency of our judgements and provides the reader with a detailed overview of methodological components of each review.
Quality of included reviews
We have provided a detailed, transparent assessment of the qual- ity of included reviews in Figure 4and Table 8and have de- scribed issues related to each of the 11 AMSTAR questions in
Methodological quality of included reviews. There is clearly a dif- ference in the number of ’yes’ responses between Cochrane reviews and non-Cochrane reviews. However, the data demonstrate that many of these differences are accounted for by poor reporting of information within some of the non-Cochrane reviews (i.e. lack of ’yes’ responses reflects an absence of, or unclear, information, rather than reflecting poor methods per se).
Within the included reviews, we have identified various methods of assessing and reporting the quality of included studies. These are briefly summarised and discussed inTable 23.
In the past, full and adequate reporting of methodological de- tails of reviews has been challenging because of the word restric- tions of a journal publication. However, this should no longer be a limitation of adequate reporting, now that most journals pro- vide opportunities for publication of online supplementary ma- terial (Hoffmann 2014a). Despite opportunities for online ma- terial, we found less comprehensive reporting in non-Cochrane
reviews, which, for example, rarely reported details of excluded studies. For reviews to be useful and inform clinical decisions, ad- equate reporting of methods is essential. We urge review authors and journal editors to ensure that minimum reporting standards are achieved. As guidelines and checklists are increasingly used by journal editors in considering study and review methodology, this endeavour should support improved reporting.
Many reviews of stroke rehabilitation interventions will include trials that explore a wide range of diverse interventions, partici- pants and outcome measures. This diversity presents additional challenges to review quality. If reviews are to inform clinical prac- tice, it is essential that they contain adequate descriptions of inter- ventions investigated and participants included. We believe that further work is required to enhance reporting and assessment of these details in a systematic and clinically relevant way, and that this will be supported by the use of tools such as the recently de- veloped template for intervention description and replication (TI- DieR) checklist (Hoffmann 2014b). Often review authors make important decisions related to whether to pool (or to not pool) data arising from relatively diverse trials. Such decisions should always be fully explored and discussed to highlight the benefits and limitations associated with the decision, and appropriate steps should be taken by review authors to avoid the introduction of bias at this stage of the review process. We believe that further work is required to establish transparent methods designed to avoid in- troduction of bias at the stage of decision making related to meta- analyses of data related to diverse interventions.
Assessment of quality of evidence in included reviews Systematically establishing the quality of evidence has been central to this overview, and considerable work has gone into ensuring objective and consistent application of GRADE levels of evidence to all comparisons contributing data to this review. Our methods of objectively determining GRADE levels of evidence, based on assessment of the quality of included reviews and the quality of trials within the included reviews, are described in the methods section, and further details are provided inAppendix 4.
Further work is clearly required to explore our methods of ap- plying GRADE levels of evidence. However, in the absence of this, we believe that our objective application and determination of GRADE levels of evidence provide substantial benefit to our overview. We have assessed the quality of evidence using a trans- parent, objective process, with consideration of both the quality of the review and the trials included within the review, while remov- ing potential risk of bias associated with subjective interpretation and application of this evidence. Further discussion related to our method of objectively determining GRADE levels of evidence is provided inAppendix 4.
Quality of evidence in included reviews: GRADE levels of evidence
39