_____________________________________________________________________________________________________ www.sciencedomain.org
Should a Trial with Lost Allocation Key be
Incorporated in Meta-analysis?
Bart G. Pijls
1*, Boudewijn L. Borger-van der Burg
2, Larry E. Frisch
3,
Anne J. Vochteloo
4and Rob G. H. H. Nelissen
11Department of Orthopaedics, Leiden University Medical Center, Leiden, The Netherlands.
2
Department of Surgery, Rijnland Hospital, Leiderdorp, The Netherlands. 3
Vancouver Coastal Health Research Institute and School of Population and Public Health, University of British Columbia, Vancouver, British Columbia, Canada.
4OCON/ZGT Orthopedic Centre, Hengelo, The Netherlands.
Authors’ contributions
This work was carried out in collaboration between all authors. Authors LEF, BB, AJV and BGP had the idea of the study. Authors BGP and LEF analyzed the data. Authors BGP, LEF, BB, AJV and RGHHN wrote the initial draft manuscript. Authors BGP and RGHHN provided methodological input and author LEF provided statistical input during the conceptual phase of the study. Critical revision of the manuscript was performed by all authors. All authors read and approved the final manuscript.
Article Information
DOI: 10.9734/BJMMR/2015/17815 Editor(s): (1) Jingli Xu, College of Pharmacy, University of New Mexico, USA. Reviewers: (1) Anonymous, University of Pretoria, South Africa. (2)Denisa Madalina Anastase, Department of Anesthesiology, Clinical Hospital of Orthopedics Foisor, Romania. (3)Carlos Torrens, Orthopaedic Department, Universitat Autonoma de Barcelona, Spain. Complete Peer review History:http://sciencedomain.org/review-history/9839
Received 27th March 2015 Accepted 10th June 2015 Published 18th June 2015
ABSTRACT
Aims: On occasion randomized controlled studies may remain unpublished because the allocation key identifying which group received active treatment and which received placebo (or alternative treatment) has been lost. The purpose of this paper is to explore whether and under what circumstances an unpublished study like NCT00250237 whose allocation key has been irretrievably lost might be incorporated into a meta-analysis, allowing the information contained to contribute, however tentatively, to the evidence base around a clinical topic.
Study Design: Review / meta-analysis
Place and Duration of Study: Vancouver Coastal Health Research Institute and School of Population and Public Health, University of British Columbia and Department of Surgery, Rijnland
Hospital, between 2009 and 2014.
Methodology: For two published trials, we calculated separate effect measures for a simulated lost allocation key, under the alternative assumptions that Groups A or B received active treatment. These two effects were compared visually to the pooled effect from a meta-analysis of all other comparable studies. We present as well several quantitative methods for reconstituting a lost allocation code, involving frequentist meta-analysis, meta-regression, and Bayesian techniques. We apply these to an analysis of NCT00250237.
Results: The simulated lost allocation keys of both trials were successfully identified through visual methods using meta-analysis. Reconstitution in this manner requires that the trial’s effect depart significantly from the null and that there exist a sufficient number of other high quality studies addressing the same clinical topic to allow unbiased calculation of a pooled summary effect. While direct reconstitution may sometimes be possible, our primary approaches are implicitly Bayesian and, as with NCT00250327, allow probabilities to be estimated for each allocation assignment. Conclusion: Under some circumstances studies with a lost allocation key may be incorporated cautiously into meta-analysis.
Keywords: Lost allocation key; RCT; meta-analysis; publication bias.
1. INTRODUCTION
Systematic reviews and the techniques of meta-analysis comprise the basis of evidence-based practice, but valid meta-analysis requires that all relevant studies – published or not – be assessed for inclusion. In 1979 Rosenthal called attention to the “File Drawer Problem,” observing that studies consigned to file drawers, and thus unpublished, are often disconfirmatory [1]. Such “negative” studies may reflect correct inference or Type II error, but in either case omitting them from meta-analysis risks serious bias.
Unpublished positive studies have attracted little attention. It is probable that some of such studies remain unpublished due to sloth, to a change in research direction, to retirement, or even to investigator death. Randomized trials face yet another risk precluding publication: the irrevocable loss of information recording which participants received active treatment. The current paper’s authors successfully completed a trial, only to discover that its allocation key (AK) had disappeared. The study (trial registration NCT00250237) investigated haloperidol versus placebo for preventing post-operative delirium in hip fracture patients, was completed in 2009, was well- designed and executed, and achieved significant results. With AK lost, researchers could not ascertain whether participants had been benefitted or harmed; and so the study was consigned to the file drawer. At least three subsequent meta-analyses have been completed without considering the results of this still-unpublished trial. The purpose of this paper is to explore under what circumstances a study like NCT00250237 with a lost AK might be
incorporated into a meta-analysis, allowing the information contained to contribute, however tentatively, to the evidence base around a clinical topic.
2. MATERIALS AND METHODS
For the purposes of this paper we define the AK of a randomized controlled trial (RCT) as information that allows the researcher to determine which of two or more study groups received active treatment(s) and which received placebo (or comparison treatment.) We consider the situation of a lost AK for two-arm trials comparing active treatment (T) with placebo (P). All patients in group A have received comparable – but unknown - treatments, either T or P, and all patients in group B have all received the alternative. For the indirect method we calculate two effect measures for the lost key study, under the alternative assumptions that Groups A or B received active treatment. In an implicitly Bayesian analysis these two effects were compared visually (or, with results not reported here, using a Student t test) to the pooled effect from a meta-analysis of all other comparable studies.
maximum-likelihood estimator (REML) for risk differences (RD). All analyses were performed with the metafor package for R version 2.13 or – for Bayesian meta analysis – WinBugs 1.4.3 [3].
3. RESULTS
3.1 Direct Approach to Reconstituting the Randomisation Key
A direct approach uses material, examinations or information available from the study itself. If marked A or B in a drug trial, left-over medication can be chemically analyzed to determine the AK. Stored blood, urine, or tissue samples, if available, can be chemically analyzed for active treatment, or in some cases such samples can be obtained and assessed even after the study has concluded [4]. None of these or similar direct approaches was feasible for study NCT00250237.
A variant on the direct method is sometimes possible when it is known from previous studies, or can be subsequently be determined, that there is effect modification by some reliably identifiable factor. The observed risk difference in NCT00250237 had a different sign in the presence and absence of dementia, which was therefore a modifier of the observed effect. NCT00250237 authors had previously conducted an observational study of haloperidol and delirium [5]. This study found that delirium was
reduced among subjects with cognitive impairment who received pre-operative haloperidol, but not among cognitively-normal subjects. That observation pointed to an allocation assignment consistent with that derived from indirect methods described below.
3.2 Indirect Approach to Reconstituting a Randomisation Key
3.2.1 Worked example in the absence of heterogeneity
The study by Cannon et al. is an example of a meta-analysis with significant treatment effect and no heterogeneity [6]. Subjects given high dose statin therapy had a significant reduction in the risk of coronary death or myocardial infarction compared to standard dose statins. The SEARCH study, published 4 years after the Cannon meta-analysis, met the latter’s inclusion criteria [7]. We counterfactually proposed that the SEARCH study’s AK had been lost, and we assigned to it two possible AK reconstitutions, A (Group A=high dose) and SEARCH-B (Group SEARCH-B = high dose.) In the absence of heterogeneity, we adopted an empirical Bayesian hypothesis that the correctly-assigned group would be more likely to have an effect falling within the confidence interval of the Cannon meta-analysis. The corresponding Forest plot in Fig. 1 confirms that SEARCH-A has a correctly--assigned AK [7,6].
Fig. 1. Forest plot of cannon meta-analysis with SEARCH-A and SEARCH-B depicted in the plot
3.2.2 Worked example when heterogeneity is present
The meta-analysis by Colditz et al. [8] found that BCG vaccine (intervention group) significantly reduced risk of tuberculosis compared to placebo (control group). Ten years after this meta-analysis Aronson published a study which met the Colditz inclusion criteria [9]. As in the example above, we assumed counterfactually that Aronson had lost the AK, and we calculated separate effect sizes for Aronson-A (group A received BCG) and Aronson-B (Group B received BCG). Since there is meta-analytic heterogeneity, a prediction interval was used instead of the confidence interval. An allocation assignment whose effect fell within the prediction interval, taking into account latitude as effect modifier, was hypothesized the more likely of the two alternatives.
In a forest plot of the Colditz meta-analysis (Fig. 2), the point estimator of Aronson-A falls within the prediction interval. Although visual inspection of interval boundaries may mislead, here we (correctly) hypothesize that Group A received BCG [10].
Geographic latitude proved a significant moderator in the Colditz meta regression: subjects in northern latitudes had a stronger protective effect from BCG immunization than did
those in southern latitudes [8]. In the absence of a factor like latitude it is also possible to meta-regress outcome on baseline risk, usually defined as the risk in controls [11,12]. A comparison can then be made to the observed control group risk in alternative allocation assignments.
3.2.3 Assessment of NCT 00250237
The odds ratios for the two alternative allocations were 0.5 (A) and 2.0 (B) with non-overlapping 95% confidence intervals. Ordinary random effects meta-analysis of five published studies gave a summary OR of 0.42 (95% CI 0.24 to 0.74). Meta-regression with risk as a moderator similarly favored allocation A, a finding confirmed by Bayesian methods utilizing uninformative priors [12]. Bayesian meta-analysis can be used to predict the interval into which the next-performed clinical trial is likely to fall, in this case a point OR estimate of 0.41 and 0.44 [13]. Six priors were used, with those that favoured between-study variance similar to that derived from already-published studies suggesting a 30 to 100-fold greater likelihood of allocation A. Even under the very wide prediction intervals generated by uninformative uniform priors on tau-squared and tau, the MCMC-simulated probability (ynew < yobs) for allocation B (ynew) was only 11% (5.6% for a prior uniform on tau).
Fig. 2. Forest plot of colditz meta-analysis with Aronson-A and Aronson-B depicted in the plot
Note that the data from Aronson-A and Aronson-B were not included in the meta-analysis, but were only plotted. the aronson study was performed at 44 degrees latitude. (latitude was a potential moderating variable recorded
4. DISCUSSION
RCT are powerful and costly tools that often expose subjects to potentially harmful interventions in the hopes of improving future patients’ clinical outcomes. It is ethically desirable that RCT results be disseminated, as emphasized by the WHO statement of April 2015 [14]. We are not aware of a previous lost AK being reported, but given the ubiquity of error in all complex human activities, we would be very surprised to learn that allocation loss had never previously occurred. The described direct and indirect methods may allow a researcher facing the misfortune of a lost AK, to hypothesize which group is most likely to have received the intervention. It should be intuitively obvious (which we have confirmed by simulations) that indirect methods best distinguish the two options for a lost AK when risk differences depart significantly from the null. The probability of correct AK assignment increases if the assignment inferred from one indirect method is confirmed by another method such as meta-regression. While generally feasible, meta regression on baseline risk should be employed with caution, ideally using Bayesian methods [15].
So, should an otherwise methodologically-sound study with lost AK be incorporated within a meta analysis? There should be no debate if the study is negative: i.e. the effect size is not clinically or statistically significant. Such studies are unlikely to have a recoverable AK because the effects of alternative allocations are statistically indistinguishable, but as with any negative study they add important meta analytic evidence. The debate centers around the approach to a study, like NCT00250237 whose outcome differed significantly from the null and conferred a number-needed-to –treat (or harm) of eight. This means that in the light of a lost allocation key only one of two possible conclusions is correct for NCT00250237. One: haldol reduces the incidence of post-operative delirium after acute hip surgery compared to placebo. Two: haldol increases the incidence of post-operative delirium after acute hip surgery compared to placebo. Given the results from a recent systematic review and meta-analysis, only option one appears valid: haldol reduces incidence of delerium [16]. The implicitly Bayesian arguments in this paper would support including a lost AK study if one or more methods established that one AK assignment was substantially more probable than the other. In an unselected review
of a large number of recent meta-analyses Tam et al found that the direction of effect of the first trial on a topic agreed with subsequent meta-analysis in 99% of cases (95% CI 97-100%) [17]. While limited to a first trial, these findings support our initial indirect approach relying on inspection of the Forest plot.
Removing a lost AK study from the file drawer might seem to introduce a form of circularity into meta-analytic reasoning, using a prior likelihood established thorough meta-analysis to reinforce a “Lysenkoist” bias toward what we already believe true. Adding this study to the pre-existing evidence base does risk reinforcing previously-introduced bias, even more there is also clinical and ethical risk from ignoring knowledge when reasonable inference about its veracity is possible. However, since one of the two allocations is “correct,” a lost AK study with significant results provides additional evidence that should be taken into consideration when interpreting meta-analytic results.
If a study with lost AK is to be incorporated into a meta-analysis, a sensitivity analysis must be done with both AK alternatives separately assessed, and in the absence of previously-published meta analysis, results should be presented both including and excluding the lost AK study. Effect sizes of each allocation should also be reported and their relative plausibilities reflected upon. We think it is possible that both allocations should be included together in the meta-analysis, with weightings of each adjusted to reflect the inferred relative probability that each is the correct AK reconstitution, and with the total weight for both equivalent to that of only one study. This approach would be methodologically difficult to implement and was not utilized in the only published meta-analysis of which we are aware in which a lost AK study (NCT00250237) has been included [16].
5. CONCLUSION
Under some circumstances studies with a lost allocation key may be incorporated cautiously into meta-analysis.
CONSENT
ETHICAL APPROVAL
As this is a methodological study (no patients included) dealing with the issue of a lost randomization key, ethical approval was not applicable.
COMPETING INTERESTS
Authors have declared that no competing interests exist.
REFERENCES
1. Rosenthal R. The file drawer problem and tolerance for null results. Psychological Bulletin. 1979;86(3):638-41.
2. Higgings, Green. Cochrane handbook for systematic review of interventions Chichester UK John Wiley and sons Ltd; 2008.
3. Viechtbauer W. Conducting Meta-Analyses in R with the metafor Package. Journal of Statistical Software. 2010;36(3).
4. Gerace E, Salomone A, Pellegrino S, Vincenti M. Evidence of Haldol (haloperidol) long-term intoxication. Forensic Sci Int. 2012;215(1-3):121-3. 5. Moerman S, Tuinebreijer WE, de Boo M,
Pilot P, Nelissen RG, Vochteloo AJ. Validation of the Risk Model for Delirium in hip fracture patients. Gen Hosp Psychiatry. 2012;34(2):153-9.
6. Cannon CP, Steinberg BA, Murphy SA, Mega JL, Braunwald E. Meta-analysis of cardiovascular outcomes trials comparing intensive versus moderate statin therapy. J Am Coll Cardiol. 2006;48(3):438-45. 7. Armitage J, Bowman L, Wallendszus K,
Bulbulia R, Rahimi K, Haynes R et al. Intensive lowering of LDL cholesterol with 80 mg versus 20 mg simvastatin daily in 12,064 survivors of myocardial infarction: A double-blind randomised trial. Lancet. 2010;376(9753):1658-69.
8. Colditz GA, Brewer TF, Berkey CS, Wilson ME, Burdick E, Fineberg HV, et al. Efficacy of BCG vaccine in the prevention of tuberculosis. Meta-analysis of the
published literature. JAMA. 1994;271(9): 698-702.
9. Aronson NE, Santosham M, Comstock GW, Howard RS, Moulton LH, Rhoades ER, et al. Long-term efficacy of BCG vaccine in American Indians and Alaska Natives: A 60-year follow-up study. JAMA. 2004;291(17):2086-91.
10. Schenker N, Gentleman J. On judging the significance of differences by examining the overlap between confidence intervals. The American Statistician. 2001;55(3):182-86.
11. Thompson SG, Smith TC, Sharp SJ. Investigating underlying risk as a source of heterogeneity in meta-analysis. Stat Med. 1997;16(23):2741-58.
12. Dohoo I, Stryhn H, Sanchez J. Evaluation of underlying risk as a source of heterogeneity in meta analysis: A simulation study of Bayesian and frequentist implementations of three models. Preventive Veterinary Medicine. 2007;81:38-55.
13. Spiegelhalter DJ, Abrams KR, Myles JP. Bayesian Approaches to Clinical Trials and Health-Care Evaluation. London, UK. John Wiley & Sons, Ltd; 2004.
14. Moorthy VS, Karam G, Vannice KS, Kieny MP. Rationale for WHO's New Position Calling for Prompt Reporting and Public Disclosure of Interventional Clinical Trial Results. PLoS Med. 2015;12(4):e1001819. DOI:10.1371/journal.pmed.1001819. eCollection 2015.
15. Dias S, Sutton AJ, Welton NJ, Ades AE. Evidence synthesis for decision making 3: heterogeneity--subgroups, meta-regression, bias, and bias-adjustment. Med Decis Making. 2013;33(5):618-40.
16. Fok M, Sepehry A, Frisch L, Sztramko R, Borger van der Burg B, Vochteloo A, et al. Do antipsychotics prevent postoperative delirium? A Systematic Review and Meta-analysis. Journal of Geriatric Psychiatry. 2015;30(4):333-44.
APPENDIX
The Prediction Interval :
(1) PI = M ± t * sqrt(T2 + se2)
PI = Prediction interval
M = Observed (calculated) mean effect
t = t value (for a certain probability e.g. 95% prediction interval) T2 = Tau squared = between studies variance
se2 = Standard error of the summary effect squared = variance of the summary effect
In the absence of heterogeneity tau squared (T2) is zero, so according to the formula by Higgings et al the prediction interval becomes the same as the confidence interval: PI = M ± t * sqrt(0 + se2)
© 2015 Pijls et al.; This is an Open Access article distributed under the terms of the Creative Commons Attribution License
(http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium,
provided the original work is properly cited.
Peer-review history:
