Assessment of Risk of Bias Among Pediatric Randomized Controlled Trials

(1)

Randomized Controlled Trials

WHAT’S KNOWN ON THIS SUBJECT: The introduction of bias into the design and conduct of RCTs can affect the believability of the results severely. The Cochrane Collaboration has designed a domain-based, evaluation tool to address internal validity and the risk of bias directly.

WHAT THIS STUDY ADDS: Many pediatric RCT reports demonstrated high risk of bias for sequence generation and allocation concealment. Industry funding and assessment of behavioral/educational interventions were associated with high risk of bias, whereas trial registration was associated with lower risk of bias.

abstract

OBJECTIVE:The goal was to assess the risk of bias among pediatric, randomized, controlled trials (RCTs) reported in 8 high-impact journals.

METHODS:We searched PubMed for all pediatric RCTs reported be-tween July 1, 2007, and June 30, 2008, in 8 journals with high impact factors. Using Cochrane Collaboration methods for risk assessment, we evaluated all reports for risk of bias according to domain (ie, ran-domized sequence generation, allocation concealment, masking, in-complete outcome data, selective outcome reporting, and other). We used multiple logistic regression to test for associations between the presence of a high risk of bias according to domain and funding source, intervention type, trial registration, and multicenter status.

RESULTS:Industry-funded RCTs were more likely to show a high risk of bias for sequence generation, compared with government-funded RCTs (adjusted odds ratio [aOR]: 6.1 [95% conﬁdence interval [CI]: 1.70 – 21.89]), and behavioral/educational trials were more likely to show a high risk of bias for sequence generation (aOR: 2.8 [95% CI: 1.06 –7.36]) and allocation concealment (aOR: 4.09 [95% CI: 1.69 –9.90]), compared with drug trials. Registered trials were less likely to have a high risk of bias for sequence generation, compared with nonregis-tered trials (aOR: 0.33 [95% CI: 0.15– 0.71]).

CONCLUSIONS:Overall, we found a large proportion of pediatric RCT reports with a high risk of bias for sequence generation and allocation concealment. Factors associated with a high risk of bias included in-dustry funding and assessment of behavioral/educational interven-tions, whereas trial registration was associated with a lower risk of bias.Pediatrics2010;126:298–305

AUTHORS:Michael T. Crocetti, MD, MPH,a_{Diane D. Amin,}

PA-C,a_{and Roberta Scherer, PhD}b

a_{Department of Pediatrics, Johns Hopkins Bayview Medical}

Center, Baltimore, Maryland; andb_{Department of Epidemiology,}

Bloomberg School of Public Health, Johns Hopkins University, Baltimore, Maryland

KEY WORDS

randomized, controlled trial, risk of bias, high-impact journal, Cochrane Collaboration

ABBREVIATIONS

RCT—randomized, controlled trial OR— odds ratio

CI— conﬁdence interval aOR—adjusted odds ratio

www.pediatrics.org/cgi/doi/10.1542/peds.2009-3121

doi:10.1542/peds.2009-3121

Accepted for publication Apr 26, 2010

Address correspondence to Michael T. Crocetti, MD, MPH, Johns Hopkins Bayview Medical Center, Department of Pediatrics, 4940 Eastern Ave, Baltimore, MD 21224. E-mail: [email protected]

PEDIATRICS (ISSN Numbers: Print, 0031-4005; Online, 1098-4275).

(2)

Randomized, controlled trials (RCTs) are considered standard for assess-ments of the efﬁcacy of therapeutic in-terventions. RCTs typically are de-signed to assess the efﬁcacy of a drug, medical or surgical procedure, or behavioral/educational intervention. RCTs provide the highest level of evi-dence because, when conducted prop-erly, they have lower risk of bias than do other study designs. Bias is a “sys-tematic error in the design of a trial that can lead to underestimation or overestimation of the true interven-tion effect.”1 _{Standard RCT design}

characteristics, such as randomiza-tion, masking, accounting for loss to follow-up monitoring, and appropriate reporting, serve to minimize bias and to ensure internal validity.

Numerous tools have been developed to assess the validity or quality of RCTs.2_{These tools typically consist of a}

scale or checklist of items thought to be critical in the design and reporting of RCTs. Unfortunately, there are nu-merous differences among the tools regarding categories for review, com-plexity, and weight assigned to the in-dividual domains thought to be critical to the control of bias.3 _Furthermore,

many of the scales concentrate on re-porting quality, rather than the inter-nal validity of a given trial.3_In

assess-ments of the risk of bias among RCTs, it is important to distinguish simple re-porting of procedures from the actual design and conduct of a study. To ad-dress directly internal validity and risk of bias, the Cochrane Collaboration designed a domain-based, evaluation tool recommended for use in system-atic reviews of trial reports.1 _The

do-mains within the tool are sequence generation, allocation concealment, masking, handling of incomplete out-come data, selective outout-come report-ing, and other potential threats to va-lidity. These domains were chosen on the basis of empirical evidence that they

affect the magnitude and direction of the treatment response and thus are mea-sures of internal validity.1

The Cochrane Collaboration, domain-based tool for assessment of the risk of bias has been used in the setting of systematic review of RCTs. In this set-ting, levels of bias are assigned to tri-als within a particular research area and results are pooled to yield an as-sessment of the believability of the treatment effect. Accompanying the assessment is a clear statement of why a particular domain was assessed as having low, high, or unclear risk of bias. Recently, the interrater reliability of the Cochrane Collaboration, risk-of-bias tool was evaluated by Hartling et al.4 _{The authors also compared this}

tool with 2 other assessment tools. They demonstrated that the interrater reliability was good for some domains but poor for others. Furthermore, the tool required longer to complete, com-pared with the other tools, and there were weak correlations between the overall risks of bias assessed with the Cochrane Collaboration, risk-of-bias tool and the other measures. We were interested in assessing whether the tool could be used to evaluate the over-all quality of RCTs conducted on a spe-ciﬁc research topic or within a medical discipline.

RCTs have the potential to alter signif-icantly the clinical care of patients and to improve health outcomes. The intro-duction of bias into the design and con-duct of a RCT can severely affect the quality of the research. Potentially more important is the fact that bias can render the results of a RCT invalid. The use of invalid results in medical decision-making may subject patients to undue risk of harm and therapeutic interventions that have no beneﬁt. Trial results are generally dissemi-nated through journal publications, and a measure of the importance of a particular article or of the journal

it-self is the impact factor.5_High-impact

journals publish numerous inﬂuential RCTs, which often set the standard of care in medicine. The general medical journals with the highest-rated journal impact factors in the 2006Journal

Ci-tation Reportswere theNew England

Journal of Medicine, theJournal of the

American Medical Association, and

The Lancet.6_{There is concern that RCTs}

performed with pediatric populations are not well represented in general medical journals.7_{Barriers to the}

pub-lication of pediatric RCTs may include a decreased burden of disease in chil-dren, compared with adults, or ethical issues related to the recruitment of pe-diatric patients. Pepe-diatric RCTs may be preferentially submitted to and ac-cepted by pediatric journals, rather than general medical journals. Never-theless, it is critical that pediatric tri-als use proper trial design and con-duct in assessments of therapeutic interventions, to minimize bias and to improve internal validity. By using the Cochrane Collaboration, risk-of-bias tool, the objective of this study was to assess the risk of bias among pediat-ric RCTs reported during a 1-year pe-riod in 3 general medical journals and 5 pediatric journals with the highest impact factors.

METHODS

We conducted a PubMed search to identify all pediatric RCT reports pub-lished between July 1, 2007, and June 30, 2008. We selected the 5 pediatric journals and 3 general medical jour-nals with the highest rated impact factors. The pediatric journals were

Pediatrics (impact factor: 5.01), the

Journal of the American Academy of

Child and Adolescent Psychiatry

(im-pact factor: 4.77), theJournal of Pedi-atrics(impact factor: 3.99),Archives of Pediatrics and Adolescent Medicine

(impact factor: 3.57), and thePediatric

Infectious Disease Journal (impact

(3)

Lancet(impact factor: 25.80), and the

Journal of the American Medical Asso-ciation (impact factor: 23.18). Addi-tional search criteria included English language, RCT, and all-child (0 –18 years of age). Trial reports were ex-cluded if the study population was pre-dominately adult, participants were assigned to study groups in a nonran-domized manner, or the study was a follow-up report from a RCT reported before July 1, 2007.

We abstracted information from the trial reports regarding trial character-istics, including the number of au-thors, the country in which the study was conducted, a broad research cat-egory, multicenter status, and whether the trial had been registered. In addi-tion, we documented funding sources (government, industry, internal hospi-tal grant, multiple sources, none, or private foundation) and trial inter-vention type (educational/behavioral, medicine/drug, nutritional supple-ment, or vaccine). By using Cochrane Collaboration methods for risk assess-ment, we evaluated all reports for risk of bias within 6 domains, including randomized sequence generation; al-location concealment; masking of par-ticipants, personnel, and outcome as-sessors; incomplete outcome data reporting; selective outcome report-ing; and other sources of bias. By using a priori definitions, we classified each domain according to risk of bias, as high, low, or unclear (definitions pro-vided in the Appendix). The assess-ment of masking and incomplete out-come data reporting was based on the primary trial outcomes. Two reviewers scored each report, and a third re-viewer adjudicated disagreements.

Before the start of the study, each re-viewer pretested the assessment tool on 1 RCT from each of the 8 journals. For analysis, we dichotomized bias as

ports were classified as having a low risk of bias for selective outcome re-porting and other biases, these do-mains were not included in the final regression analysis. We performed a simple descriptive analysis, calculat-ing means for continuous variables and proportions for categorical vari-ables. We used multivariate logistic re-gression to test for an association be-tween the presence of a high risk of bias according to domain and the inde-pendent variables of funding source, intervention type, author number, and trial registration status. Regression models were explored by using likeli-hood ratio tests, forward and back-ward stepwise selection for each in-dependent variable, and interaction terms (Stata 9 [Stata, College Station, TX]). Colinearity among the indepen-dent variables in the final model was checked by using variance inflation factors. The mean variance inflation factor was 1.06 (range: 1.00 –1.11). A large proportion of the covariate pat-tern (137 of 146 RCTs) consisted of unique values; therefore, we used the Hosmer-Lemeshow goodness-of-fit test to check the logistic model for each domain. Odds ratios (ORs) with 95% confidence intervals (CIs) are re-ported. A 2-sidedPvalue ofⱕ.05 was considered statistically significant.

RESULTS

Trial Report Characteristics

We identiﬁed 244 trial reports that met the search criteria. Ninety-eight arti-cles were excluded, which left 146 RCTs for review. Studies were excluded because they included only adult par-ticipants (25 studies), were follow-up studies (49 studies), or were not ran-domized (24 studies). The majority of RCTs (116 [79%] of 146 RCTs) were re-ported in pediatric journals (Table 1). The mean number of authors for each

article was 9 (median: 7 authors [range: 2–29 authors]), and an investi-gator with a PhD degree was listed as an author in almost all of the reports (131 [90%] of 146 RCTs). Sixty-ﬁve (45%) of the studies were multicenter, and 86 (59%) reported a clinical trial registration number. Reported fund-ing sources and types of research are presented in Table 1. Most of the trials were conducted in the United States (63 [43%] of 146 RCTs) or in multiple countries (18 [12%] of 146 RCTs). Eigh-teen different medical categories were represented, with infectious diseases (38 [26%] of 146 RCTs) and neonatol-ogy (27 [18%] of 146 RCTs) being most common.

Risk of Bias

The bias designation for all of the RCTs according to domain is shown in Fig 1. Seventy-six (52%) of the trial reports required adjudication for ⱖ1 of the risk-of-bias domains. A high/unclear risk of bias was found in 41% of re-ports (60 of 146 rere-ports) for sequence generation, 57% (83 of 146 reports) for Journal

Pediatrics 56 (39)

Journal of Pediatrics 21 (14)

The Lancet 15 (10)

Archives of Pediatrics and Adolescent Medicine

13 (9)

Journal of the American Academy of Child and Adolescent Psychiatry

13 (9)

Pediatric Infectious Disease Journal 13 (9)

New England Journal of Medicine 8 (5)

Journal of the American Medical Association

7 (5)

Funding source

Government 44 (30)

Industry 36 (25)

Multiple sources 32 (22) Private foundation 22 (15)

No funding 10 (7)

Internal hospital grant 2 (1) Type of research

(4)

allocation concealment, 19% (28 of 146 reports) for masking, 11% (16 of 146 reports) for incomplete outcome data, 2% (3 of 146 reports) for selective out-come reporting, and 2% (3 of 146 re-ports) for other biases. 2 3 4 5 Tables 2 through 5 show unadjusted ORs and adjusted ORs (aORs) for a high risk of bias for the sequence generation, allo-cation concealment, masking, and in-complete outcome reporting domains. Industry-funded RCTs were more likely (aOR: 6.10 [95% CI: 1.70 –21.89]) and privately funded RCTs (aOR: 0.23 [95% CI: 0.06 – 0.97]) were less likely to show a high risk of bias for sequence gener-ation, compared with government-funded RCTs. Behavioral/educational trials were more likely to show a high risk of bias for sequence generation (aOR: 2.80 [95% CI: 1.06 –7.36]) and al-location concealment (aOR: 4.09 [95% CI: 1.69 –9.90]), compared with drug trials. Registered trials (aOR: 0.32 [95% CI: 0.14 – 0.73]) and trials funded by pri-vate foundations (aOR: 0.23 [95% CI: 0.06 – 0.97]) were less likely to have a high risk of bias for sequence genera-tion, compared with nonregistered trials. As the number of authors in-creased, trial reports were less likely to have a high risk of bias for incom-plete outcome reporting (aOR: 0.78 [95% CI: 0.62– 0.98]).

DISCUSSION

By using a domain-based tool, we as-sessed the risk of bias among

pediat-ric RCTs reported in general medical and pediatric, high-impact journals during a 1-year period. The Cochrane Collaboration, domain-based,

risk-assessment tool was introduced as a potentially reliable way to assess the quality of individual RCTs. It uses pre-deﬁned bias indicators within domains

that have been shown empirically to affect trials’ internal validity by alter-ing the size of the treatment effect. We found that pediatric RCTs are con-0

20 40 60 80 100 120 140 160

Unclear High Risk Low Risk

Num

ber

n = 146

FIGURE 1

Risk of bias according to Cochrane Collaboration domain.

TABLE 2 High Risk of Bias for Sequence Generation Domain (N⫽146)

Variable Unadjusted OR (95% CI)

P aOR (95% CI) P

No. of authors 0.96 (0.89–1.03) .26 1.01 (0.92–1.11) .82 Registered 0.37 (0.19–0.74) .005 0.32 (0.14–0.73) .006 Funding

Government Reference

Industry 2.56 (1.03–6.33) .04 6.10 (1.70–21.89) .005 Private 0.23 (0.06–0.89) .03 0.23 (0.06–0.97) .05 Internal grant 1.44 (0.08–24.63) .80 2.74 (0.14–55.12) .51 Multiple sources 0.57 (0.21–1.50) .25 0.60 (0.21–1.71) .34

None 2.17 (0.53–8.79) .28 1.85 (0.41–8.36) .42

Research type

Medicine/drug Reference

Vaccine 1.65 (0.56–4.85) .36 0.55 (0.14–2.21) .40 Educational/behavioral 1.39 (0.65–2.96) .39 2.80 (1.06–7.36) .04 Nutritional supplement 0.55 (0.16–1.94) .35 1.42 (0.31–6.50) .65

The adjusted regression model included author number, registration status, funding source, and research type. ORs of⬎1 indicate a high risk of bias. Hosmer-Lemeshow goodness of ﬁt:␹

2

8⫽_9.02;_P⫽_.34.

TABLE 3 High Risk of Bias for Allocation Concealment Domain (N⫽146)

P aOR (95% CI) P

No. of authors 0.96 (0.89–1.03) .28 1.02 (0.93–1.11) .71 Registered 0.80 (0.41–1.57) .52 0.69 (0.32–0.1.50) .35 Funding

None 0.78 (0.19–3.18) .72 0.78 (0.17–3.54) .75

Research type

2

8⫽_7.40;_P⫽_.49.

(5)

ducted throughout the world and across many medical categories; how-ever, a large proportion of the trials were determined to have a high or un-clear risk of bias for sequence gener-ation and allocgener-ation concealment. Bias within these reporting domains may signiﬁcantly affect the internal validity of the trial results, and caution should be taken in interpretation of the ﬁnd-ings of these studies.

The Cochrane Collaboration, domain-based, risk-assessment tool repre-sents a method to assess objectively the internal validity of RCTs. The ﬁrst domain in the tool, randomized se-quence generation, protects against

selection bias and increases the chance that, on average, predictor variables are balanced between inter-vention groups. We found a high or un-clear risk of bias for sequence gener-ation in 41% of the trial reports. Authors often reported that the trial was randomized without stating how the randomization sequence was gen-erated. In this study, industry-funded RCTs were 6 times more likely to have a high or unclear risk of bias for se-quence generation, compared with government-funded RCTs. Studies re-vealed that randomized drug trials funded by for-proﬁt organizations such as industry sources were 4 to 5

trials funded by nonproﬁt organiza-tions.8,9_{For results such as these to be}

believable, it is imperative that

pediat-ric RCTs funded by industry sources exercise more care in their reporting of the methods used to assign partici-pants to study groups.

Allocation concealment further pro-tects against selection bias. When per-formed properly, allocation conceal-ment ensures unpredictability in the next treatment assignment. In 57% of trial reports, we found a high or un-clear risk of bias for allocation

con-cealment. RCTs in which allocation concealment is inadequate yield exag-gerated treatment effects.10,11_{All RCTs}

should have methods in place to shield investigators and participants from knowing the next treatment assign-ment during the period of time from sequence generation until the

partici-pant is placed in a treatment group. We found that trials assessing behavioral or educational interventions were 4 times more likely to have a high or un-clear risk of bias for allocation con-cealment, compared with drug trials.

Masking of participants, personnel, and outcome assessors protects against information bias. We found that 19% of the trial reports had a high or unclear risk of bias for masking and, although results did not reach

statistical signiﬁcance (P⫽.07), trials involving behavioral or educational in-terventions were 3 times more likely to have a high risk of bias for this domain, compared with drug trials. The most likely explanation for this is that mask-ing participants and assessors to the intervention in behavioral/educational trials is extremely difﬁcult or

impossi-ble, because of the underlying design. In addition, outcomes in behavioral/ educational trials tend to be subjec-tive, which places such trials at higher No. of authors 0.95 (0.86–1.05) .33 0.99 (0.88–1.11) .83

Registered 0.91 (0.40–2.10) .83 0.86 (0.34–2.17) .76 Funding

None 0.97 (0.18–5.40) .98 0.97 (0.17–5.74) .98

Research type

2

8_⫽_8.5;_P_⫽_.39.

TABLE 5 High Risk of Bias for Incomplete Outcome Domain (N⫽146)

P aOR (95% CI) P

No. of authors 0.82 (0.68–0.99) .04 0.78 (0.62–0.98) .04 Registered 0.38 (0.13–1.10) .07 0.55 (0.18–1.72) .30 Funding

Industry 1.02 (0.28–3.67) .97 0.62 (0.12–3.24) .57 Private 0.63 (0.12–3.43) .60 0.65 (0.11–3.73) .63 Internal grant

Multiple sources 0.42 (0.08–2.24) .31 0.41 (0.07–2.38) .32

None 0.70 (0.08–6.60) .76 0.36 (0.03–3.71) .39

Research type

2

(6)

risk of bias for an exaggerated treat-ment effect.1

Attrition bias occurs when not all ran-domly assigned study participants complete the trial or some outcome assessments are not completed and not taken into account in the analysis. When data are missing, they should be balanced across intervention groups and an intention-to-treat analysis per-formed. In addition, statistical meth-ods used to deal with missing data should be described. Wood et al12

showed that missing outcome data of-ten are not handled with adequate sta-tistical methods in RCT reports pub-lished in high-impact general medical journals. Overall, 11% of the trial re-ports in this study had a high or un-clear risk of bias for incomplete out-come data reporting. Trial reports were 22% less likely to be at high risk for incomplete outcome reporting for each additional author. It is not clear why increased numbers of authors would be associated with proper re-porting of incomplete outcomes.

In 2005, the International Committee of Medical Journal Editors developed a policy that information about RCTs should be deposited in a clinical trial registry.13 _{The purpose of this policy}

was to make the details of trials with human subjects transparent to the public. ClinicalTrials.gov, a large, US-based, trial register, currently con-tains⬎70 000 studies from 164 coun-tries.14 _{The number of trials in the}

registry continues to increase, and this may be an indication that more journals are requiring trial registra-tion numbers for pediatric RCT sub-missions. Clinical trial registration is required by 5 of the 8 high-impact jour-nals included in this study and is en-couraged by 2 others. We found that only 59% of pediatric RCTs (86 of 146 RCTs) reported trial registration num-bers. Furthermore, for journals that require registration numbers, only

69% of RCTs (68 of 99 RCTs) reported registration numbers. On the basis of our ﬁndings, efforts to improve trial registration for pediatric RCTs are needed. Registered trials were 68% less likely to have a high risk of bias for sequence generation, compared with nonregistered trials. Increasing the number of pediatric RCTs that are reg-istered would make trial information more accessible to the public and also could be a marker for a reduced risk of bias in trial design.

There were some limitations in this study. The risk assessment was fo-cused on pediatric RCT reports pub-lished during a 1-year period, and the majority of the reports were published in pediatric journals. Therefore, our findings may not be generalizable to RCTs performed with adult popula-tions and pediatric RCTs reported in general medical journals. In addition, the domain-based, risk-assessment tool developed by the Cochrane Collab-oration is not a validated instrument. Even with prespecified definitions, we found a large amount of disagreement among reviewers, in that 53% of stud-ies required adjudication. These re-sults are similar to those reported by Harlting et al4_{and reflect the amount}

of subjective judgment inherent in as-sessment of some of the domains of the Cochrane Collaboration, risk-of-bias tool. For a systematic review, the assessment is accompanied by a spe-ciﬁc reason for scoring a domain as having a high, low, or unclear risk of bias, to ensure transparency. Our re-sults and those of Hartling et al4

sug-gest that using the tool without the accompanying explanations may be problematic. This limitation is high-lighted by our assessment of selective outcome reporting and other biases. Although we found few studies with possible selective outcome reporting bias, this might be because of lack of content expertise or protocols to

com-pare with the published report. Con-versely, by using prespeciﬁed deﬁni-tions with adjudication, we were able to identify factors associated with a high risk of bias. Finally, given the de-sign of this study, we could assess only what was reported and not necessarily what actually occurred during the con-duct of the study.

CONCLUSIONS

Pediatric RCTs are conducted throughout the world, across numer-ous medical categories, to test many different types of interventions. How-ever, there was incomplete adher-ence to journal requirements for trial registration. Overall, we found a large proportion of pediatric trial re-ports with a high risk of bias for sequence generation and alloca-tion concealment. Factors associ-ated with a high risk of bias included industry funding and assessment of behavioral/educational interven-tions, whereas trial registration, private foundation funding, and increased author numbers were as-sociated with a lower risk of bias. Results of pediatric trials with a high risk of bias should be interpreted with caution.

APPENDIX: CODEBOOK FOR ASSESSMENT OF RISK OF BIAS

Allocation Sequence Generation

1. Low risk: random number table, computer-generated, with minimization.

2. High risk: odd or even date of birth, rule based on date of admission, hospital or clinic record number, clinician judgment, participant preference, or intervention avail-ability, or tossing coins, shufﬂing cards or envelopes, throwing dice, or drawing lots.

3. Unclear: randomization is stated but process is not described.

(7)

(pharmacy-, telephone-, or Internet-based); for drug trials, medication containers are numbered and iden-tical in appearance and placebo drugs are made to taste and to look like intervention drugs; envelopes numbered, sealed, and opaque.

2. High risk: any procedure in which participants or researchers could have foreseen allocation; assign-ment envelopes unsealed, see-through, or not numbered; any al-ternation procedure that would allow an individual to predict allocation.

3. Unclear: insufﬁcient evidence that allocation concealment was appro-priate (eg, envelopes were used but it was not described whether they were numbered, sealed, and opaque).

Masking Procedures

1. Low risk: masking procedures for all key participants, including study participants, treatment administra-tors, and outcome assessors, were described and it was unlikely that masking could have been broken; partially masked but outcome as-sessment was masked and un-masked participants were not thought to introduce bias; un-masked but outcome was not thought to be inﬂuenced by masking.

2. High risk: no masking or incomplete masking for an outcome that was likely to be affected by masking; masking procedures could have

3. Unclear: insufﬁcient evidence to state yes or no; study did not ad-dress masking.

Incomplete Outcome Data

1. Low risk: no missing outcome data; reasons for missing data were un-likely to be related to outcomes (such as censored data); missing outcome data were balanced across intervention groups, there were similar reasons for missing data, and reasons were unrelated to outcomes; data were analyzed with intention-to-treat approach ac-cording to randomization group; statistical methods were de-scribed, were used to deal with missing data, and were not thought to introduce bias.

2. High risk: reasons for missing data were likely related to out-comes (ie, disproportionate num-bers of dropouts in the interven-tion group because of adverse effects or exclusion of participants who failed to experience improve-ment); intention-to-treat analysis performed only for participants re-maining in the study.

3. Unclear: insufﬁcient reporting of dropouts and exclusions for clear decision; numbers randomized in each group were not clearly reported.

Selective Outcome Reporting

1. Low risk: all outcomes described are included and reported in analy-sis; for registered trials, all

out-are reported.

2. High risk: not all prespeciﬁed out-comes are reported;ⱖ1 outcome is not reported completely; analytic methods or subsets of data used to report outcomes were not pre-speciﬁed; authors fail to report key outcome expected for study.

3. Unclear: insufﬁcient information for clear decision.

Other Sources of Bias

1. Low risk: study seems to be free of other sources of bias.

2. High risk: for cluster-randomized trials, individuals were recruited to the study after clusters were ran-domized, complete clusters were lost from trial, or statistical meth-ods were not used to account and to correct for clustering; for cross-over trials, no or insufﬁcient wash-out period and treatments were likely to have been carried over from one period to the next, so out-comes might differ depending on the order in which participants received treatments; study was stopped early before calculated sample size was reached, with no explanation given; extreme baseline imbalance between study groups.

3. Unclear: insufﬁcient information for assessment of whether another source of bias exists.

ACKNOWLEDGMENT

Dr Scherer is a member of the CONSORT group and the Cochrane Collaboration.

REFERENCES

1. Higgins JPT, Green S, eds. Cochrane Hand-book for Systematic Reviews of Interventions. Version 5.0.1 [updated September 2008]. Bal-timore, MD: Cochrane Collaboration; 2008. Available at: www.cochrane-handbook.org. Accessed November 10, 2008

2. Moher D, Jadad AR, Nichol G, Penman M, Tugwell P, Walsh S. Assessing the quality of randomized controlled trials: an anno-tated bibliography of scales and check-lists. Control Clin Trials. 1995;16(1): 62–73

3. Ju¨ni P, Witschi A, Bloch R, Egger M. The haz-ards of scoring the quality of clinical trials for meta-analysis.JAMA. 1999;282(11): 1054 –1060

(8)

random-ised controlled trials: cross sectional study.

BMJ.2009;339:b4012

5. Garﬁeld E. The history and meaning of the journal impact factor.JAMA.2006;295(1): 90 –93

6. ISI Web of Knowledge. Journal Citation Reports. Available at: http://admin-apps.isiknowledge. com/JCR/JCR?SID⫽2DaK7OpgGi51pLo8CaG. Accessed June 7, 2010

7. Cohen E, Uleryk E, Jasuja M, Parkin PC. An absence of pediatric randomized con-trolled trials in general medical journals.

J Clin Epidemiol.2007;60(2):118 –123

8. Als-Nielsen B, Chen W, Gluud C, Kjaergard LL. Association of funding and conclusions in randomized drug trials: a reﬂection of

treatment effect or adverse events?JAMA.

2003;290(7):921–928

9. Lexchin J, Bero LA, Djulbegovic B, Clark O. Pharmaceutical industry sponsorship and research outcome and quality: sys-temic review. BMJ. 2003;326(7400): 1167–1176

10. Pildal J, Hro´bjartsson A, Jørgensen KJ, Hilden J, Altman DG, Gøtzsche PC. Impact of allocation concealment on conclusions drawn from meta-analyses of randomized t r i a l s . I n t J E p i d e m i o l . 2 0 0 7 ; 3 6 ( 4 ) : 847– 857

11. Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias: dimensions of methodological quality associated with

es-timates of treatment effects in controlled trials.JAMA.1995;273(5):408 – 412

12. Wood AM, White IR, Thompson SG. Are miss-ing outcome data adequately handled? A re-view of published randomized controlled trials in major medical journals.Clin Trials.

2004;1(4):368 –376

13. DeAngelis CD, Drazen JM, Frizelle FA, et al. Is this clinical trial fully registered? A state-ment from the International Committee of Medical Journal Editors. JAMA. 2005; 293(23):2927–2929

14. US National Institutes of Health. About ClinicalTrials.gov. Available at: www. clinicaltrials.gov/ct2/home. Accessed April 1, 2009

Ratting Out the Enemy in Africa:It’s hard to believe but a Dutch company has

successfully trained African giant pouched rats, which have poor vision but superb senses of smell, to detect landmines. According to an article inThe New York Times(Kristof ND, June 16, 2010), these large (30 inch long) rats are too light to set off mines, but upon exploring a suspected area, can point with their noses to buried mines and literally rat them out. In fact this particular species of rat, once trained over many months, can do as much land surveillance for mines in 20 minutes as a human can do in 2 days. What is even more impressive is that these rats can also use their noses to detect cases of tuberculosis. When sputum samples are given to the rats, they apparently can detect the bacteria through smell faster than a technician can look for them with a microscope. According to the article, a technician in Tanzania can screen 40 samples a day while one giant rat can do the same amount of work in only seven minutes. As to cost, the company notes that a years worth of bananas to feed one rat is only $36 and $100 will provide the materials for breeding these rodents.

Noted by JFL, MD

(9)

DOI: 10.1542/peds.2009-3121 originally published online July 12, 2010;

Services

Updated Information &

http://pediatrics.aappublications.org/content/126/2/298

including high resolution figures, can be found at:

References

http://pediatrics.aappublications.org/content/126/2/298#BIBL

This article cites 11 articles, 2 of which you can access for free at:

Subspecialty Collections

ub

http://www.aappublications.org/cgi/collection/psychosocial_issues_s

Psychosocial Issues

al_issues_sub

http://www.aappublications.org/cgi/collection/development:behavior

Developmental/Behavioral Pediatrics following collection(s):

This article, along with others on similar topics, appears in the

Permissions & Licensing

http://www.aappublications.org/site/misc/Permissions.xhtml

in its entirety can be found online at:

Information about reproducing this article in parts (figures, tables) or

Reprints

http://www.aappublications.org/site/misc/reprints.xhtml

(10)

Assessment of Risk of Bias Among Pediatric Randomized Controlled Trials

Randomized Controlled Trials

abstract

DOI: 10.1542/peds.2009-3121 originally published online July 12, 2010;

DOI: 10.1542/peds.2009-3121 originally published online July 12, 2010;

2010;126;298

Pediatrics

Michael T. Crocetti, Diane D. Amin and Roberta Scherer

Assessment of Risk of Bias Among Pediatric Randomized Controlled Trials

http://pediatrics.aappublications.org/content/126/2/298

located on the World Wide Web at:

The online version of this article, along with updated information and services, is