Introduction
8.1
In many countries manufacturers have a regulatory requirement to monitor the adverse effects of their drugs and may, therefore, be a useful source of additional information on adverse effects. A review of celecoxib concluded that clinical trial reports produced by or for manufacturers were an ‘ideal source of information for systematic reviews and meta-analyses’.462 Much of the data on adverse effects is
contained in industry funded studies and this is likely to escalate as the percentage of studies funded by industry sources is increasing.488
Data from manufacturers is not always readily available12 and may be classed as ‘commercial in confidence’ by the company owning the data. Furthermore, documents produced by manufacturers can be extremely long and difficult to navigate. In addition, studies produced by or for manufacturers may be subject to publication and selective reporting bias,452, 489, 490 as adverse effects may be suppressed or omitted from published studies, particularly when they are not statistically significant,452, 459, 460, 486 or results from only selected stages of the trial are presented.490 Studies that find an increased risk of an adverse effect may never be published.491
There is a large body of literature that has identified an association between
industry funding and better study outcomes.489 These studies have tended to focus on effectiveness outcomes, with the primary aim of comparing the beneficial effects reported and the source of funding for a study. Better study outcomes, however, can be as a result of a more effective intervention or a lower adverse effects profile or a combination of the two. Research has indicated that industry sponsored meta- analyses yielded lower odds ratios for adverse effects than those reported by academic based meta-analyses.21, 492, 493 However, this reporting of lower rates of adverse effects may not appear in primary studies. In palliative care and cancer care it has been suggested that pharmaceutical companies may report adverse effects more comprehensively than non-industry funded studies,494 although the suppression of trial data on suicide with seroxat suggests that industry may withhold adverse effects data.495 This research aims to systematically review the
methodological literature concerning the reporting of adverse effects and any potential association with source of funding (such as industry or non-profit organisations).
Methods
8.2
8.2.1 Inclusion criteria
A methodological evaluation was considered eligible for inclusion in this review if it compared the results or interpretation of reported adverse effects data according to funding source (for example, adverse effects data in pharmaceutical industry research versus data from non-profit organisations, or from one manufacturer versus another).
8.2.2 Data extraction
Information was collected on the selection criteria, interventions and adverse effects, the number, study design and funding sources of studies included in the methodological evaluation, and the outcomes used in assessing differences between studies.
8.2.3 Assessment of methodological quality
The following criteria were used to assess the quality of the existing methodological evaluations;
1. Confounding factors by study design: Did the researchers select comparison
groups (i.e. data from different funding sources) that were equally matched? For instance, did the industry funded studies share similar aims, designs and sample sizes as those that were non-industry funded? If not, were there adjustments for potentially confounding factors that could affect the
association between funding and the nature of the adverse effects data? The following confounding factors were looked for to see if they had been
considered: study design; methodological quality; type of intervention and control intervention; sample size; disease area; type of adverse effects.
2. Missing data or misclassification: How often were the researchers able to
3. Blinding: Were the researchers aware of the funding source when they were
judging the nature of the adverse effects data?
4. Validity and Representativeness: Did the researchers select an adequate
sample of studies (in terms of size, diversity of topics and range of adverse effects) that were reasonably reflective of current literature?
Results
8.3
8.3.1 Included studies
Six methodological evaluations met the inclusion criteria (Appendix B: Table 15.11).99, 496-501 All six were concerned with drug interventions, with five of the six evaluations limited to the adverse effects of a single agent or single class of drugs.99, 498-501 Two methodological evaluations were limited to specific adverse effects,99, 498 whereas the other methodological evaluations included any adverse effects. Only one methodological evaluation assessed funding source and reporting of safety data across a wide range of diseases and drugs.496, 497 The number of studies included in the methodological evaluations ranged between 10 and 504 with only two methodological evaluations including more than 100 studies.496, 497, 499
Half of the methodological evaluations focused on adverse effects data within clinical trials496, 497, 499, 500 and two included observational data.99, 498 One had a mixture of reports of original research, reviews and letters.501 Most methodological evaluations compared manufacturer funding with non-manufacturer funding, though one evaluation looked for differences in adverse effects data in research funded by competing manufacturers.500
8.3.2 Excluded studies
There were two methodological evaluations excluded from this review (Appendix B: Table 15.12).247, 248, 502 One502 contained duplicate data from an included
methodological evaluation,498 whereas another was excluded as the categories of funding source were unclear, but were unlikely to include industry funded studies.247,
248
8.3.3 Summary of methodological quality
Four of the methodological evaluations used some form of adjustment for potentially confounding factors, although the comprehensiveness of those factors varied (Appendix B: Table 15.11).99, 496-499 A major constraint in assessing an association between source of funding and the reporting of adverse effects was the lack of information on funding source. Only two methodological evaluations described the number of studies not reporting any funding source, both these evaluations included only trial data and reported that 17.3% and 28.6% of studies did not disclose any funding source.496, 497, 499 Blinding was reported in only two evaluations, one which tested the effect of blinding on a subsample of included studies and found that blinding did not impact on the results.496, 497 Overall, the assessment of quality and validity showed that the Als-Nielsen et al 2003 evaluation, which included both studies not reporting funding source and considered blinding, was probably the most robust (Appendix B: Table 15.11).496, 497
Definitions of manufacturer associated funding varied, as did the methods and outcome measures used to assess the association between funding and adverse effects reporting, making it difficult to pool the results of the methodological evaluations identified.
8.3.4 Selective reporting
Als-Nielsen et al 2003 looked at a diverse range of RCTs and noted that trials funded by for-profit organizations were more likely to report adverse events (128/146, 88%) than trials funded by non-profit organisations (32/67, 48%).496, 497
8.3.5 Magnitude of risk of harm
It may be hypothesized that the risk of harm from the sponsor's product might be downplayed in industry funded studies. Three of the four methodological evaluations which measured the magnitude of the risk of adverse effects support this
hypothesis.99, 498, 499
A subgroup evaluation from Kemmeren et al 2001's meta-analysis showed that the pooled data from industry funded studies yielded a weaker association between third generation oral contraceptives and venous thrombosis.498 Similarly, Juni et al
2004's meta-analysis of cardiovascular events and rofecoxib showed that studies funded by Merck were associated with greater cardioprotective effects of naproxen (a comparator for rofecoxib), implying a lesser risk of harm from Merck's product (rofecoxib).99 However, the weakness of this evidence is that they were post-hoc subgroup analyses, involving only a small number of studies (11 studies in Juni et al 200499 and 10 studies in Kemmeren et al 2001498) and subject to confounding, as no adjustments were made for any study design or patient characteristics.
Nieto et al 2007's evaluation of inhaled corticosteroids reported that statistically significant results for adverse effects were found less frequently in pharmaceutical industry funded studies, whereas non-industry funded studies were more likely to report significant harm.499
Conversely Als-Nielsen et al 2003 noted that a higher frequency of adverse effects tended to be found in the experimental arm of industry funded trials than trials funded by non-profit organisations.496, 497
8.3.6 Confounding factors
The differences between the results or conclusions of studies funded by industry and non-profit organisations could reflect other factors such as the chosen
interventions and disease area, different study designs, methodological quality, and study size.
Nieto et al 2007 found that studies funded by industry differed from those not funded by industry and were more likely RCTs; multicentre; to use a parallel design in prospective comparative studies; to state that their primary objective was studying efficacy, to use lower dosages of the medication; and to have a larger sample size and shorter follow-up times. The studies also differed in the methods used to investigate adverse effects. Industry funded studies were more likely to limit the assessment to only non-specific clinical data (such as medical history) and/or laboratory data (such as blood count) or cortisol metabolism (such as plasma or urinary cortisol level), and less likely to assess other specific adverse effects such as growth (height) or bone metabolism (densitometry). An adjusted prevalence ratio as reported by Nieto et al 2007499 0.94 (95% CI 0.77 to 1.15) suggested that the difference associated with funding might be mediated by other variables in the analysis.
8.3.7 Interpretation of adverse effects data
The included studies revealed some interesting potential associations between funding source and the subjective interpretation or conclusions regarding adverse effects data. For example, Nieto et al 2007 found that authors of pharmaceutical company funded studies were more likely than authors of non-pharmaceutical studies to conclude that a drug was safe, even among studies that found a statistically significant increase in adverse effects.499 Similarly, Rochon et al 1994 found that a manufacturer associated drug was often judged to be less toxic, even though this interpretation was not always supported by a test of statistical
significance.500 Finally, Als-Nielsen et al 2003 noted an association between favourable recommendations for a product and the manufacturer's sponsorship, irrespective of the actual magnitude of treatment benefit or safety results seen in the trial.496, 497
The study by Juni et al 200499 also indicated that conclusions might differ with studies funded by industry indicating larger protective effects of an adverse effect in the drug comparator. This enabled authors to conclude that the difference in
adverse effects between the experimental group and the comparator was a result of a protective effect in the comparator group rather than an increased risk of adverse effects in the experimental group.
There is possible potential for error and bias when trying to judge whether the data interpretation and conclusions of a study are excessively favourable or not. Stelfox et al 1998501 and Als-Nielsen et al 2003496, 497 attempted some degree of blinding of the reviewers but none of the remaining four methodological evaluations used any blinding.
8.3.8 Competing manufacturers
Just as it may be hypothesized that studies by manufacturers with a financial interest in the intervention are more likely to have favourable conclusions, it may also be hypothesized that competing manufacturers are more likely to emphasize concern over safety of a rival intervention.503 One methodological evaluation looked at this possible association,501 finding that the reverse might be true and that
authors who are neutral or supportive of the safety of an invention were more likely to have a financial interest with competing manufacturers. It would appear that
neutral or supportive authors are more likely to have a financial relationship with any manufacturer of the intervention or competing product. This study had some
limitations, including a lack of a temporal analysis (it is not known whether support of the intervention preceded funding from the manufacturer), loose definitions of an association with industry funding, and failure in checking the appropriateness of the conclusions of the authors against the actual adverse effects data of the studies.
Discussion
8.4
This systematic review has identified somewhat mixed evidence surrounding the postulated link between industry funding and more favourable reporting of adverse effects data. Bearing in mind the limitations of this review (see below), it is only possible to draw tentative conclusions. Firstly, there is no strong evidence that funding source leads to selective reporting of adverse effects outcomes that favoured the sponsor’s product. Indeed, Als-Nielsen et al 2003, probably the methodological evaluation with the strongest quality criterion, found that the opposite was true, with industry funded studies providing more complete reporting and higher rate of adverse effects for the experimental arm.496, 497 Unlike non-profit organization funded studies, pharmaceutical companies hoping to submit a
licensing application could be more focused on providing an accurate depiction of adverse events, as the data might be subjected to rigorous regulatory scrutiny. Indeed, it is possible that the information submitted to the regulatory authorities is less positive than that seen in the published articles.504
There is also no strong evidence that industry funded studies present a lower
magnitude of risk of harm from the sponsor’s product, although pharmaceutical trials have been accused of using design modifications to ascertain lower adverse effects. Such methods might potentially include: using lower doses of the intervention and higher doses for the controls; monitoring for adverse effects using open-ended or non-specific questions; the use of eligibility criteria and run-in periods to exclude patients prone to adverse effects; a focus on a single adverse effect or a narrow range of related adverse effects to obscure harms of the drug; repeated analysis of data until any extra risk of adverse effects disappears; and the choice of
inappropriate comparators or interventions known to have few adverse effects.27, 124,
This systematic review indicates that funding source may impact on the nature of the authors’ interpretation and conclusions regarding the safety profile. However, the interpretation of adverse effects data relies not only on statistical significance, but also on subjective judgements on clinical relevance, preventability, and absolute risk.
Limitations
8.5
There were a number of methodological issues of concern with the evaluations included in this review. First of all, all the methodological evaluations were ‘observational’ in nature. While some of them had pre-defined objectives496, 497, 501
others were post hoc or subgroup analyses. Confounding was a major problem in most of the methodological evaluations, where the baseline features (e.g. study design, patient population, primary objectives) of the industry funded studies might have differed from those of the non-industry funded studies. This is particularly apparent in Nieto et al 2007 where the observed differences became non-significant after adjustment for confounding factors.499
There is also the possibility of reporting or publication bias with respect to methodological evaluations.487 Journal editors may look more favourably upon articles that show biased reporting of adverse effects in industry funded studies, or researchers who do not find any industry-related bias might choose not to submit their articles for publication. Equally, researchers finding evidence of industry funded bias may avoid publicizing the results so as not to jeopardize any industry funding ties that they might have.
The generalizability of the data is also contentious. It would be unfair to draw broad conclusions about bias in all industry funded studies when the data are limited to a few studies or to only a specific class of drugs. Moreover, reporting
recommendations have changed over time, with tightening of regulatory
requirements, and the publication of the Consolidated Standards of Reporting Trials (CONSORT) statement on harms.32 Existing methodological evaluations have not taken into account temporal changes, or the availability of complete adverse effects data from unpublished company trial reports available from trials registries such as the GlaxoSmithKline Clinical Trials Registry (http://ctr.gsk.co.uk/welcome.asp) and ClinicalStudyResults.org (www.clinicalstudyresults.org).
Failure to accurately classify funding source is the most prominent weakness in the methodological evaluations. In Nieto et al 2007’s evaluation, 87 studies (17.3%) were categorised as non-industry funded, despite there being no information on funding source.499 Misclassification of such a large number of studies could have a major influence on the direction and magnitude of any link between funding and adverse effects data. The largest methodological problem though, lies with the difficulty in verifying authorship and the reliability of financial declarations in published papers. Two recent papers have highlighted problems with ghost authorship and inaccurate financial disclosures (e.g. not disclosing a financial interest in one article, but declaring industry funding in another publication).507, 508 If studies categorized under non-industry funding were misclassified and were actually industry funded, this would dilute the strength of any argument that non-industry funded studies provided less-biased reports of adverse effects.
A considerable amount of subjectivity was involved in trying to determine whether the interpretation and conclusions of a study were biased towards the sponsor’s product. Reviewers who were critical of the pharmaceutical industry might have taken a harsher view in finding fault with industry funded studies, while those supportive of the industry might have been less likely to judge the presence of bias. Unfortunately, blinding and inter-rater reliability were key parameters that were seldom specified by the methodological researchers.
Conclusions
8.6
Industry funding may not be a major threat to bias in the reporting of the raw adverse effects data, though bias might be introduced in the interpretation and conclusions of the industry funded studies.
The limitations of the included methodological evaluations in this review suggests that further research is required in this area in order to draw any firm conclusions on the impact of including or excluding industry sponsored studies in systematic
reviews of adverse effects. In the meantime efforts should be made where possible to include all relevant studies and be explicit about the sources of funding of
Summary
8.7
This review of six methodological studies in the literature comparing reporting of adverse effects and funding source indicates that there is no strong evidence that funding source leads to selective reporting of adverse effects. However, funding source may impact on the author’s interpretation and conclusions.
Additional information to that in the published literature can be obtained from industry funded data.