While still in its infancy, there is much debate about the intended design and scope of a national active medical product safety surveillance system204, 205. An active surveillance system will involve a systematic process for analyzing multiple observational healthcare data sources, including administrative claims and electronic health records, to better understand the effects of medical products by estimating temporal relationships between exposure and outcomes. The active surveillance system can be used to 1) characterize known side effects, 2) monitor preventable adverse events, and 3) explore remaining uncertainties. The goal of the active surveillance system is to contribute supplemental information to other existing sources of safety information (including pre-clinical data, clinical trials, and spontaneous adverse event reporting) to support decision-making about appropriate use of medical products for regulatory agencies, providers, and patients.
Influents of risk: • Demographics: Age Gender Race Location • Comorbidities • Concomitant medications • Health service utilization • Utilization practice: dose,
duration, frequency • Socioeconomic status • Personal health: smoking
status, BMI • Provider characteristics • Environmental risks Measures of risk: • Clinical trials • Spontaneous adverse event reporting • Epidemiologic studies • Registries • Observational databases: • Administrative claims • Electronic medical records Sources of risk:
Known side effects
Avoidable Preventable Adverse Events Medication and Device Error Product Defects Injury or Death Remaining Uncertainties: Unexpected side effects
Unstudied uses Unstudied populations Unavoidable
Figure 4: Conceptual framework for active surveillance
Figure 4 provides a conceptual framework for active surveillance. There are various sources of risk of medical products that can result in injury or death, including known side effects, medication and device error, product defects, and other remaining uncertainties. These risks are influenced by many factors, including patient characteristics (such as demographics, comorbidities, concomitant medications, and health service utilization), health system factors (such as utilization practice and provider behavior), and other environmental sources.
Discovery of how treatment effects vary by baseline risk is one of the important contributions of post-marketing surveillance of drugs206. The current measures of risk include clinical trials, spontaneous adverse event reporting systems, epidemiologic studies, and registries. Active surveillance offers the opportunity for the systematic use of observational healthcare databases, such as administrative claims and electronic health records, to improve our
measures of the sources of risks. Analyses against these data can account for the measurable influents of risk to provide robust, supplemental information that can be used to both identify
and evaluate potential drug safety issues. While evaluation studies have been common practice for decades, use of these data in a formal exploratory analytic framework is new and requires further research to determine its relative contribution to such a system.
When considering drug safety in a causal inference framework, one can consider Hill’s considerations of 1) strength, 2) consistency, 3) specificity, 4) temporality, 5) biologic gradient, 6) plausibility, 7) coherence, 8) experimental evidence, and 9) analogy54. The strength of association should be considered because stronger associations may be more compelling, but weak associations do not rule out causal connections207. Consistency refers to the repeated observation of an association in different populations under different
circumstances. Specificity relates to the number of causes that lead to a specific effect, and the number of effects produced from a given cause. Temporality refers to the necessity that the cause precedes the effect. Biologic gradient addresses the degree to which there is a dose-response relationship, where the amount of response increased with increased exposure. Plausibility reflects the scientific rationale for the existence of an association, typically in drug safety, related to the mechanism of action and the biologic pathways that lead to the effect. Coherence is the degree to which the interpretation of the association does not conflict with the current understanding of the natural history and biology of the disease. Experimental evidence for drug safety analyses typically refers to evidence that comes from human randomized clinical trials, but can also include randomized pre-clinical experiments in animal models.
An active drug safety surveillance system can apply Hill’s considerations as part of its process for generating hypotheses. Specifically, analyses conducted across a network of observational databases can be used to identify potential drug safety issues based on strength,
consistency, specificity, and temporality. Specifically, methods produce estimates of the strength of temporal associations between exposure and subsequent outcomes. Applying the methods to multiple sources provides an assessment of consistency, as formal tests for heterogeneity can be used to measure differences between source populations. Evaluating multiple outcomes for each drug and multiple exposures for each outcome can provide insights into the specificity of any specific drug-outcome association. However, these exploratory analysis results will not be sufficient to address issues of biologic plausibility, and the use of observational data does not meet the same standards of evidence that come from a randomized experimental design. Methods for studying dose effects requires further research, as the degree to which dose and amount of exposure can be accurately measured and used within a hypothesis generating framework is undetermined.
While hypothesis generating analyses are inherently exploratory in nature, basic principles of formal evaluation can be applied to raise the collective confidence in the reliability of the process. Research questions and statistical analyses should be specified in advance, with all methodological considerations addressed during study planning rather than after study completion. This includes decisions around definitions of exposure and outcome, inclusion/exclusion criteria imposed on the sample, and strategies for statistical
adjustment150. Analysis processes should be fully transparent and reproducible, and should minimize subjective assessment to improve the generalizability of the approach. Many of these principles are well-defined in guidelines for conducting full evaluation studies13, 208-210 but have not yet been adopted for exploratory analyses. With these principles in place, hypothesis generation can play an important role in an active surveillance system’s contribution to causal inference of drug safety issues. These exploratory analyses can
identify and prioritize areas that warrant further examination. Evaluation studies may be used to refine estimates of the strength of the association, but attention can particularly focus on biologic plausibility and coherence to put the preliminary results in proper clinical context with other evidence, including clinical trials, pre-clinical data, spontaneous adverse events, and other epidemiologic studies.
CHAPTER THREE: METHODS
3.1 Overview
This study is a methodological experiment to evaluate the performance of a novel analysis technique for active drug safety surveillance. The analytical approach, called COMParator-Adjusted Safety Surveillance (COMPASS), is described (section 3.2). The evaluation of COMPASS was conducted across five observational data sources (described in section 3.3) by exploring the method’s ability to identify known safety issues associated with ACE inhibitors. The experimental design, including the selection of the sample test cases of true adverse reactions and negative controls for the drug class and individual ingredients is highlighted in section 3.4. The performance measures used to assess COMPASS
performance are discussed in section 3.5. The remainder of the chapter provides specific analyses conducted to support the following aims:
Aim 1: Characterize the performance of COMPASS in identifying known safety issues
associated with ACE inhibitor exposure within an administrative claims database
Aim 2: Evaluate consistency of COMPASS estimates across a network of disparate
databases