Deviation from the optimal study approach: no one-size fits-all

In the case of medical devices, both therapeutic and non-therapeutic, such pragmatic randomised trials present a greater methodological challenge than in the case of med- ical drugs. For example, in clinical studies of devices there is often an intricate inter- play between the technical complexities of the device, its user (often specific skills are needed), and ‘user/interpretation learning-curve effects’, compromising intervention adherence and sometimes even the possibility of randomised allocation (see also Sec- tion 4.7.1. and Appendix IV, part A). Indeed, the benefits and risks of a device often depend not only on the device itself but are a result of that complex interplay.

Non-randomised or observational studies of therapeutic devices are even more challenging, however. Selecting the proper study subjects, choosing a comparison group, adjusting for other influential factors, and addressing learning curve issues all require the conscious selection of study design, conduct and analysis plan. For example, when existing clinical databases or registries are used to obtain evidence of the potential benefits and risks of device use in practice, relevant information about other influences is often not – or only partially – available. This compromises any valid inferences about its true benefits and risks (see Section 4.7.1. and Appendix IV, part B). Moreover, for diagnostic, screening, prognostic or monitoring devices, i.e. devices without a direct therapeutic effect on patients’ outcomes, a single randomised com- parative study may even not be indicated or may be very cumbersome.56 _{Such devices} may profit from a network or linked-evidence approach instead (see Sections 4.7.2 and 4.7.3, and Appendix IV, parts C+D). Clearly, there is no one-size-fits-all approach to evaluating the benefits of medical devices.

4.5.1 Reminder of optimal clinical study approach and overarching prin- ciple

The challenge is thus to identify the type of evidence and study approach that is required, given the specifics of the device (e.g. therapeutic or information-generat- ing, invasive or non-invasive, requiring user interference/interpretation or not), its intended context, intended indication, and targeted individuals. However, we strongly recommend maintaining amental picture of the optimal randomised comparative effectiveness study when designing or choosing any alternative research approaches, to 56  Bossuyt P.P.M., Lijmer J.G., Mol B.W., Randomised comparisons of medical tests: sometimes invalid, not always efficient. Lancet 2000;356:1844-1847; Lord S.J., Irwig L., Simes R.J., When is measuring sensitivity and specificity sufficient to evaluate a diagnostic test, and when do we need randomized trials? Ann Intern Med 2006;144:850-5; Biesheuvel C.J., Grobbee D.E., Moons K.G., Distraction from randomisation in diagnostic research. Ann Epidemiol 2006;16:540-4; Li- jmer J.G., Bossuyt P.M., various randomized designs can be used to evaluate medical tests. J Clin Epidemiol 2009;62:364-73.

allow for valid inferences about the benefits/added benefits of device use for health or health care. Technical capa- bilities of the device Intermediate effects induced by the device Intended clinical benefits for tar- geted individuals or care providers by the device Technical performance/ Analytical performance Clinical performance/ Clinical validity Clinical benefits/ Clinical utility

Role of comparison group

Safety

Figure 4.1 Pathway through which devices may eventually lead to benefits/added be-

nefits for health or health care, and how each step is linked to a specific approach. The pathway ranges from the technical or analytical performance of a device through its intermediate changes or effects on health or health care outcomes to the final benefits for patients, care providers or society at large.

Indirect versus direct evidence

To obtain direct evidence of the health or health care benefits of a device, alternative study approaches may include simpler or more tailored randomised studies and non-randomised comparative studies. But there are also approaches that gener- ate indirect evidence of the device’s intended health or health care benefits. These include single study approaches followed by some kind of network or linked-evidence approach. For some devices (e.g. devices that are simply a minor modification of an existing device), a non-comparative or perhaps even only technical performance study can be justified using a network or linked-evidence approach. For other devices, direct evidence obtained in randomised comparative effectiveness studies or at least

a non-randomised comparative study may be required. This will be discussed and illustrated in Section 4.7.

A clear description of the pathway (see also Section 4.6) through which a device may ultimately affect health or health care outcomes, and which type, provides rele- vant clues for designing the following proper study in the life cycle of a device (see Fig- ure 4.1). Studies focusing on earlier steps in this pathway generate indirect evidence of the benefits of a device for health or health care. This indirectness may relate to various sources, including:

• outcomes, e.g. short-term/intermediate rather than long-term outcomes, diagnostic accuracy outcomes rather than patient-relevant outcomes, clinical performance rather than clinical utility outcomes, care provider rather than patient outcomes;

• use of the device, e.g. by experts in specialised care rather than less experienced users in regular care;

• type of device, e.g. a new diagnostic device may lead to better or earlier detection of a disorder, whereas its benefits for improved therapeutic decision-making, let alone for improved patient health, has not been proven as yet;

• individuals studied, e.g. investigating a device in a sample of healthy individuals rather than in a sample taken from the intended context and patient population.

Bias versus precision

In addition to the directness or indirectness of a study on the health care benefits of a device, there are two other key dimensions of quality to consider in clinical research on devices (Figure 4.2). These are:

• risk of bias (invalidity);

• precision of study estimates.

Every type of clinical study is vulnerable to specific biases, which we discuss in detail in Section 4.7 and in the corresponding Appendix IV. Shortcomings in study design or execution can produce incorrect or invalid results and thus incorrect infer- ences about clinical performance, validity, benefits or utility. Minimising such bias is the aim of every clinical study. Generally speaking, the more one deviates from the optimal study approach, the more challenging it is to draw valid inferences about the benefits/added benefits of a device for heath or health care.Awareness of this deviation is a major step forward, however. Section 4.7 shows how to make the best possible inferences about a device’s benefits or added benefits for health care when deviating from the optimal study approach.

Imprecision arises when study results are based on a small study sample. Impre- cision leads to uncertainty about the study results. Proper statistical analysis can describe the uncertainty in study results by using confidence intervals.

Figure 4.2 integrates the overarching principle and pathway through which devices may lead to benefits or added benefits for health or health care, and the three main dimensions of quality for evidence obtained from device evaluations (indirect- ness of evidence; risk of bias; precision of estimates).

Technical features of medical device Framing the pathways of benefits: Intermediate outcomes

(Added) benefits for health or healthcare Direct evidence: • Right device • Right population/setting • Right comparison • Right outcome(s) Indirect evidence due to:

• Differences in device or device use • Differences in targeted individuals/setting • Differences in comparison group

• Differences in outcome/endpoints Dimensions of quality of evidence: Direct evidence Linked evidence Risk of bias Precision of estimates

Figure 4.2 Relationship between the pathway through which devices may lead to ben-

efits/added benefits for health or health care and the three dimensions of quality for evidence (indirectness of evidence, risk of bias, precision of estimates).

The framework outlined in Figure 4.2 is further discussed in the remaining sections. Section 4.6 provides guidance by explaining how and why we must describe the path- way through which a device leads to benefits (and risks). Section 4.7 then describes various research approaches for generating evidence of the benefits of device use for the targeted individuals (patients or professionals). Here we focus on two main issues: the directness or indirectness of the evidence and the risks of bias (internal invalidity). Precision of study results (estimates) is a more straightforward concept for which the discussion in Appendix v suffices.

Appendix IV provides a detailed overview of the many different study designs that