-Identify and prioritize tasks according to relative risk to the user beyond estimates derived from analytical techniques;
-Guide development of use scenarios to be employed during subsequent design validation testing;
-Identify use-related hazardous situations leading to the development of risk mitigation strategies;
-Evaluate trade-off considerations and effectiveness of design enhancements, training and instructions for use;
-Guide modification of the device design to optimize the user interface with respect to device safety and effectiveness; and
-Clarify the dynamics of device-interaction associated with known or suspected use error scenarios.
2.3.2. Summative (validation) testing
Summative testing is performed in the latter stages of the design process as part of device verification and validation. Summative testing requires formal acceptance criteria according to the design inputs established at the beginning of the design phase (see Figure 4 in section 2.2) (Weinger et al. 2010). Validation testing is used to establish safe and effective performance of a medical device using actual users in realistic conditions. The use of iterative formative usability tests, performed throughout the early and mid-stages of the design process should ensure that there are little or no usability issues discovered during summative testing. Simulated use testing is identified by both the IEC and FDA standards as an appropriate tool for formative and validation testing of medical devices:
FDA Draft Guidance on Applying Human Factors to Medical Devices
The FDA guidance document, as controlled under 21 CFR 820.30 (section 2.2.1) lays out the requirements for validating medical devices, and makes reference to simulated use in: Section 10 Human Factors Validation Testing
35
The human factors validation test demonstrates that the intended users of a medical device can safely and effectively perform critical tasks for the intended uses in the expected use environments. It is particularly important during validation testing to use a production version of the device, representative device users, actual use or simulated use in an environment of appropriate realism, and to address all aspects of intended use.
Simulated use is discussed as a separate section in the standard: Section 10.1 Simulated Use Validation Testing
The conditions under which simulated use testing is conducted should be sufficiently realistic to enable the results of the testing to be generalized to anticipate actual use. The need for realism is therefore driven by the analysis of users, use environments, the device user interface and intended uses. To the extent that environmental factors are found to affect user performance, they should be incorporated into the simulated use environment (e.g., dim lighting, multiple alarm conditions, distractions, multi-tasking and workload).
Application of Usability Engineering to Medical Devices (IEC 62366)
IEC 62366 advocates the use of simulation to explore worst case scenarios, complex failures and environmental interactions while validating medical devices specifically section: D.5.13 Simulated clinical environments and field- testing
‘Simulated clinical environments permit evaluation in a controlled manner in a setting containing some or all of the essential attributes of the actual clinical environment for which the MEDICAL DEVICE is being designed. Simulations facilitate creation of worst-case USE SCENARIOS and complex failures. A high- RISK MEDICAL DEVICE or one involving tasks that are more complex can be tested in high-fidelity simulators, such as a full-scale, simulated operating room with functional manikin. High-fidelity simulation allows the test team to evaluate dynamic interactions among multiple MEDICAL DEVICES, personnel, and task constraints’.
2.4. Simulation in usability testing of medical devices
Simulation and simulated use testing are specified in US and European regulations as in appropriate human factors test method during medical device design. Simulation has long been adopted as an effective method of teaching
36
techniques and evaluation in many fields. Examples of professions that embrace simulation-based training include aerospace (flight simulators), the military (realistic war games), and nuclear engineering (systems simulators) (Halamek et al. 2000). Simulators are designed to recreate some aspect of the environment and / or the task at hand. In healthcare, this can range from silicone vein blocks for the practicing of venous puncture, through increasing levels of complexity and realism to the simulation of entire operating theatres (Maran and Glavin 2003).
The term ‘simulator’ when used in the context of healthcare usually applies to a device that’s simulates a patient, or a part of a patients anatomy, and interacts with the actions of the user in an accurate manner without invasive procedures required (Gaba 2004). The development of human patient vascular simulators began in the late 1960s, accelerating in the late 1980s and early 1990s. Simulators are used to teach basic skills, such as respiratory physiology and cardiovascular haemodynamics, and advanced clinical skills, e.g. management of difficult airways. Simulation offers distinct educational advantages, especially for learning how to recognise and to treat rare, complex, unseen clinical problems. The costs of simulator-based educational programmes include facility, equipment and personnel (Good 2003).
Central to the idea is usability testing for medical devices is the need to approximate real life conditions as closely as possible (as set out in both the FDA and IEC standards). For the treatment of aortic aneurysms and aortic stenosis, it is essential to evaluate both the prostheses and delivery system designs in a simulated environment before in-vivo testing (Sulaiman et al. 2008). In vivo animal models are used extensively as a means of evaluating new endovascular devices. These tests are, however, expensive procedures that usually only facilitate the use of one device per animal before it is euthanised.
37
Animal models rarely mimic the disease state that is being targeted. For example with aortic stenosis, prosthetic valves are implanted into aortic valves where no calcifications may be present, and with aortic aneurysms stent grafts may be deployed in the aorta but without the presence of an actual aneurysm. In both cases, evaluation of devices in a simulated anatomical model would provide usability testing that more closely resembles real world environments.
2.4.1. Suspension of disbelief
The key to effective simulation based evaluation or training is achieving a ‘suspension of disbelief’ on the part of the operator (Halamek et al. 2000). That is, there must be a sufficient level of fidelity for the operator to think and feel like they are functioning within a real environment. Simulators are controlled environments in which varying clinical scenarios can be experienced on demand. These scenarios can be scaled to fit the level of the operator, from engineers evaluating a prototype device to a consultant training on a new interventional technique. Some of the key advantages of simulators are shown in Table 4. Of critical importance to any simulator is the consideration of ‘fidelity’. Farmer et al. (1999) stated that ‘fidelity’ is the extent to which the appearance and behaviour of the simulator/simulation match that of the simulated system, however confusion abounds over the ambiguous use of fidelity.
TABLE 4:ADVANTAGES OF SIMULATORS (MARAN AND GLAVIN 2003)