The study design combining disguised observation and interviews directly afterwards enabled us to gain an insight into different types of alert handling, including SB behavior, that cannot be evaluated with the uncombined methods. The interviews revealed useful and very relevant safety information that would have remained hidden by analysis of override reasons from dropdown boxes. Less familiar alerts could be studied in a relatively small time frame, whereas disguised observation is time consuming and only reveals information on the emerging (mainly known) alerts. The closing questions on study validity revealed that the study resembled the normal work environment to a great extent.
This study was performed in one hospital, with one CPOE and only 18 physicians who were willing to participate, resulting in selection bias. However the great variety in handling and reasoning suggests that inclusion of other respondents would probably not result in different findings. It appeared to be very difficult to recruit residents, especially the younger ones and those from surgery, because the first years of residency take mainly place outside the academic hospital, and residents in surgery were not very interested in drug prescribing. Therefore, the effect of level of training on performance could not be studied. A drawback of the study design was that it was impossible to check whether the proposed monitoring of serum levels or
patient parameters would be performed in reality. Nine residents (50%) said they would check the QT interval on the ECG, but in general practice ECGs are rarely made when QT alerts are overridden [12]. The percentage of correct monitoring could therefore be lower than stated by the respondents. Furthermore, surgical residents said they would never initiate beta-blockers or ACE inhibitors themselves, but they were asked to prescribe them during the study. Incorrect handling, rules or reasoning for newly prescribed drugs could therefore be less than during the study.
It can be questioned whether classification of behavior according to the SRK model was always completely correct. During observation, KB behavior was easily detected, because the decision took a relatively long time, scenarios were muttered, or information sources were consulted. Based on observation only, the choice between SB and RB behavior appeared to be very difficult however. We used the interviews to assign categorization of cognitive levels, which might have resulted in incorrect post hoc categorizations. The actions for which clear reasons were given in the interviews were categorized as RB, although these could have been performed on a SB level and reasons constructed afterwards. On the other hand, sometimes people cannot remember why they acted in a certain way, even if the time interval between the action and the interview is short. In our study this could have resulted in SB categoriza- tion in the case of actions performed at a RB level. Furthermore, a person’s task performance may involve more than one cognitive level at once and the cognitive control varies along a continuum, making classification difficult sometimes [6]. In case of doubt, we followed Hobbs in consistently choosing the higher performance level [6].
It can further be questioned whether the SRK model is adequate for obtaining an insight into how cognitive processes play a role in (erroneous) alert handling. The SRK model is a basic model in cognitive psychology describing information processing and is widely used to identify error types, which may range from strong habits, used unconsciously, to cognitive overload at the conscious level. It provides a common terminology for human factors studies and is one of the few tools that can be used to describe the interaction between a person and a task in terms of the cognitive demands of the task [6]. Although cognitive load theory also involves informa- tion processing, this theory mainly focuses on learning and instruction in complex cognitive domains, which was not the focus of this study [23].
The main goal of this study was not to correctly categorize all performance levels, but to gain an insight into alert handling and corresponding errors impairing patient safety as a starting point for future studies on improvement of drug safety alerting. We therefore developed a study simulating the normal work environment, instead of designing a laboratory study perfectly able to categorize performance levels but not resembling daily life.
Chapt
er 2.3
68
COnCluSIOnS
Drug safety alerts were mainly handled rule based, but incorrect rules or reasoning were often used to justify actions. Residents in surgery justified their actions incorrectly twice as often as residents in internal medicine. Main causes of errors were rules that were incorrect or not applicable, such as monitoring of incorrect serum levels or patient parameters, among others. Insufficient training and low specificity played a role in erroneous alert handling. Furthermore, a second alert in one pop-up screen was often overlooked. Roughly a quarter of residents showed signs of alert fatigue.
Future research should include usability studies to investigate how alerts should be pre- sented to be safe and acceptable to clinicians (several alerts in one pop-up, clear and concise alert texts, nonintrusive alerts).
Acknowledgements
The authors thank all respondents for their participation. The authors thank Shantie Anant for performing the tests.
referenCeS
1. Van der Sijs H, Aarts J, Vulto A, Berg M. Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc 2006;13:138-47.
2. Reason J. Human error. Cambridge, Cambridge University Press1st Ed, 1990.
3. Moray N. Error reduction as a systems problem. In Bogner MS. Human error in medicine. New Jersey,
Lawrence Erlbaum Associates, 1st Ed, 1994.
4. Wickens CD, Hollands JG. Engineering psychology and human performance. New Jersey, Prentice
Hall, 3rd Ed, 2000.
5. Rasmussen J. Skills, rules and knowledge, signals, signs, and symbols, and other distinctions in human performance models. IEEE Trans Syst Man Cybern 1983;13:257-66.
6. Hobbs A, Williamson A. Skills, rules and knowledge in aircraft maintenance: errors in context. Ergo- nomics 2002;45:290-308.
7. Van der Sijs H, Aarts J, Van Gelder T, Berg M, Vulto A. Turning off frequently overridden drug alerts: limited opportunities for doing it safely. J Am Med Inform Assoc 2008;15:439-448.
8. Kalmeijer MD, Holtzer W, van Dongen R, Guchelaar H-J. Implementation of a computerized physician order entry system at the Academic Medical Centre in Amsterdam. Pharm World Sci 2003;25:88-93. 9. Van der Sijs H, Mulder A, Van Gelder T, Aarts, J, Berg M, Vulto A. Drug safety alert generation and
overriding in a large Dutch university medical centre. Pharmacoepidemiol Drug Saf 2009 (in press). 10. Weaver JL, Bradley KA, Hancock PA, Szalma JL, Helmick JA. Skills, rules, and knowledge: an experimen-
tal test of performance decrements as a function of stressor exposure. Int Symp Aviation Psych 2003. 11. Van den Tweel AMA, van der Sijs IH, van Gelder T, Knoester PD, Vulto AG. Computerized medication
alert signals: does the MD no longer need the PharmD? Eur J Hosp Pharm 2006;12:30-2.
12. Van der Sijs H, Kowlesar R, Klootwijk APJ, Nelwan SP, Vulto AG, Van Gelder T. Clinical relevant QTc- prolongation due to overridden drug-drug interaction alerts: a retrospective cohort study. Br J Clin Pharmacol 2009;67:347-54.
13. Grizzle AJ, Mahmood MH, Ko Y, Murphy JE, Armstrong EP, Skrepenek GH, Jones WN, Schepers GP, Nichol P, Houranieh A, Dare DC, Hoey CT, Malone DC. Reasons provided by prescribers when overrid- ing drug-drug interaction alerts. Am J Manag Care 2007;13:573-80.
14. Hsieh TC, Kuperman GJ, Jaggi T, Hojnowski-Diaz P, Fiskio J, Williams DH, Bates DW, Gandhi TK. Char- acteristics and consequences of drug-allergy alert overrides in a computerized physician order entry system. J Am Med Inform Assoc 2004;11:482-91.
15. Shah NR, Seger AC, Seger DL, Fiskio JM, Kuperman GJ, Blumenfeld B, Recklet EG, Bates DW, Gandhi TK. Improving acceptance of computerized prescribing alerts in ambulatory care. J Am Med Inform Assoc 2006;13:5-11.
16. Nolan TW. System changes to improve patient safety. BMJ 2000;320:771-3. 17. Reason J. Human error: models and management. BMJ 2000;320:768-70.
18. Sittig DF, Krall MA, Dykstra RH, Russell A, Chin HL. A survey of factors affecting clinician acceptance of clinical decision support. BMC Med Inform Decis Mak. 2006;6:6.
19. Feldstein A, Simon SR, Schneider J, Krall M, Laferriere D, Smith DH, Sittig DF, Soumerai SB. How to design computerized alerts to ensure safe prescribing practices. Joint Comm J Qual Saf 2004;30:602-13. 20. Krall MA, Sittig DF. Clinicians’ assessment of outpatient electronic medical record alert and reminder
usability and usefulness requirements. Proc AMIA 2002;400-4.
21. Kuilboer MM, van Wijk MAM, Mosseveld M, van der Lei J. AsthmaCritic. Issues in designing a non- inquisitive critiquing system for daily practice. J Am Med Assoc 2003;10:519-24.
22. McDonald CJ, Wilson GA, McCabe GP. Physician response to computer reminders. JAMA 1980;244:1579-81.
23. Paas F, Renkl A, Sweller J. Cognitive load theory: instructional implications of the interaction between information structures and cognitive architecture. Instruct Sci 2004;32:1-8.