Evaluating Nurses' Satisfaction With Two Nursing
Information Systems
Reza Khajouei, PhD, Reza Abbasi, MSc
Evaluating user satisfaction is one of the methods to ensure the usability of information systems. Considering the impor-tance of nursing information systems in patient health, the ob-jective of this study is to evaluate nurses' satisfaction with two widely used nursing information systems (Peyvand Dadeh and Tirazhe) in Iran. This descriptive-analytical study was done on 230 nurses in all teaching hospitals of Kerman Univer-sity of Medical Sciences in 2015. Data were collected using an augmented version of a questionnaire developed by IBM. Data were analyzed by SPSS.16 using descriptive and analytical statistical methods including t test, analysis of variance, and Pearson correlation coefficient. The mean of overall satisfaction with the two systems was 61 ± 2.2 and 74 ± 2.4, respectively. The mean of satisfaction with different systems dimensions, that is, ease of use, infor-mation quality, and interface quality, was, respectively, 24 ± 1.9, 26 ± 9.7, and 12 ± 4.7 for Tirazhe and 29 ± 1.1, 39 ± 1.04 and 13 ± 5.3 for Peyvand Dadeh sys-tem. Nurses’ satisfaction with both systems was at a me-dium level. The majority of nurses were relatively satisfied with the information quality and user interface quality of these systems. The results suggest that designing nursing information systems in accordance with their users’ need improves usability. Hence, policy and decision makers of healthcare institutions should invest on usability when pur-chasing such systems.
KEY WORDS: Computer System Usability Questionnaire, Nursing information system, Satisfaction, Usability evaluation
I
n many hospitals, information technology is widely used in nursing practice and supports nurse’s daily activities.1A nursing information system (NIS) is a subsystem of a
hospital information system (HIS), which deals with differ-ent aspects of nursing, especially documdiffer-entation and storage of nursing records.2It has been shown to improve planning of care, accessibility of nursing documents, and readability of information. Moreover, it reduces duplicate documentation, increases the completion rate of nursing documentation and medical examinations, and improves nurses’ workflow. In nurs-ing care, NIS improves the quality of data required for nursnurs-ing decision making and, subsequently, the quality of patient care. Nurses can spend the extra time saved by the system on caring for patients.3–5 Despite many benefits of NISs, studies have shown that some of them do not have a desirable design, lead-ing to usability problems for users.6–9Usability is regarded as the help given by the system to the users in performing their roles, resulting in a better understanding of the system, system ease of use, and preventing mental and physical pressures when using the system.10,11The International Organization for Stan-dardization defines usability as“the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified con-text of use.”12
In addition to increasing clinical productivity, usability can reduce user exhaustion, confusion, and errors, and ulti-mately, it raises their level of satisfaction.11Without user sat-isfaction, the system will be abandoned. Accordingly, due to the high costs of designing the system, significant finan-cial resources will be needed to purchase and implement a substitute information system.10Therefore, it is essential to evaluate user satisfaction as one of the important com-ponents of NIS usability.
Various questionnaires were designed to evaluate the us-ability of information systems. Among them, the Computer System Usability Questionnaire (CSUQ) proposed by IBM13 evaluates the quality of information and satisfaction with system user interface. Although some questionnaires are appropriate for a specific goal or for collecting data from a specific group, CSUQ can be used in various environ-ments with different users.14To our knowledge, the CSUQ has not yet been extensively used to evaluate the satisfaction of users with health information systems. Except one study that was done on an electronic medical record (EMR),15 other studies using CSUQ were done on Web sites or administrative systems.16–18
Copyright © 2017 Wolters Kluwer Health, Inc. All rights reserved.
Author Affiliations:Medical Informatics Research Center, Institute for Futures Studies in Health, and Department of Health Information Management and Technology, Faculty of Management and Medical Information Sciences (Dr Khajouei); and Health Service Management Research Center, Institute for Futures Studies in Health (Mr Abbasi), Kerman University of Medical Sciences, Kerman, Iran.
This work was supported by the Medical Informatics Research Center, Kerman University of Medical Sciences (no. 94/262).
The authors have disclosed that they have no significant relationship with, or financial interest in, any commercial companies pertaining to this article.
Corresponding author:Reza Abbasi, MSc, Health Services Management Research Center, Institute for Futures Studies in Health, Kerman University of Medical Sciences, PO Box: 7616911320, Haft-bagh Highway, Kerman, Iran ([email protected]).
Due to the effect of NIS on patients’ health and the need for an appropriate user interface to help information quality, the objective of this study is to evaluate user satisfaction with NIS. For this purpose, two commonly used NISs in the uni-versity hospitals were evaluated using the CSUQ.
METHODS
This descriptive-analytical study evaluated nurses’ satisfac-tion with two widely used informasatisfac-tion systems in four teach-ing hospitals of Kerman University of Medical Sciences in 2015. At the time of study, 899 nurses were working in these hospitals. A sample of 269 nurses, calculated by Cochrane formula and the error level of 5%, were invited to participate in the study. The sample size was determined by stratified sampling proportionate to the number of nurses in each hospital. The sample size of nurses in Shafa, Afzalipoor, Bahonar, and Beheshti hospitals was calculated to be 90, 83, 76, and 20, respectively.
System
The evaluated NISs are the subsystems of two HISs released by Tirazhe and Peyvand Dadeh companies (Tehran, Iran). These systems are used in 186 and 60 hospitals in Iran, re-spectively. Using these systems, nurses are able to do the fol-lowing activities: recording nursing services, requesting medication from the hospital pharmacy, recording medica-tions used by patients (Figure 1), communicating orders to radiology (Figure 2) and laboratory departments, sharing documents (eg, lab results) with other parts of hospitals (Figure 3), and ordering diets for patients.
Data Collection
Data were collected with the approval of ethics commit-tee of Kerman University of Medical Sciences (IR.kmu. REC.1394.726).We used the CSUQ developed by IBM to collect the data. This questionnaire was initially translated into Persian accurately. Then, to ensure the cross-cultural adaptation, the translated questionnaire was translated back to English by a proficient English translator.
The questionnaire included three sections:
1.Demographic information such as gender, age, educational level, and work experience, which was added by researchers.
2.Questions related to nurses' satisfaction with the NIS. This section consisted of three dimensions: system ease of use (questions 1–8), information quality of system (questions 9–15), and the user interface quality (questions 16–18). In addition, overall satisfaction with the system was evalu-ated based on questions 1 to 18. These questions could be answered based on a 7–point Likert scale from 1 to 7. One indicated the lowest satisfaction, and 7, the highest satis-faction score.
3.Open-ended questions for writing the most positive and neg-ative aspects of systems.
Content validity of the CSUQ was confirmed by three medical informatics experts and a nurse. The reliability of the questionnaire was tested on questionnaires filled out by 20 random participants using Cronbach’s α (α =.88). Data were collected in three shifts by distributing questionnaires proportionate to the number of shift nurses in each clinical ward. In each ward, the questionnaires were given to the nurses exiting patient rooms until the sample size of the cor-responding shift was obtained. Participants’ consensus was implied by their agreement to fill out the questionnaires.
FIGURE 1. The screen for requesting medication for the patient (Tirazhe systems).
FEATURE
ARTICLE
and collected the questionnaires once completed.
Data Analysis
For data interpretation, first, the minimum and maximum scores that could be earned for each dimension were calcu-lated by multiplying the lowest and highest score of each question to the number of questions of each dimension. Then, based on these scores, we classified the overall satisfac-tion with the system and satisfacsatisfac-tion with each dimension
scores were considered as follows—for overall satisfaction: low, 18 to 53; medium, 54 to 89; and high, 90 to 126; for ease of use dimension: low, 8 to 23, medium, 24 to 39; and high, 40 to 56; for the information quality dimension: low, 7 to 20; medium, 21 to 34; and high, 35 to 49; and for the user interface quality dimension: low, 3 to 8; medium, 9 to 14; and high, 15 to 21.
Data were analyzed by IBM SPSS Statistics version 16 (IBM, Armonk, NY) using descriptive and analytic statistics.
After checking the normality of data, we usedt test to com-pare the mean satisfaction of participants in terms of their gender and history of working with computers, analysis of variance was used for educational level and employment status, and Pearson correlation coefficient was used for assessing the relationship between quantitative data such as nurses’ satisfaction with age, work experience, and the daily time nurses spent to use the system.
RESULTS
Of 269 nurses, 230 responded to the questionnaires (85%). Demographic information of the participants for both sys-tems is shown in Table 1. The mean (SD) age and work experience of the participants were 32 (7) and 9 (7) years, re-spectively. About 88% of the participants were women, and nearly 95% of them had a bachelor’s degree. In total, 76% of the nurses used Tirazhe and 24% Peyvand Dadeh NIS. Also, the majority (29%) of the participants had contrac-tual employment status, and those with temporary (a 2-year obligatory contract) had the lowest frequency (22%). More than 70% of the participants stated that they had used a computer before working with NIS.
The mean overall satisfaction scores of nurses with both NISs are shown in Table 2. There was no significant rela-tionship between the mean overall satisfaction scores of nurses and the gender, educational level, and employment status of participants for both systems (P > .05).
There was a significant difference between the mean sat-isfaction scores of those who had already used a computer and those who had not used a computer before using the NIS (P = .042). The results showed an inverse and nonsignif-icant correlation between the overall satisfaction and work experience (r = −0.1, P = .099) as well as the age and overall satisfaction with the nurses (r = −0.05, P = .42). There was a direct, weak, and nonsignificant correlation between the users’ satisfaction and their level of using the system (r = 0.02, P = .711).
Comparing the Usability of Two NISs: Tirazhe and
Peyvand Dadeh
Table 3 illustrates the mean nurses’ satisfaction with various aspects of the usability of the two NISs. In general, the mean nurses’ overall satisfaction, their satisfaction with the system ease of use, and satisfaction with the information quality of the Peyvand Dadeh system were significantly higher than those who used the Tirazhe system (P ≤ .003). There was no significant difference between the mean nurses’ satisfac-tion with the user interface of two systems (P = .19).
Table 1.
Demographic Information of the Participants
for Both Systems
Demographic Information Tirazhe, n (%) Peyvand Dadeh, n (%) Gender Female 163 (93) 42 (77) Male 13 (7) 12 (23) Age 21–30 89 (51) 28 (52) 31–40 55 (31) 22 (41) 41–50 29 (16) 4 (7) >50 3 (2) 0 Educational level Bachelor's degree 168 (95) 51 (94) Master's degree 8 (5) 3 (6) Employment status Permanent 51 (29) 8 (15) Contractual 47 (27) 20 (37) Arbitrary 39 (22) 16 (30) Temporary 39 (22) 10 (18)
History of working with computer
Yes 127 (73) 40 (74)
No 48 (27) 14 (26)
Table 2.
Overall Satisfaction Scores of Nurses in Both
Systems Based on Their Demographic Information
Demographic Information Overall Satisfaction, Mean ± SD Gender Female 64 ± 2.33 Male 69 ± 2.43 Educational level Bachelor's degree 64 ± 2.35 Master's degree 67 ± 2.25 Employment status Permanent 61 ± 2.28 Arbitrary 66 ± 2.62 Contractual 63 ± 2.2 Temporary 67 ± 2.16
History of working with computer
Yes 81 ± 3.51
No 59 ± 2.01
Table 3.
Nurses
’ Satisfaction With the Usability of
Two Nursing Information Systems
Dimensions of the Usability
Tirazhe, Mean ± SD Peyvand Dadeh, Mean ± SD Ease of usea 24 ± 1.9 29 ± 1.1 Information qualityb 26 ± 9.7 39 ± 1.04 Interface qualityc 12 ± 4.7 13 ± 5.3 Overall satisfactionb 61 ± 2.2 74 ± 2.4 a P ≤ .003. b P < .0001. c P = .19.
FEATURE
ARTICLE
Table 4 shows the distribution of nurses according to their level of satisfaction for dimensions of the usability of two NISs. The overall satisfaction for more than half of nurses with two systems was at a medium level. In general, the level of nurses’ satisfaction with system ease of use, information quality, and user interface of the Peyvand Dadeh was higher than with that of Tirazhe system.
Overall, the mean and median scores of the nurses’ satis-faction with the productivity of these systems, information quality, screen layout, and also organization of information for both systems were 5, on a scale from 1 to 7. The examin-ing of questions related to system alert messages, message display and online help, and user-friendliness of the systems revealed that about 44% of the Tirazhe and 57% of the Peyvand Dadeh users were dissatisfied with alerts received about obvious problems. Thirty-eight percent of the Tirazhe and 60% of the Peyvand Dadeh users were dissatisfied with functionalities such as online help, presentation of alerts, and unclear messages. About 41% of the Tirazhe and 51% of the Peyvand Dadeh users did not enjoy working with the screen of either system.
In response to the open-ended questions, 56 nurses re-ported on the positive aspects and 44 on the negative aspects of the systems. Examples of most important positive and neg-ative aspects are shown in Table 5. In addition to expressing the positive and negative aspects, respondents demanded the provision of clear error messages and corresponding solu-tions when users make mistakes. Use of more eye-catching colors in the design; improving the system speed and upgrading systems; adding number of test requests, includ-ing photographic, such as cone beam computed tomogra-phy; and making the design of the system easier to learn are among other requested design modifications that partic-ipants believed would enhance the usability of the systems.
majority of nurses were relatively satisfied with the informa-tion quality and user interface of both Tirazhe and Peyvand Dadeh NISs.
There was no significant relationship between overall sat-isfaction and gender, educational level, and employment sta-tus. The results showed no significant correlations between the degree of satisfaction and work experience, as well as age. In some studies,19–25there was no correlation between age and user satisfaction. Contrary to the above result, in the surveys conducted by Khajouei26on computerized phy-sician order entry systems and Kim27on NISs, the age of users affected their satisfaction so that older users were more satisfied than their younger colleagues were. This difference in the result might be because of the different questionnaires used. Also, the Kim study assessed satisfaction with the NIS, degree of satisfaction influence, degree of satisfaction safety and security, and degree of satisfaction efficacy and effective-ness. In addition, the Khajouei study investigated a comput-erized physician order entry system that is different from NIS. In our study, there was no significant correlation be-tween the overall users’ satisfaction and use of NIS. In the Bahnassy study,19the degree of nurses’ satisfaction was corre-lated with the use of HIS.
There was no significant relationship between overall sat-isfaction and gender, educational level, and employment sta-tus. Likewise, two previous studies19,26revealed that gender had no relationship with users’ satisfaction.
In our study, there was a significant correlation between having working experience with computer and nurses’ sat-isfaction. The results of some previous studies19,26were consistent with ours and the results of others21,22,25were in-consistent. This disagreement might be because the later studies were published more than 15 years ago. In the past, computers were not popular in healthcare and computer training was mostly done in schools and universities. Today, people can receive computer training almost everywhere and nurses have many opportunities and time to study com-puter and experience HISs.27
The results of this study show that nurses are relatively satisfied with both NISs and these systems can improve the efficiency and productivity of nurses. Page and Schadler15 evaluated user’s satisfaction before and after redesigning an EMR by use of results of the CSUQ. Comments of users were considered in all phases to redesign this system, and the results of the reevaluation showed a high correlation be-tween system usability and users’ satisfaction, as well as effi-ciency and effectiveness. The results of other studies that had been done on different systems of NIS17,28–31indicated that ease of use can increase efficiency and improve nurses’ Dimensions of the
Usability of the System Scale Tirazhe Peyvand Dadeh
Ease of use Low 97 (55%) 20 (37%)
Medium 66 (38%) 25 (46%) High 3 (7%) 9 (17%)
Information quality Low 64 (36%) 7 (13%) Medium 81 (46%) 24 (44%) High 31 (18%) 23 (43%)
Interface quality Low 53 (30%) 15 (28%) Medium 76 (43%) 20 (37%) High 47 (27%) 19 (35%)
Overall satisfaction Low 71 (40%) 11 (20%) Medium 88 (50%) 29 (54%) High 17 (10%) 14 (26%)
workflow. Users are likely to be more satisfied with informa-tion systems if they believe that using the system will increase performance and productivity.32,33
According to the Michel-Verkerke34study, the NIS is in general considered to be easy to use but training is needed. About half of the users were satisfied with the ease of using both NISs. Likewise, in the Ologeanu-Taddei et al35study, evaluating the ease of use of an HIS and its usefulness, 50% of nurses reported its ease of use.
Information quality, as one of the determiners of the hospital healthcare level, is a critical element in a hospi-tal.36 Similar to the Michel-Verkerke,34 study results, the majority of users were satisfied with information quality and thought studied systems have positive effects on quality of information, but need improvement. As in-dicated by Michel-Verkerke, the achieved quality of infor-mation depends largely on the data entry by the nurses themselves, and controversy exists between the required information quality and the effort needed to accomplish it. According to Ammenwerth,3despite the technical problems of NISs, they are useful and have significantly improved information quality.
The interaction between humans and computers via a user interface has a significant impact on ease of use.37In this study, the mean score of satisfaction with the user interfaces of both NISs was equal to 4 (ie, at a medium level) and the majority of nurses were relatively satisfied with the user in-terfaces’ quality. Consistent with this result, Madathil38
showed a relative and significant satisfaction of patients with the user interfaces of a hospital research management sys-tem. Likewise, in the Blagec et al39study, more than 60% of participants agreed that the usability interface design of Medication Safety Code (MSC) was appealing. As well, half of the interviewed pharmacists stated that they appreci-ated the conciseness and ease of use of the MSC interface.
Although these systems are different, the users were satisfied with their user interfaces.
In our study, about half of the nurses were dissatisfied with alerts to fix problems. The results of open questions showed that some nurses reported that absence of proper alerts con-tributes to nurses’ errors for entering data and wastes their time. It has been demonstrated that nurses and physicians demanded additional alerts regarding drug allergies, wrong prescriptions, and noncompliance with the test results.26 According to Horsky et al,40poor display of alerts increases users’ mental efforts and requires spending more time searching for information.
Usability is important in establishing and improving the quality.41Various questionnaires have been used for evaluating the usability of health information systems,19,22,24,35,42–44and only one study15has used the CSUQ in this domain. We used the CSUQ in this study since it is designed for large sample sizes, which makes it suitable for the nursing population. More-over, it is suitable for field testing situations and provides a general evaluation of the system.13 The CSUQ evaluates satisfaction with system ease of use and user interface, as well as information quality.
Previous usability evaluations of HISs45,46 identified a number of usability problems such as using inappropriate colors and functions prolonging user-system interaction. These types of problems were also reported by our partici-pants. Our participants reported other problems, such as the presence of several codes in the systems, which are diffi-cult to memorize, and unsuitable colors that cause eyes strain and subsequently headache and fatigue. There was also dif-ficulty in modifying or removing some of the recorded mea-sures in the system. Previous studies have shown that many of aforementioned usability problems can be fixed by simpli-fying user interfaces and adding online help to the system when required,47,48 using clear labels,49 designing user
Table 5.
The Most Positive and Negative Aspects of the Two Systems
System Positive Aspects Negative Aspects
Tirazhe Readily performed tasks (n = 11) Low-speed software (n = 8) Easy access to information, easy to use, high speed system,
reduced workload (n = 3)
Provokes eyestrain and, subsequently, headache (n = 5) Appropriate classification of drugs, appropriate system
information, easy to learn, reduced errors (n = 2)
System disconnection (n = 4) Facilitates information communication, real-time entry of
information, improves documentation, convenient information retrieval (n = 1)
Increases time on task, increases workload (n = 2)
Excessive use of codes, difficulty in modifying or canceling actions, difficult to learn, difficult to work (n = 1)
Peyvand Dadeh Easy access to information and lab results (n = 4) Hangs and slows down, increases workload, absence of error messages, difficulty in resolving problems, absence of required information such as cone beam computed tomography and some other tests (n = 1) Reduces time on tasks, accelerates working speed (n = 3)
Increases productivity (n = 2) Real-time entry of information (n = 1)
according to the result, Peyvand Dadeh had significantly higher information quality and ease of use than Tirazhe did. The mean score of interface quality was also higher in Peyvand Dadeh. Overall, nurses were more satisfied with the Peyvand Dadeh system. Based on the explanatory responses of the participants, positive aspects of Peyvand Dadeh, such as easy access to information and requiring less time to accomplish the tasks, and specific negative aspects of Tirazhe, such as confusing colors, which provoke headaches, contribute to these differences. This study has the limitation of recruiting all nurses using two independent NISs in all teaching hospitals of one univer-sity as the sample of study. It can be concluded that more ac-curate results can be achieved by data collection in a broader geographic range from other users of these two systems. However, due to the almost identical distribution of nurses in Iranian hospitals and the similarity of their activities, the participants in this study can be considered as a representa-tive sample of total users of these systems.
The users of these systems have not been engaged in the development of either of the systems. Hence, we could not directly evaluate the effect of user’s contribution on the us-ability of systems. However, we sought the perspective of users on this subject. The results showed that users identified aspects of the systems that diminish the usability because they are not aligned with nurse’s needs and expectations. Also they specified certain functionalities of the system that could im-prove usability if they would have been added to the systems. As examples the participants required clear error messages with instruction to recover form error, providing informa-tion related to frequent tasks and using appropriate colors in the design. Previous studies48,49have also reported that these changes can improve interaction of users with systems. This study evaluated the satisfaction of nurses with two widely used information systems by addressing three impor-tant dimensions of the systems (ease of use, information qual-ity, and interface quality). Evaluating systems from the prospective of their end users helps to improve aspects of the systems that really matter in utilization of the system but may have been underestimated by the designers and eventually improve the outcome of the system. Users can best identify and express the gaps between their workflow and the design of a system.50This will help to attain the main goal of developing health information systems, which is im-proving the quality of care provided to patients. The results of this study shed light on the aspects of the systems that is praised by users and make their interaction convenient. This study also highlighted the design weaknesses of the systems and the aspects that users required to be improved. The re-sults can be helpful for developers of health information
a system.
CONCLUSION
Nurses’ satisfaction with using both systems was at a medium level. The majority of nurses were relatively satisfied with the information quality and user interface of these systems.
The results of this study can be helpful for designers and developers of health information systems to develop and im-prove systems according to their users’ needs. Usability is de-fined in the context of user-system interaction, user tasks, and system goals.
Based on the results, we recommend that policy and deci-sion makers of healthcare institutions pay special attention to and invest in usability when purchasing such systems and consider the barriers that may impact users’ interaction with systems.
The results suggest that designing NIS in accordance with their users’ need improves usability. Specifically updating the system, reducing the number of unfamiliar codes for nurses and replacing them with information congruent to nurse’s activities, classifying this information on the screen, pro-viding online or offline help, and presenting appropriate alerts on the occurrence of an error can improve the de-sign and usability of systems. Since this study was based on the results of a survey on how the nurses’ computer ex-perience impact on their satisfaction with NIS, we suggest providing computer training courses for nurses prior to working with the system.
Acknowledgment
The authors thank the nurses who participated in this study.
References
1. Ball MJ, Edwards MJ, Hannah KJ. Introduction to Nursing Informatics. New York: Springer; 2006.
2. Ioanna P. Nursing documentation and recording systems of nursing care. Health Sci J. 2007;(4).
3. Ammenwerth E, Rauchegger F, Ehlers F, Hirsch B, Schaubmayr C. Effect of a nursing information system on the quality of information processing in nursing: an evaluation study using the HIS-monitor instrument. Int J Med Inform. 2011;80(1): 25–38.
4. Hou IC, Yu HC, Hwang YJ, Cheng J, Wu RS, Chang P. The evaluation of the nurses’ work performance after nursing information system implementation in one medical center in Taiwan. Stud Health Technol Inform. 2009;146: 708. 5. Larrabee JH, Boldreghini S, Elder-Sorrells K, et al. Evaluation of
documentation before and after implementation of a nursing information system in an acute care hospital. Comput Nurs. 2001;19(2): 56–65. quiz 66–58.
6. Khajouei R, Jaspers MW. The impact of CPOE medication systems’ design aspects on usability, workflow and medication orders: a systematic review. Methods Inform Med. 2010;49(1): 3–19.
7. Koppel R, Metlay JP, Cohen A, et al. Role of computerized physician order entry systems in facilitating medication errors. JAMA. 2005;293(10): 1197–1203.
8. Shu K, Boyle D, Spurr C, et al. Comparison of time spent writing orders on paper with computerized physician order entry. Stud Health Technol Inform. 2001;84(pt 2): 1207–1211.
9. Tierney WM, Miller ME, Overhage JM, McDonald CJ. Physician inpatient order writing on microcomputer workstations: effects on resource utilization. JAMA. 1993;269(3): 379–383.
10. Khajouei R, Peek N, Wierenga PC, Kersten MJ, Jaspers MW. Effect of predefined order sets and usability problems on efficiency of computerized medication ordering. Int J Med Inform. 2010;79(10): 690–698.
11. Marcus A. 2 Chapter—User Interface Design's Return on Investment: Examples and Statistics. Cost-Justifying Usability. 2nd ed. San Francisco, CA: Morgan Kaufmann; 2005: 17–39.
12. International Organization for Standardization. ISO 9241: Ergonomic Requirement for Office Work With Visual Display Terminals (VDTs) _ Part 11. Guidance on Usability. Geneva, Switzerland; 1998.
13. Lewis JR. IBM computer usability satisfaction questionnaires: psychometric evaluation and instructions for use. Int J Hum Comput Interact. 1995; 7(1): 57–78.
14. Lewis JR. User Satisfaction Questionnaires for Usability Studies. Boca Raton, FL: International Business Machines Corporation; 1991.
15. Page CA, Schadler A. A nursing focus on EMR usability enhancing documentation of patient outcomes. Nurs Clin North Am. 2014;49(1): 81–90.
16. De Lucia A, Francese R, Passero I, Tortora G. SmartBuilding: a people-to-people-to-geographical-places mobile system based on augmented reality. UBICOMM 2010. 2010.
17. Rusli NM, Hassan S, Liau NE. Usability analysis of students information system in a public university. J Emerg Trends Eng Appl Sci. 2013;4(6): 806–810.
18. Tullis TS, Stetson JN. A comparison of questionnaires for assessing website usability. Paper presented at: Usability Professional Association Conference (UPA 2004). Minneapolis, Minnesotta; 2004: 7–11.
19. Bahnassy AA. Nurses satisfaction with the use of health information system (HIS) in A Saudi tertiary care medical center. Int J Adv Res Comput Sci Software Eng. 2015;5(5): 56–61.
20. Lee TT, Lee TY, Lin KC, Chang PC. Factors affecting the use of nursing information systems in Taiwan. J Adv Nurs. 2005;50(2): 170–178. 21. Marasovic C, Kenney C, Elliott D, Sindhusake D. Attitudes of Australian
nurses toward the implementation of a clinical information system. Comput Nurs. 1997;15(2): 91–98.
22. Mazzoleni MC, Baiardi P, Giorgi I, Franchi G, Marconi R, Cortesi M. Assessing users' satisfaction through perception of usefulness and ease of use in the daily interaction with a hospital information system. Proceedings: a conference of the American Medical Informatics Association/…AMIA Annual Fall Symposium. AMIA Fall Symposium. 1996: 752–756.
23. Palm J-M, Colombet I, Sicotte C, Degoulet P. Determinants of user satisfaction with a clinical information system. Age (years). 2006;1(1): 1. 24. Sittig DF, Kuperman GJ, Fiskio J. Evaluating physician satisfaction regarding
user interactions with an electronic medical record system. In: Proceedings of the Annual Symposium on Computer Application in Medical Care, Hanley & Belfus, Philadelphia; 1999: 400–404.
25. Sleutel M, Guinn M. As good as it gets? Going online with a clinical information system. Comput Nurs. 1999;17(4): 181–185.
26. Khajouei R, Wierenga P, Hasman A, Jaspers MW. Clinicians satisfaction with CPOE ease of use and effect on clinicians' workflow, efficiency and medication safety. Int J Med Inform. 2011;80(5): 297–309.
27. Kim S-Y. Factors affecting the degree of satisfaction for nursing information system. Stud Health Technol Inform. 2005;122: 523–526.
28. Lee F, Teich JM, Spurr CD, Bates DW. Implementation of physician order entry: user satisfaction and self-reported usage patterns. J Am Med Inform Assoc. 1996;3(1): 42.
29. Lindenauer PK, Ling D, Pekow PS, et al. Physician characteristics, attitudes, and use of computerized order entry. J Hosp Med. 2006;1(4): 221–230. 30. Rosenbloom ST, Talbert D, Aronsky D. Clinicians’ perceptions of clinical
decision support integrated into computerized provider order entry. Int J Med Inform. 2004;73(5): 433–441.
31. Weiner M, Gress T, Thiemann DR, et al. Contrasting views of physicians and nurses about an inpatient computer-based provider order-entry system. J Am Med Inform Assoc. 1999;6(3): 234–244.
32. Mawhinney CH, Lederer AL. A study of personal computer utilization by managers. Inform Manage. 1990;18(5): 243–253.
33. Vlahos GE, Ferratt TW. Information technology use by managers in Greece to support decision making: amount, perceived value, and satisfaction. Inform Manage. 1995;29(6): 305–315.
34. Michel-Verkerke MB. Information quality of a nursing information system depends on the nurses: a combined quantitative and qualitative evaluation. Int J Med Inform. 2012;81(10): 662–673.
35. Ologeanu-Taddei R, Morquin D, Bourret R. Understanding the perceived usefulness and the ease of use of a hospital information system: the case of a French university hospital. Stud Health Technol Inform. 2015;210: 531.
36. Ratnaningtyas DD, Surendro K. Information quality improvement model on hospital information system using six sigma. Proc Technol. 2013;9: 1166–1172.
37. Sutiono AB, Qiantori A, Prasetio S, et al. Designing an emergency medical information system for the early stages of disasters in developing countries: the human interface advantage, simplicity and efficiency. J Med Syst. 2010;34(4): 667–675.
38. Madathil KC, Koikkara R, Obeid J, et al. An investigation of the efficacy of electronic consenting interfaces of research permissions management system in a hospital setting. Int J Med Inform. 2013;82(9): 854–863. 39. Blagec K, Romagnoli KM, Boyce RD, Samwald M. Examining perceptions of
the usefulness and usability of a mobile-based system for
pharmacogenomics clinical decision support: a mixed methods study. Peer J PrePrints. 2015;3: e1910.
40. Horsky J, Kaufman DR, Oppenheim MI, Patel VL. A framework for analyzing the cognitive complexity of computer-assisted clinical ordering. J Biomed Inform. 2003;36(1): 4–22.
41. Youngblood NE, Mackiewicz J. A usability analysis of municipal government website home pages in Alabama. Gov Inf Q. 2012;29(4): 582–588. 42. Hadji B, Dupuis I, Leneveut L, Heudes D, Wagner J-F, Degoulet P.
Determinants of continuance intention in a post-adoption satisfaction evaluation of a clinical information system. Stud Health Technol Inform. 2014;205: 990–994.
43. Hortman PA, Thompson CB. Evaluation of user interface satisfaction of a clinical outcomes database. Comput Inform Nurs. 2005;23(6): 301–307. 44. Sinha RK, Kurian S. Assessment of end user satisfaction of hospital
information system informatic Al Spitalului. Manage Health. 2014;18(3). 45. Agharezaei Z, Khajouei R, Ahmadian L, Agharezaei L. Usability evaluation
of a laboratory information system. Director General. 2013;10(2). 46. Montazeri M, Khajouei R, Sabermahani F. Evaluation of radiology and
pathology subsystems of hospital information systems. JKerman Univ Med Sci. 2015;22(2): 194–204.
47. Grammenos D, Akoumianakis D, Stephanidis C. Integrated support for working with guidelines: the Sherlock guideline management system. Interact Comput. 2000;12(3): 281–311.
48. Nabovati E, Vakili-Arki H, Eslami S, Khajouei R. Usability evaluation of laboratory and radiology information systems integrated into a hospital information system. J Med Syst. 2014;38(4): 1–7.
49. Atashi A, Khajouei R, Azizi A, Dadashi A. User Interface problems of a nationwide inpatient information system: a heuristic evaluation. Appl Clin Inform. 2016;7(1): 89–100.
50. Rojas CL, Seckman CA. The informatics nurse specialist role in electronic health record usability evaluation. Comput Inform Nurs. 2014;32(5): 214–220.
