USABILITY EVALUATION METHODS:
A LITERATURE REVIEW
ANKITA MADAN
CSE Department, Amity University, Noida, U.P. 201303, India
[email protected] http://www.amity.edu
SANJAY KUMAR DUBEY
CSE Department, Amity University, Noida, U.P. 201303, India
[email protected] http://www.amity.edu
Abstract:
Usability is an important factor for all software quality models. It is the key factor in the development of successful interactive software applications. Usability is the most widely used concept in the software engineering field and defines the software system’s demand and use. Due to such wide importance of this quality factor various usability evaluation methods are proposed by usability experts and researchers. This paper presents a comprehensive study of different usability evaluation methods. The objective of this paper is to lay down the intensive and conceptual study of the usability concepts.
Keywords: Usability; software; system; approach.
1. Introduction
Demand for quality software system is increasing rapidly. But at the same time there is wide range of rejections of software systems inspite of great expenses on their development. This is due to non-interaction of the system and the failure of the software system to fulfill their tasks. Usability is a product attribute that influences the quality of a software system. It is a transient and elusive concept which has various sub attributes related to it for the explanation of its abstractness. There exists several quality models like McCall (1977), FURPS(1987), Capability Maturity Models(1989), IEEE(1992), Dromey (1995), ISO(1991,1998,2001), given by researchers and experts, which accounts usability as an indispensable quality attribute for the development of a quality software system. Usability is defined as ‘the ease with which a user can learn to operate, prepare inputs for, and interpret outputs of a system or component’ (IEEE Std.1061, 1992). Usability correlates with the functionality of the system and helps in its evaluation. The lack of usability causes failure of the software system that leads to a substantial monetary loss, user dissatisfaction, staff unproductivity and time wastage. Therefore, usability evaluation is very important for the process of designing usable software system. But still there are no apt criteria or models for usability evaluation because of its fuzzy characteristics. Therefore, this paper presents extensive survey of the usability concepts and evaluation. The tour to the evolution of usability concept and evaluation methods over the past three decades has been covered in this paper.
2. Usability Models
satisfaction. The international organization of standardization gave a model consisting of three basic sub attributes namely effectiveness, efficiency, and satisfaction (ISO 9214-11, 1998). Moving ahead, ISO 9126 (2001) laid down the following sub attributes of usability namely understandability, learnability, operability, attractiveness, usability compliance. Usability model and their definitions are mentioned in Table 1.
Table 1. Taxonomy of Usability Models
Model Sub-Attributes Definitions
Eason Model (1984)
Task Frequency Number of times a task is performed by a user. Openness Extent to which a task is modifiable.
User Knowledge The knowledge that the user applies to the task. It may be appropriate or inappropriate.
Motivation How determined the user is to complete the task.
Discretion The user's ability to choose not to use some part of a system. System Ease of
learning
The effort required to understand and operate an unfamiliar system.
Ease of use The effort that is required to operate a system once it has been understood and mastered by the user.
Task match The extent to which information and functions that a system provides matches the needs of the user.
Shackel Model (1991)
Effectiveness It is described as system’s performance is better than some required level, by some required percentage of the specified target range of users, within some required portion of the range of usage environments.
Learnability It is the training of users after some specific time from installation of the system. Also, includes user’s re-learnability time for training and support systems.
Flexibility It is the positive changes or variations in the system to the existing ones. Attitude It is the acceptance of users within their levels of discomfort, tiredness,
frustration and personal effort.
Nielsen model (1993)
Learnability The system should be easy to learn and understand. It should be easy for the user to get their job or task executed using the software system. Efficiency Efficiency of the system is directly related to its productivity. The more
efficient a system is its throughput is correspondingly high.
Memorability It is best suited for intermittent users. The user can return to the system’s previous state without starting away from the beginning.
Errors The error rate in any system should be less. If any error is occurred, the system should be able to recover from it.
Satisfaction It is the pleasant feeling that user gets while or after using the system. It can be observed as likeability for the system and fulfillment of specified task.
ISO 9241-11 (1998)
Effectiveness It is the performance measure of a system to complete a specified task or goal successfully within time.
Efficiency It is the successful completion of a task by a system. It relate to accuracy and completeness of the specified goal.
Satisfaction It is acceptability of a system by the users, in specified context of use.
ISO 9126 (2001)
Understandability The capability of the software product to enable the user to understand whether the software is suitable, and how it can be used for particular tasks and conditions of use.
Learnability The capability of the software product to enable the user to learn its application.
Operability The capability of the software product to enable the user to operate and control it.
Attractiveness The capability of the software product to be attractive to the user.
3. Literature Survey
Down the time usability has been studied and discovered by great researchers and scholars. It is a multi-dimensional concept that opens areas for research. It has evolved over time and has got its relevance in many aspects.
Foley and Van Dam (1982) described it with respect to user interface guidelines as a property of the syntactic and semantic analysis of a user interface. Smith and Moiser (1984) made the next attempt by describing it as a product attribute, which defines the concept by naming product or system attributes or qualities that influence the usability. In the same year Eason supported the view, usability is the question of how well users can use that functionality. Gould (1985) defined usability as any system designed for people to use should be built by keeping in mind that it should be easy to learn and remember, it should be useful , it should contain functions that people really in their work and be easy and enjoyable to use. B. Shackel (1986) gave an outstanding definition of usability that could be used throughout the system development lifecycle. As per the description the system must accomplish following criteria: effectiveness, learnability, flexibility and attitude. Doll and Torkzadeh (1988) presented a model for satisfaction measurement called End User Computing Satisfaction Instrument (EUCSI). This was used for specific application. Ravden and Johnson (1989) they presented a usability evaluation mechanism, software inspection and gave a detailed checklist of 122 items divided into 9 dimensions. Igbaria and Parasuraman (1989) considered fun to be very influential in acceptance of any software system. Booth (1989) thought it difficult to specify and measure flexibility of a system and believed that being useful should be fundamental to usability, thus he modified Shackel’s criteria into usefulness, effectiveness, learnability (or ease of use), and attitude (or likeability). Polson and Lewis (1990) suggested for the user interface design solutions. They contributed by giving problem solving strategies for novice users when they interact with the complex interface.
part of the interface usability. The sub attributes focus on how to make the product easy to understand, easy to learn, efficient to use, less erroneous and pleasurable; ii) Apparent usability (Kurosu and Kashimura 1995, Tractinsky 1997) - is more related to the visual impression of the interface. Nielsen (1995) presented “Discount usability engineering”; Botman (1996) “Do it yourself usability evaluation’. Butler (1996) dealt with usability engineering which includes software system models, user models, interface’s model, links between these, development of the standards and prototyping activities. Harrison and Rainer (1996) reviewed a model used for computing satisfaction –EUCSI, which is defined as “an affective attitude towards a specific computer application by someone who interacts with the application directly”. It included following sub attributes: content, accuracy, format, timeliness and ease of use”. Products are tools and high degree of usability can be determined when the error rate of usability is minimum was given by Kanis (1997) and Hollnagel (1997). Gluck (1997) correlated Usability to usefulness and usableness. Usableness answers to the question “Can I invoke this function?” and Usefulness answers to question “Did it really help me?” or “Was it worth the effort?” ISO 9241-11(1998) is “Guidance on usability” which discusses usability for the purposes of system requirement specifications and its evaluation. Lecerof and Paterno (1998) provided a definition addressing importance of a system to users, efficiency of software system, user’s subjective feelings, learnability, and a system’s safety feature. Thomas (1998) categorized usability sub attributes into three categories: outcome, which includes effectiveness, efficiency and satisfaction; process, defines ease of use, interface, learnability, memorability and error recovery; and task which defines functionality and compatibility. Microsoft also regarded usability as strategies to attract user’s reactions for a system and utilizing them into the various development stages (Veldof, Prasse, and Mills, 1999). However, Head (1999) pointed a simple easy to use interface is the main value point for usability: the core value of “usability is rooted in cognitive science - the study of how people perceive and process information through learning, the use of memory, and attention”. Design guidelines contain instructions and principles that are required in building of an effective user friendly interface. This was conceptualized by the methods described into five categories: design rules, ergonomic algorithms, style guide, standards and collection of guidelines (Vanderdonckt, 1999). Usability concept was regarded for pedagogical value as an important for e-learning systems. Hence, improvement based on the UE results made the systems more usable, still such a system may not have any pedagogical sense (Squires and Preece, 1999)”. In technical writing, clear and accurate definitions are critical” (Alred, Brusaw, and Oliu, 2000) and there is a requirement of concise usability definition as it is potentially affected by technical/system and human factors. Arms (2000) stated that usability comprised of aspects including interface design, functional design, data and metadata, and the computer systems and networking.
data (appropriately); Compare means; Consider using expectation measures; Use the System Usability Scale (SUS); Show frequency distributions; Combine different metrics; Use appropriate tools; Present data appropriately.Thomas S. Tullis, (2009) explained some of the myths regarding usability, regarded them as ‘Top Ten Myths about Usability’. Gardner-Bonneau D. (2010), he talked about the software system’s capability to sustain the changes in the technical prospects without hampering the usability effectiveness.Jennifer C. Romano Bergstrom et al. (2011) carried out a demonstration and explained the benefits and challenges faced by the designers while usability testing of website design. Table 2. gives the comprehensive overview of the usability concepts.
Table 2. Quick review at the development of usability concept from 1982-2011
Researchers Usability Concepts
Foley and Van Dam (1982)
User interface guidelines.
Smith and Moiser (1984)
Described usability as product’s attribute.
Eason (1984) Interrelated usability and functionality.
Gould(1985) Defined usability in terms of learnability, usefulness and ease of use.
Shneiderman (1986) Guidelines for error prevention, discussed the system’s response time, data entry within HCI.
Shackel(1986) Defined usability with the factors effectiveness, learnability, flexibility and attitude. Tyldesley (1988) Mentioned 22 factors that could be used to build the metrics and specifications.
Doll & Torkzadeh (1988)
End User Computing Satisfaction Instrument (EUCSI).
Ravden & Johnson (1989)
Presented software inspection as usability evaluation mechanism.
Igbaria & Parasuraman (1989)
Enjoyability is directly proportional to acceptance of a system
Booth (1989) He modified Shackel’s criteria into usefulness, effectiveness, learnability, and attitude.
Polson & Lewis (1990) He gave problem solving strategies for novice users to interact with the complex interface.
Holcomb & Tharp (1990)
Presented a software usability model for the system designers to decide which usability sub attributes should be included.
Brian Shackel (1991) Elaborated the usability concept.
Mayhew (1992) Reviewed usability principles to describe the desirable properties of the interface. Grudin (1992) Practical acceptability of the system within the various categories like cost, support,
system usefulness.
Nielsen (1993) Presented usability heuristics for the inspection method of usability evaluation. He classified usability to, learnability, efficiency, memorability, errors, and satisfaction. Dumas & Redish
(1993)
explained their definition of usability on the basis of focus on users, usability means, use of product by users for productivity, users are busy people trying to accomplish tasks, decision of user about when the product is easy to use.
Preece et al. (1993) Categorized usability into sub attributes namely: safety, effectiveness, efficiency and enjoyableness.
Beimal et al. (1994) Principles of acceptance for usability.
Nielsen & Levy (1994) Worked on user satisfaction assessment of product. Logan (1994) Divided usability into social and emotional dimension.
Caplan(1994) Defined apparent usability as an important consideration in the design of a software system.
Preece et al. (1995) Related usability to overall performance of the system and user satisfaction.
Lamb (1995) Claimed usability as a wider concept which includes content usability, organizational usability and inter organizational usability.
Guillemette (1995) Reviewed and defined usability with respect to effective use of information system. Kurosu & Kashimura
(1995)
Divided usability into Inherent usability and Apparent usability.
Researchers Usability Concepts
Butler (1996) Dealt with usability engineering. Harrison & Rainer
(1996)
Reviewed a model used for computing satisfaction –EUCSI.
Kanis & Hollnagel (1997)
High degree of usability can be determined when the error rate of usability is minimum.
Gluck (1997) Correlated Usability to usefulness and usableness.
Tractinsky(1997) Contributed in explaining the concept of Apparent usability. Lecerof & Paterno
(1998)
Declared functionality being essential to usability.
Thomas (1998) Categorized usability sub attributes into three categories: outcome, process, and task.
ISO 9241-11(1998) “Guidance on usability” which discusses usability for the purposes of system requirement specifications and its evaluation.
Veldof, Prasse, & Mills (1999)
Related usability, user’s reaction and system development
Vanderdonckt (1999) Design guidelines and principles to build an effective user friendly interface. Kengeri et al. (1999) Explained usability using effectiveness, likability, learnability and usefulness. Squires & Preece
(1999)
Usability concept was regarded for pedagogical value for e-learning systems.
Arms (2000) Aspects of usability that are interface design, functional design, data and metadata, and the computer systems and networking.
Alred et al. (2000) Related usability to technical/system and human factors.
Battleson et al.(2001) Explained interface design that is easy to learn, remember, and use, with few errors. Hudson (2001) The concept of web usability was described.
Turner(2002) Illustrated a checklist for the evaluation of usability. Blandford &
Buchanan (2002)
Explained usability in terms of technical, cognitive, and social design. Also, looked into the future work on methods for analyzing usability.
Palmer (2002) Explained usability in context of web usability. Oulanov & Pajarillo
(2002)
Interface effectiveness as one of the most important aspects of interaction”.
Matera et al. (2002) Gave “Systematic usability evaluation”. Guenther (2003)
Pack (2003)
Illustrated the difficulties in defining usability.
Campbell & Aucoin (2003)
Explained usability as a relationship between tools and its users.
Abran et al. (2003) Referred usability as a set of multiple concepts, performance of the system, execution time of a specified task, user satisfaction and ease of learning.
Whitney Quesenbery (2001,2002, 2003)
Presented “the five E’s of usability” which include effectiveness, efficiency, engagement, error tolerance, and ease of learning.
Villers (2004), Drigus & Cohen (2005), Miller (2005)
Expressed usability evaluation methods should consider pedagogical factors.
Krug(2006) Studied usability from the user’s perspective based on their experience. Dee & Allen (2006) End-user interface conforms to usability principles.
Seffah, Donyanee, Kline & Padda (2006)
Gave 10 usability factors namely, efficiency, effectiveness, productivity, satisfaction, learnability, safety, trustfulness, accessibility, universality, and usefulness are associated with twenty-six usability measurement criteria.
Brophy & Craven (2007)
Explained web usability.
Tom Tullis & Bill Albert (2008)
Presented ‘Tips and Tricks for Measuring the User Experience’.
Thomas S. Tullis (2009)
Explained ‘Top Ten Myths about Usability’.
Gardner-Bonneau (2010)
Explained the effectiveness sustained by the software system when technical changes are made to it.
Jennifer C. Romano Bergstrom et al. (2011)
4. Usability evaluation
Various methods are available in the literature for usability evaluation like Inspection, DRUM, QUIS, SUMI MUSIC, Empirical testing.
4.1 Inspection
This method is proposed by Boehm et al. (1976).The users are observer or testing and evaluation of the design layout of the software system is done by the experts. It provides expert’s views and opinions which are essential for development of various aspects of the software system. The two most widely used inspection methods are:
4.1.1. Heuristic Evaluation
It is fast, cheap and easy method to figure out the shortcomings and problems in a user interface design. The evaluators use the usability principles or the heuristics for its implementation.
4.1.2. Cognitive Walkthrough
This method is based on assessment of the user interface by the experts who consider the opinion and experience of the users. It is useful in identifying the problems of user interface.
4.2 Empirical testing
This method was proposed by Marciniak, J.J., (2002). It is a lab oriented methodology which accounts user experience as a requirement for the design and development of the software system. It also examines performance and attitude of the users involved in testing the system (Lund, A. M., 1997). The naive users are allowed to interact with the system and the behavior of the user and the system’s response is recorded. The system can either be a prototype or the final product on which the testing is performed. In case of prototype system, amendments can be made for the successful design of the product whereas in the case of final product user acceptance can be measured. Accordingly, if required a system can be discarded.
4.3 Metrics for Usability Standards in Computing (MUSiC)
MUSiC (Bevan, 1995; Macleod et al., 1997) was developed at the National Physical Laboratory, UK, for the purpose of quantitative and qualitative data required to support usability engineering. This method evaluates the measures of effectiveness and efficiency as follows:
• Effectiveness is the capability of a software system to carry out the specified task successfully. It is defined as a function of two components, the quantity of the task attempted by the users, and the quality of goals they achieve (Miles Macleod et al., 1998).
Effectiveness = f(Quantity, Quality)
Task Effectiveness (TES), = (Quantity*Quality) % 100
• Efficiency and cost of task performance, this can be formulated by calculating the amount of effort put into that is the basically the input. Hence, two definitions can be generated which are mentioned as:
User Efficiency = Effectiveness where [Task Time= time spent by user to complete the task] Task Time
Human Efficiency = Effectiveness Effort
4.4 Software Usability Measurement Inventory (SUMI)
SUMI was developed by University College Cork Software Usability Measurement Inventory as part of the MUSIC project (Kirakowski, Porteous and Corbett, 1992). It measures the quality of the software system from the end user’s point of view. It consists of industry standardized questionnaire statements which are answered by the user according to whether they Agree, Don’t Know, or Disagree. This internationally structured 50-item questionnaire is available in variety of languages for the convenience of the users like in English, German, Dutch, Spanish and Italian. It is not at all time consuming and takes 10 minutes or so. We can decide to administer it on paper or on the other hand can decide for the internet, online option.
4.5 Diagnostic Recorder for Usability Measurement (DRUM)
DRUM (Macleod and Rengger, 1993) is a software tool developed at NPL within the MUSIC project, for usability evaluation. DRUM has a graphical user interface, online context-sensitive help and a comprehensive user manual (Macleod et al., 1992). It analyses the tests of a product and derives performance-based usability metrics from the results and send it to the usability engineer. The video session recorded is analyzed in real time during first pass of recording. The DRUM increases the pace of the analysis greatly and automates the activity wherever possible. The component of the DRUM, Log Processor provides calculation to its database of performance measures and performance based usability metrics, which includes task time; snag, Search and Help Times; Efficiency; Relative Efficiency; Productive Period (Miles Macleod, Rosemary Bowden and Nigel Bevan, 1998). There is a tabular and graphical representation of the measures and metrics. The report is given to the product’s designers who are concerned with the usability defects.
4.6 The Questionnaire for user interaction Satisfaction (QUIS)
QUIS (Chin et al. 1988, Harper and Norman 1993) evaluation is based on the factors mentioned in the ‘User Evaluation of Interactive Computer Systems’ given by Ben Shneiderman (1986). QUIS is designed in modular format to get section wise accessibility as well as to specific aspects. This questionnaire provided is effective in providing guidance in the design or redesign of systems. It also helps the evaluators find the areas of potential improvement in the software system. Thereby, it serves as a testing instrument in usability laboratories and operates on the concrete product features and the user experience.
5. Conclusion
Usability concept has been under focus over the years and has evolved with different definitions by researchers. Different attributes have been built for a clear view of usability and its aspects. The usability has been decomposed into several sub attributes which are hypothetical constructs to define the success of a system. User involvement plays key role in determining the software usability after it has been developed. This paper has surveyed research papers, published articles and views of usability experts to describe usability models, usability evaluation methods and has determined the sub attributes of usability which form the basis for the usability evaluation of the software system. The paper will be beneficial for the both the students and the researchers who are working in the field of software engineering. There is still a dilemma about appropriate selection of measurement technique for usability evaluation of a software system. To find such an appropriate technique is the future scope of this paper.
References:
[1] Alred, G. J.; Brusaw, C.T.; Oliu, W.E. (2000). Handbook of technical writing, 6th edn. New York: Macmillan.
[2] Abran, A.; Khelifi, A.; Suryn; Seffah, Ahmed W. (2003): Consolidating the ISO Usability Models. Proceedings of International Software Quality Management Conference Springer). Glasgow, Scotland, UK.
[3] Alonso-Ríos, D.; Vázquez-García, A.; Mosqueira-Rey, E.; Moret-Bonillo, V. (2010): Usability: A Critical Analysis and a Taxonomy Department of Computer Science, University of Coruña, Spain, Int. Journal Of Human–Computer Interaction, 26(1), pp. 53–74. [4] Arms, W. Y. (2000). Digital libraries. Cambridge, Mass.: MIT Pr.
[5] Battleson, B.; Booth, A.; Weintrop, J. (2001): Usability testing of an academic library Web site: A case study. The Journal of Academic Librarianship, 27(3), pp. 188-198.
[6] Bevan, N. (1995): Measuring usability as quality of use, Software Quality Journal4, pp. 115–130.
[7] Bevan, N.; Macleod, M. (1994): Behaviour and Information Technology, Usability measurement in context Nigel Bevan and Miles Macleod National Physical Laboratory, Teddington, Middlesex, UK. 13, pp. 132-145.
[8] Blandford, A.; Buchanan, G. (2002a): Usability for digital libraries. Proceedings of the second ACM/IEEE-CS Joint Conference on Digital Libraries. New York: ACM Press, pp. 424.
[10] Boehm, B.W.; Brown, J.R.; Lipow, M. (1976): Quantitative Evaluation of software quality, international Conference on Software Engineering Proceedings.
[11] Booth, P. (1989). An introduction to human-computer interaction. Hillsdale, USA: Lawrence Erlbaur Associates Publishers.
[12] Botman, H. (1996): Do-it-yourself usability evaluation: Guiding software developers to usability. Taylor & Francis, London, pp. 59-66.
[13] Butler, K. A. (1996): Usability engineering turns 10, Interactions, 3, pp. 59-75.
[14] Campbell, K.; Aucoin, R. (2003). Value-based design of learning portals as new academic spaces, In: Jafari, A. and M. Sheehan, Designing Portals: Opportunities and Challenges, Hershey, PA: IRM Press, pp. 162–185.
[15] Caplan, S.H. (1994). Making usability a Kodak product differentiator. In: Wiklund, M.E. (Ed.), Usability in Practice. AP Professional, NY, pp. 21–58.
[16] Chin, J.P.; Diehl, V.A.; Norman, K.L. (1988): Development of an instrument measuring user satisfaction of the human-computer interface. Proc. ACM CH1'88 Conf. (Washington, DC 15-19 May), pp. 213-218.
[17] De Villers, R. (2004): Proceedings of the annual research conference of the South African institute of computer scientists and information technologists on IT research in developing countries, Usability evaluation of an e-learning tutorial: criteria, questions and case study. pp. 284-291.
[18] Doll, W. J.; Torkzadeh, G: (1988): The Measurement of End-User Computing Satisfaction, MIS Quarterly, pp. 259-274.
[19] Dringus, L.P.; Cohen, M.S., (2005): Proceedings 35th Annual Conference Frontiers in Education. FIE'05, an adaptable usability heuristic checklist for online courses. pp. T2H-6.
[20] Dee, C.; Allen, M. (2006): A Survey of the Usability of Digital Reference Services on Academic Health Science Library Web Sites. The Journal of Academic Librarianship, 32(1), pp. 69-78.
[21] Dumas, J.S.; Redish, J. A. (1993). Practical Guide to Usability Testing, Ablex Publishing Norwood NJ.
[22] Eason K. D. (1984): Towards the experimental study of usability, Behaviour and Information Technology, 3(2), pp. 133-143. [23] Foley, J.; Dam, A. (1982): Fundamentals of Interactive Computer Graphics. Rea, Addison-Wesley, USA.
[24] Gardner-Bonneau, D. (2010): Is Technology Becoming More Usable or Less and With What Consequences, Journal of Usability Studies, 5(2), pp. 46-49.
[25] Gluck, M. (1997): A descriptive study of the usability of geospatial metadata. Annual Review of OCLC Research. Accessed: October, 2011, www.oclc.org/research/publications/arr/1997/gluck/gluck_frameset.htm.
[26] Gould, J.D.; Lewis, C. (1985): Designing for Usability, Key Principles and What Designers Think, Communications of the ACM, 28(3).
[27] Grudin, J. (1992): Utility and Usability: Research Issues and Development Contexts, Interacting with Computers, 4(2), pp. 209-2 . [28] Guenther, K. (2003): Assessing Web site usability. Online, 27 (2), pp. 65-68.
[29] Guillemette, Ronald A. (1995). The evaluation of usability in interactive information systems. In Human factors in information systems: Emerging theoretical bases, Jane M. Carey. Norwood, N.J.: Ablex.
[30] Harrison, A.W.; Rainer Jr, R.K. (1996). A General Measure of User Computing Satisfaction. In Computers in Human Behavior, 12(1), pp. 79-92.
[31] Head, A. (1999): Web redemption and the promise of usability. Online, 23(6), pp. 20-32.
[32] Holcomb, R.; Tharp, A. (1991): Users, a software usability model and product evaluation, Interacting with Computers, Butterworth-Heinemann, Oxford, UK, 3(2), pp. 155-166.
[33] Hix, D.; Hartson, H.R. (1993). Developing User Interfaces: Ensuring usability through product and process, chap2, Wiley and Sons, NY.
[34] Hollnagel, E. (1997): Cognitive ergonomics or the mind at work. Proceedings of the 13th Triennial Congress of the International Ergonomics Association, Tampere Finland 1997, Finnish Institute for Occupational Health, Helsinki, 3, pp. 3-5.
[35] Hudson, L. (2001): From theory to (virtual) reality. Library Journal, 126 (11), pp. 12-15.
[36] IEEE Std. 1061. (1992): IEEE standard for a software quality metrics methodology, New York, IEEE Computer Society Press. [37] Igbaria, M. and Parasuraman, S. (1989): A path analytic study of individual characteristics, computer anxiety, and attitudes toward
microcomputers. Journal of Management, 15, pp. 373-388.
[38] ISO 9126. (1991): Software Product Evaluations- Quality characteristics and guidelines for their use, ISO DIS 9126.
[39] ISO 9241. (1998): Ergonomics requirements for office work with visual display terminals (VDTs) – Part 11: Guidance on usability. [40] ISO/IEC 9126-1, (2001): Software engineering - Product quality -Part 1: Quality model.
[41] Bergstrom, J. C. R.; Olmsted-Hawala, E. L.; Chen, J. M.; Murphy, E. D. (2011): Conducting Iterative Usability Testing on a Web Site: Challenges and Benefits Journal of Usability Studies, 7(1), pp. 9-30
[42] Kanis, H. (1997): Usability centered research for everyday product design. Proceedings of the 13th Triennial Congress of the International Ergonomics Association, Tampere Finland, Finnish Institute for Occupational Health, Helsinki, 2, pp. 153-155.
[43] Krug, S. (2006). Don't make me think: A common sense approach to Web usability. Berkeley, CA: New Riders Publishing.
[44] Kurosu, M.; Kashimura, K. (1995): Apparent usability vs. inherent usability: Experimental analysis on the determinants of the apparent usability. Conference on Human Factors and Computing Systems. New York: ACM Press, pp. 292-93
[45] Lamb, R. (1995). Using online resources: Reaching for the *.*s. In Digital Libraries’95, F. M. Shipman, R. Furuta, and D. M. Levy, 137–46. Austin, TX: Department of Computer Science, Texas A&M University.
[46] Leventhal & Barnes, (2009). Defining Usability and Models of Usability Engineering: Process, Products and examples, chapter 3. [47] Lecerof, A.; Paterno, F. (1998): Automatic Support for Usability Evaluation. IEEE Transaction on Software Engineering, 24(10), pp.
863-888.
[48] Logan, R.J. (1994): Behavioral and emotional usability: Thomson consumer electronics. In: Wiklund, M.E. (Ed.), Usability in Practice. AP Professional, NY, pp. 59-82.
[49] Lund, A. M. (1997): Expert ratings of usability maxims. Ergonomics in Design. A study of the heuristics design experts consider important for good design. 5(3), pp. 15-20.
[50] Macleod, M.; Drynan, A.; Blayney, M. (1992): DRUM User Guide. National Physical Laboratory, DITC, Teddington, UK.
[51] Macleod, M.; Bowden, R.; Bevan, N.; Curson, I.; (1997): The MUSiC performance method, Behaviour and Information Technology 16 , pp. 279-293.
[52] Marciniak, J.J. (2002): Encyclopedia of software Engineering, 2, 2nd edn, Chichester: Wiley.
[53] Matera, M.; Costabile, M.F.; Garzotto, F.; Paolini, P.; e Inf, D.E. (2002): SUE inspection: an effective method for systematic usability evaluation of hypermedia, IEEE Transactions on Systems, Man and Cybernetics, Part A, 32(1), pp. 93-103.
[54] Mayhew, D. J. (1992): Principles and guidelines in software user interface. Prentice Hall, Englewoog Cliffs, NJ.
[55] Macleod, M.; Rengger, R. (1993): The Development of DRUM: A Software Tool for Video-assisted Usability Evaluation National Physical Laboratory DITC HCI Group Teddington, Middlesex, TW11 0LW, UK.
[57] Nielsen, J. (1993): Usability Engineering. Academic press, San Diego, CA.
[58] Nielsen, J.; Levy, J. (1994): Measuring usability: Preference vs. Performance, Communications of the ACM, 37 (4), pp. 66-76. [59] Nielsen, J. (1995): Scenarios in Discount Usability Engineering in Caroll, J.M., (Ed), Scenarios-Based Design: Envisioning Work and
Technology in System Development. John Wiley and Sons.
[60] Oulanov, A.; Edmund, F. Y. Pajarillo, (2002): CUNY + Web: Usability study of the Web-based GUI version of the bibliographic database of the City University of New York (CUNY). The Electronic Library 20 (6), pp. 481–87.
[61] Pack, T. (2003): Fiddling with the Internet dials: Understanding usability. Online, 27 (2), pp. 36-38.
[62] Palmer, J. W. (2002): Web site usability, design, and performance metrics. Information Systems Research, 13(2), pp.151 - 167. [63] Polson, P.G.; Lewis, C. H. (1990): Theory–based design for easily learned interfaces. Human-Computer Interaction, 5, pp.191-220. [64] Porteous, M.; Kirakowski, J.; Corbett, M. (1993). SUMI User Handbook. Human Factors Research Group, University College Cork,
Ireland.
[65] Preece, J.; Benyon, D.; Davies, G.; Keller, L.; Rogers, Y. (1993). A guide to usability: Human factors in computing. Reading, MA: Addison-Wesley.
[66] Preece, J. Y. Rogers; H. Sharp; D. Benyon; S. Holland, T. (1994). Carey, Human-Computer Interaction, Addison Wesley.
[67] Quesenbery, W. (2003): Dimensions of usability. In Albers, M., & Mazur, B., Content and complexity: Information design in technical communication. Mahwah, NJ: Lawrence Erbaum Associates.
[68] Ravden, S.; Johnson, G. (1989). Evaluating usability of Human computer Interfaces: A practical method. Ellis Hardwood Limited, New York.
[69] Seffah, A.; Donayaee, M.; Kline, R.B.; Padda, H.K. (2006): Usability measurement and metrics: A consolidated model, Software Quality Control, 14(2).
[70] Shneiderman, B. (1986): Designing the user interface: Strategies for effective human-computer interaction. Addison-Wesley, Reading MA.
[71] Shackel, B. (1986): Ergonomics in design for usability. In Harrison, M. D. and A. F. Monk (Ed.) Proceedings of the Second Conference of the British Computer Society Human Computer Interaction Specialist Group: people and Computers – Design for Usability Cambridge. British Computer Society Human Computer Interaction Specialist Group, New York: Cambridge University Press, pp. 44-64.
[72] Shackel, B. (1991). Usability—Context, framework, definition, design and evaluation, in B. Shackel and S. Richardson, Human Factors for Informatics Usability, Cambridge, MA: University Press, pp. 21–38.
[73] Squires, D.; Preece, J. (1999): Predicting quality in educational software: Evaluating for learning, usability and the synergy between them, interacting with computers, 11(5), pp. 467-83.
[74] Smith, S.; Mosier, J. (1984): Design Guidelines for the User Interface for Computer-Based Information Systems, Bedford, MA: The MITRE Corporation.
[75] Thomas, R. L. (1998). Elements of performance and satisfaction as indicators of the usability of digital spatial interfaces for information-seeking: Implications for ISLA. PhD diss., Univ. of Southern California.
[76] Thomas S. Tullis, (2009): Top Ten Myths about Usability,Simmons College.
[77] Tractinsky, N. (1997): Aesthetics and apparent usability: Empirically assessing cultural and methodological issues. CHI’97 Conference proceedings, pp. 115-122.
[78] Tullis, T.; Albert, B. (2008): Tips and Tricks for Measuring the User Experience Usability and User Experience, UPA-Boston's Seventh Annual Mini UPA Conference.
[79] Turner, S. (2002): The HEP test for grading Web site usability. Computers in Libraries 22 (10), pp. 37–39.
[80] Tyldesley, D. A. (1988): Employing usability engineering in development of office products. Computer Journal, 31(5), pp. 431-436. [81] Vanderdonckt, J. (1999): Development Milestones towards a Tool for Working with Guidelines. Interacting with Computers, 11 (4). [82] Veldof, J. R.; Prasse, M. J.; Mills, V. A. (1999): Chauffeured by the user: Usability in the electronic library. Journal of Library
