Current Usability Metrics for Mobile Computing Evaluation
3. Research Approach
This section explains the steps taken to review the literature. A systematic literature review (SLR) method will be used in this phase to ensure that previous studies on usability evaluation relevant to this study are considered. The journals selected in this study were the top journals in HCI from 2006 until 2008. The total of 409 journal papers was reviewed based on keywords “usability”, “evaluation” and “metric”. Only 26 out of 409 journal papers were selected for further review in obtaining the guidelines and usability metric for mobile application development. Table 2 below describes the journal papers that were reviewed.
Table 2. Journal papers reviewed.
Journal Year Candidate Selected
ACM Transactions on Computer-Human
Interaction 2006–2008 54 8
Human-Computer Interaction 2006–
2008 36 2
International Journal of Human-
Computer Interaction 2006–2008 97 5
International Journal of Human- Computer Studies
2006– 2008
222 11
Total 409 26
The review is based on a conception of usability that is similar to ISO 9241-11 (1998) and Bevan and MacLeod (1994). This conception merely discusses studies related to usability evaluation rather than the broad concept of usability. We analyse the quality characteristics of each measure to ensure that no duplication exists. Interestingly, we found that most of the studies employed effectiveness, efficiency, and satisfaction as quality characteristics, which appear in ISO 9241 as well. Thus, we decided to make these three characteristics base guidelines and to designate others as sub-guidelines. Table 3 describes the most popular usability guidelines obtained from the literature. In the following section, we review the approaches that previous studies utilised to identify the issues that arise in creating usability metrics and discuss the results.
Table 3. Usability guidelines from the literature.
No Guidelines Description
1 Completeness The extent or completeness of users’ solutions to tasks. 2 Accurate The accuracy with which users complete tasks.
3 Fewer errors or
no errors Errors made by the user during the process of completing a task. 4 Ease of data input The data input process should be simple.
5 Ease of output
use The output should be very simple and accurate. 6 Ease to install Application installation should be user friendly. 7 Response time The system must respond in an appropriate time. 8 Simple The application should be straightforward.
9 Time The duration of tasks or parts of tasks.
10 Ease to learn The user interface must be designed for the user to learn easily.
11 Application size The space used by the application should be appropriate. 12 Battery power
used The battery power use by the application. 13 Wireless
connectivity
The application should easily connect to a network. 14 Features available Appropriate features available on an application. 15 Satisfaction with
the interface Measures satisfaction as the interface that users prefer using. 16 Provide
support/help The help information given by the application should be useful. 17 Safety User should be safe and secure while using the
application.
4. Results and discussion
With reference to Table 3, we simplify the usability metric by removing the guidelines that are not relevant to interaction. Duplicate guidelines are omitted as well, to assure that they have no effect on other guidelines. Since this study focuses on mobile computing, we create several new guidelines related to mobility—for instance, “touch screen facilities,” “safety while driving,” and “automatic update”—which do not appear in existing literature. In Table 4, we present the results by categorising quality characteristics and their corresponding usability guidelines and metrics.
Table 4. Usability metrics for mobile computing.
Quality
Characteristic Usability Guidelines Usability Metrics
Simplicity • Ease of data input • Ease of output use • Ease of installation • Ease of learning Effectiveness Accuracy • Accuracy • Absence of errors • Success
Time taken • To respond
• To complete a task Efficiency Features • Support/help • Touch-screen facilities • Voice guidance • System resource
Safety • While using the application • While driving
Satisfaction
Attractiveness • User interface
In the current literature, several issues arise with respect to creating usability metrics. For example, few studies employ experts to assess quality. Expert assessments are needed to evaluate subjective data and usability defects (Bevan & MacLeod, 1994). Hence, we would suggest employing experts to create usability metrics, as expert judgement is one of the essential methods in usability inspection. We also note that measures on how to use interfaces are rarely employed. Current measurements for interfaces focus more on the use of colours; layout and information structuring; consistency of the terminology; consistency of the interaction mechanisms; and how to fit in small-screen devices. Metrics for interface use will be different for mobile applications due to the novelty and size of mobile devices.
The measure of satisfaction is another important element in measuring usability, and questionnaires are one of the techniques employed to measure user satisfaction. However, many studies are not using validated questionnaires to assess satisfaction. A large number of questionnaires have been developed to assess the user’s subjective satisfaction of the system and related issues. These include the Questionnaire for User Interface Satisfaction (QUIS), developed by Chin et al. (1988) and SUMI, developed by Kirakowski & Corbett (1993). Finally, we found that many studies combine both objective and subjective measures—for example, “learnability of an interface” and “time needed to master an interface into a single metric.” We believe that such measurements should be separated rather than included in a single metric.
5. Conclusion
We have reviewed usability metrics employed in 26 studies published in core HCI journals, drawing attention to several issues relating to the production of these measurements. The metrics created in this paper could serve as alternatives to current usability metrics in mobile computing. In addition to the issues raised in this study, we intend, in future research, to explore other challenges in creating usability metrics, particularly in the area of interaction between humans and mobile applications.
References
Abrahams, P. (2008) 'The future of usability testing,' Bloor Research; available at: http://www.it-director.com/business/compliance/content.php?cid=10495.
Ahmed, S., Mohammad, D., Rex, B.K., and Harkirat, K.P. (2006) 'Usability measurement and metrics: a consolidated model,' Software Quality Control 14 (2), 159–178.
Ballard, B. (2005) User Interface Design Guidelines for J2me Midp 2.0. Lawrence, KS: Little Springs Design, Inc.
Barnard, L., Yi, J.S., Jacko, J.A., and Sears, A. (2005) 'An empirical comparison of use-in- motion evaluation scenarios for mobile computing devices,' International Journal of Human-Computer Studies 62 (4), pp. 487–520.
Bevan, N. and MacLeod, M. (1994) 'Usability measurement in context,' Behaviour and Information Technology 13 (1), 132–145.
Björk, S., Redström, J., Ljungstrand, P., and Holmquist, L.E. (2000) 'POWERVIEW: using information links and information views to navigate and visualize information on small displays,' Proceedings of the 2nd International Symposium on Handheld and Ubiquitous
Computing, Bristol, UK.
Brewster, S., Chohan, F., and Brown, L. (2007) 'Tactile feedback for mobile interactions,' Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Washington, D.C., pp 159–162.
Chin, J.P., Diehl, V.A., and Norman, K.L. (1988) 'Development of an instrument measuring user satisfaction of the human-computer interface,' Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Washington, D.C., pp. 213–218.
Hornbæk, K., & Law, E. L. C. (2007) ‘Meta-analysis of correlations among usability measures,’ Paper presented at the Proceedings of the SIGCHI conference on Human factors in computing systems.
Dong-Han, H., Jeongyun, H., Fossick, P., Wong, W., Park, S., Chiwon, S., and Bradley, M. (2006) 'Conceptual framework and models for identifying and organizing usability impact factors of mobile phones,' Proceedings of the 18th Australia Conference on Computer- Human Interaction: Design: Activities, Artefacts and Environments, Sydney, pp. 261–268. Drummond, J.S. and Themessl-Huber, M. (2007) 'The cyclical process of action research: the
contribution of Gilles Deleuze,' Action Research 5 (4), pp. 430–448.
Hornbæk, K. and Law, E.L.-C. (2007) 'Meta-analysis of correlations among usability measures,' Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, San Jose, CA, pp. 617–626.
International Organization for Standardization (ISO) (1998) 'ISO 9241-11' (Guidance on usability); available at: www.iso.org/iso/home.htm.
Isomursu, M., Tähti, M., Väinämö, S., and Kuutti, K. (2007) 'Experimental evaluation of five methods for collecting emotions in field settings with mobile applications,' International Journal of Human-Computer Studies 65 (4), pp. 404–418.
Kirakowski, J. and Corbett, M. (1993). 'SUMI: the Software usability measurement inventory,' British Journal of Educational Technology 24 (3), pp. 210–212.
Kjeldskov, J. and Stage, J. (2004) 'New techniques for usability evaluation of mobile systems,' International Journal of Human-Computer Studies 60 (5/6), pp. 599–620.
MacKenzie, I.S. and Soukoreff, R.W. (2003) 'Phrase sets for evaluating text entry techniques,' CHI '03 Extended Abstracts on Human Factors in Computing Systems, Ft. Lauderdale, FL, pp. 754–55.
Macleod, M. and Rengger, R. (1993) 'The development of DRUM: a software tool for video- assisted usability evaluation,' Proceedings of the HCI'93 Conference on People and Computers VIII, Loughborough, UK, pp. 293–309; available at:
http://www.nigelbevan.com/papers/drum93.pdf.
Scapin, D. and Law, E.L.-C., eds. (2007) Review, Report and Refine Usability Evaluation Methods (R3UEMs). Athens: COST294-MAUSE 3rd International Workshop.
Stafford, T.F. and Gillenson, M.L. (2003) 'Mobile commerce: what it is and what it could be,' Communications of the ACM 46(12), pp. 33–34.
Steinar, K. and Ljungberg, F. (1999) 'Mobile informatics: innovation of IT use in mobile settings: IRIS'21 workshop report,' ACM SIGCHI Bulletin 31 (1), pp. 29–34.
Zhang, D. and Adipa, B. (2005) 'Challenges, methodologies, and issues in the usability testing of mobile applications,' International Journal of Human-Computer Studies 18 (3), pp. 293–308.