The Impact of the User Interface on SimulationUsability and Solution Quality
289
0
0
Full text
(2) The Impact of the User Interface on Simulation Usability and Solution Quality. by Bruce Montgomery [email protected]. A dissertation submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy in Computer Information Systems. Graduate School of Computer and Information Sciences Nova Southeastern University 2011.
(3) We hereby certify that this dissertation, submitted by Bruce Montgomery, conforms to acceptable standards and is fully adequate in scope and quality to fulfill the dissertation requirements for the degree of Doctor of Philosophy. _____________________________________________ Maxine S. Cohen, Ph.D. Chairperson of Dissertation Committee. ________________ Date. _____________________________________________ Timothy J. Ellis, Ph.D. Dissertation Committee Member. ________________ Date. _____________________________________________ Sumitra Mukherjee, Ph.D. Dissertation Committee Member. ________________ Date. Approved:. _____________________________________________ Amon B. Seagull, Ph.D. Interim Dean. ________________ Date. Graduate School of Computer and Information Sciences Nova Southeastern University 2011.
(4) An Abstract of a Dissertation Submitted to Nova Southeastern University in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy. The Impact of the User Interface on Simulation Usability and Solution Quality by Bruce Montgomery June 2011 This research outlines a study that was performed to determine the effects of user interface design variations on the usability and solution quality of complex, multivariate discrete-event simulations. Specifically, this study examined four key research questions: what are the user interface considerations for a given simulation model, what are the current best practices in user interface design for simulations, how is usability best evaluated for simulation interfaces, and specifically what are the measured effects of varying levels of usability of interface elements on simulation operations such as data entry and solution analysis. The overall goal of the study was to show the benefit of applied usability practices in simulation design, supported by experimental evidence from testing two alternative simulation user interfaces designed with varying usability. The study employed directed research in usability and simulation design to support design of an experiment that addressed the core problem of interface effects on simulation. In keeping with the study goal of demonstrating usability practices, the experimental procedures were analogous to the development processes recommended in supporting literature for usability-based design lifecycles. Steps included user and task analysis, concept and use modeling, paper prototypes of user interfaces for initial usability assessment, interface development and assessment, and user-based testing of actual interfaces with an actual simulation model. The experimental tests employed two interfaces designed with selected usability variations, each interacting with the same core simulation model. The experimental steps were followed by an analysis of quantitative and qualitative data gathered, including data entry time, interaction errors, solution quality measures, and user acceptance data. The study resulted in mixed support for the hypotheses that improvements in usability of simulation interface elements will improve data entry, solution quality, and overall simulation interactions. Evidence for data entry was mixed, for solution quality was positive to neutral, and for overall usability was very positive. As a secondary benefit, the study demonstrated application of usability-based interface design best practices and processes that could provide guidelines for increasing usability of future discrete-event simulation interface designs. Examination of the study results also provided suggestions for possible future research on the investigation topics..
(5) Acknowledgements There were many people who deserve my sincere gratitude for their inspiration and support in my efforts to perform my doctoral research. I’d like to thank my supervisors from positions I held during my doctoral studies: Michael Slack, Howard Sykes, and Lucas Clarke. I am grateful to the support each of them provided for this work, both financial and personal. Another supervisor from two prior positions, Michael Sharp, was a particular motivation to my desire for continuing education. In the ten years I worked with Mike, and in years to follow, he was an example for me of a thoughtful and caring leader and an innovative and insightful researcher. He has shaped much of my professional and personal outlook and I am indebted to him for his support, guidance, and friendship. I would be remiss if I did not recognize the participants in this study, including the interface designers and the host of test subjects. In every case of the over 50 people involved, any request I made for assistance was immediately answered. I was struck over this period by the unhesitating generosity in time provided by all the participants, and I certainly could not have completed this study without their engaged involvement; my genuine thanks to all who came to my aid. Sincere appreciation is also due for my dissertation committee from Nova Southeastern University. Dr. Cohen, my chair, provided ten years of patient encouragement and inspired my interest in usability with her HCI class; Dr. Ellis provided true guidance in thorough research methods and the peer-review process; and Dr. Mukherjee renewed my interest in simulation and decision support, and also helped me identify a research topic. I have encouraged others to attend NSU, largely based on the high level of quality and engagement evidenced in the doctoral program. Thanks for your guidance and support. My dear friend, Dr. Martin Czigler, a gifted researcher, was my initial inspiration many years ago for wanting to pursue a doctorate. Martin dove in at the last stage of the study to help assess and structure my statistical analysis, and the study is much stronger for his involvement. I value his assistance as well as his continuing friendship and support. My parents, Helen and Bruce Montgomery, Sr., encouraged me from my earliest days to read and appreciate math and science. Maybe all those years of science fairs finally lead to something… Thanks for being there to help me find my way. And finally but most importantly, my deep gratitude for the support of Pattie, my wife, and Sean and Rhiannon, my children, who have patiently endured my efforts to work through this doctoral pursuit. I hope during my studies, as with every day, that I have shown that they are my foremost concern, and that I truly appreciate the gift of their support and love, which I hope to try to repay many times over in the days to come..
(6) Table of Contents Approvals ii Abstract iii Acknowledgements iv List of Tables vii List of Figures viii Chapters 1. Introduction 1 Problem Statement and Goal 1 Relevance and Significance 5 Barriers and Issues 8 Assumptions, Limitations, and Delimitations 12 Definitions of Terms 13 Summary 14 2. Review of the Literature 15 Historical Overview 15 Literature Specific to Topic 18 Summary 44 3. Methodology 45 Research Questions 45 Research Methods 47 Study Procedures 47 Study Activities Summary by Stages 49 Study Activities – Content and Support 51 Deliverables 66 Formats for Results 70 Required Resources 71 Summary 72 4. Results 73 Overview 73 Results and Deliverables of Study Stages 73 Experimental Data Analysis 94 Findings 97 Summary of Results 113 5. Conclusions, Implications, Recommendations, and Summary 115 Conclusions 115 Implications 125 Recommendations 127 Summary 129. v.
(7) Appendixes A. Heuristic Evaluation Form Outline 135 B. Final Subject Consent Form 138 C. Paper Prototyping Task Outline 141 D. Post Test Questionnaire 143 E. Final UML Simulation Interface Use Case 148 F. Final UML Simulation Application Sequence Diagram 150 G. Final UML Simulation Application Component Diagram 152 H. Final UML Simulation Interface Cluster Diagram 154 I. Final UML Simulation Activity Diagram 156 J. Paper Prototyping Task Profiles 158 K. IRB (Institutional Review Board) Permission Letter 162 L. Permission Letter for Test Facility Use 164 M. Test Plan Handouts 166 N. Initial Paper Prototypes 170 O. Final Paper Prototypes 175 P. Assessment Notes from Heuristic Analysis 182 Q. Test Task Handout 187 R. Mathematica Source Code for Simulation Engine 190 S. Session Notes Form 194 T. Raw Collected Experimental Data 196 U1. Data Analysis Details – Data Entry 202 U2. Data Analysis Details – Analysis 212 U3. Data Analysis Details – Usability Questionnaire 218 V. Summary of Data Analysis Significance and Means Comparison 242 W. Mathematica Source Code for Data Analysis 245 X. Subject Comments on Usability Questionnaires 249 Y. High-fidelity Prototypes for Heuristic Analysis 254 Z. Final Application Interface 263. Reference List 269. vi.
(8) List of Tables. Tables 1. Usability principles for simulation software from Ören and Yilmaz (2005) 27 2. Typical alternative interface design differences from interaction elements and usability heuristics 56 3. Deliverables for the overall study effort 67. vii.
(9) List of Figures. Figures 1. Comparison of typical usability-based design cycle vs. process for study 48 2. High-level use case for simulation interface 53 3. Discrete-event simulation problem modeled as a UML activity diagram 55 4. Preliminary UML system component diagram 59 5. Comparison of results for basic and improved interface data entry times 118 6. Comparison of results for basic and improved interface data errors and task failures 118 7. Comparison of results for basic and improved interface analysis task times 119 8. Comparison of results for basic and improved interface analysis errors and task failures 120 9. Comparison of results for PSSUQ (Lewis, 1993) aggregate measures 121. viii.
(10) 1. Chapter 1 Introduction. Problem Statement and Goal Problem statement The problem addressed by this study is to determine whether selected user interface design variations significantly affect the usability and solution quality of complex, multivariate discrete-event simulations. If usability design and test techniques can be demonstrated experimentally to improve simulation interaction and results, the argument for including such techniques in simulation development lifecycles will be strengthened. General benefits of usability methods are understood. Bias and Mayhew (2005) outline the general benefits of such efforts, including increases in user productivity, decreases in errors, and reduced cost of training and support. But the impact in simulation is potentially higher due to the complex nature of such applications, which employ complex mathematical models that evolve over time using variations of model inputs and examining their effect on output performance measures (Law, 2007). The creation of input data and models is generally held to be the most time consuming element of discrete-event simulations (Randell & Bolmsjo, 2001) and resulting output can be difficult to interpret, making it hard to recognize differences between system interrelationships and randomness (Banks, Carson, Nelson, & Nicol, 2010). The need for.
(11) 2 improved interface support is noted in some sources (Palaniappan, Sawhney, & Sarjoughian, 2006) with rare specific calls for usability in simulation development lifecycles (Ören & Yilmaz, 2005), yet many simulations that include interface design do not address usability (Heilala, Montonen, Salmela, and Pasi (2007) for example).. Problem background While there is a nearly intuitive understanding that a lack of effective user-interfaces could inhibit simulation use, development, and analysis, many simulation packages, especially those targeted at complex modeling tasks, are developed with a minimal focus on the HCI aspects of the eventual product. As an example, one overview of a building energy simulation program, based on an extremely complex model, is intentionally designed with the barest of interfaces – simple text-based data file input and output (Crawley, Winkelmann, Lawrie, & Pedersen, 2001). The user interface is left to thirdparty developers. Papamichael (1998) points out that in such large building simulation models, “informed decisions require the management of vast amounts of information” about combinations of options and performance criteria. Yet, most building energy simulation programs are “developed by researchers, for research purposes, and are not easy to use” (Papamichael, 1998, p. 1-2).. Goals The overall goal of this study was to determine the effects of user interface design variations on the usability and solution quality of complex, multivariate discrete-event simulations. Experimental interface designs varied the level of usability in data entry and validation, application flow and presentation, user feedback, error prevention and.
(12) 3 recovery, and help sub-systems. Specific effects impacted by these interface design variations included interaction time, error rates, and user satisfaction for common simulation interactions such as data input and model specification, parameter changes for simulation experiments, review of simulation results, and user support (Kuljis, 1996).. Demonstrating measurable effects through experimental assessment of interface usability on both simulation use and solution quality may bring more focus on including usability design in simulation development. Specifically, evaluating the solutions derived from alternate varied interfaces to a single core simulation provides a quantitative measure of usability importance not available in the current literature. Other beneficial aspects of the study includes identifying which HCI aspects contribute to effective simulation use, as well as identifying usability issues specific to those elements, through standard usability assessment. Finally, the study illustrates use of usability design and assessment methods in simulation development, providing some guidance for interested developers.. Research questions The four key research questions for this study include: . What are user interface considerations for discrete-event simulation models?. . What are best practices for designing an interface to a simulation application?. . How is usability best evaluated for simulation interfaces?. . What is the actual effect of increased usability for specific interface elements on simulation operations, such as data input and solution analysis?.
(13) 4 For simulation applications, there are consistent sets of characteristic operations that must be considered, regardless of the simulation topic. In development of an assessment criteria for simulation environments, Tewoldeberhan and Bardonnet (2002) outline these operations, including model development, input modes, testing, execution, animation, output, and other user considerations. Design of the interface to address these common operations must also consider the user profile. As discussed in Galitz (2007), certain user groups, such as novice users, may have differing interface needs that may affect interface designs for these typical simulation operations, including aids to recognition memory, simplified tasks and vocabulary, and informative feedback.. Measures for evaluation were developed from prior simulation and usability research. Dumas and Redish (1999, p. 184) suggest a combination of quantitative performance measures, such as timed or counted tasks and observations, and qualitative subjective measures, such as ratings, preferences, and commentary. Gutwin and Greenberg (1999, p. 256) used selected measures such as task completion times, perception of effort, overall preference, and strategy evaluation in their study of usability of groupware. The Common Industry Format for Usability Test Reports (National Institute of Standards and Technology, 2001) also outlines accepted reporting formats and suggested metrics. In addition, these NIST guidelines suggest the use of pre-published and validated questionnaires for user satisfaction measurement, including the System Usability Scale (SUS), the After-Scenario Questionnaire (ASQ), or the Post-Study System Usability Questionnaire (PSSUQ) (Brooke, 1996; Lewis, 1993). Suggestions for use of standard questionnaires and related data analysis are also discussed in Tullis and Albert (2008)..
(14) 5 Relevance and Significance. Problem scope Simulation is a widely used technique for complex modeling tasks. Law (2007) lists simulation applications such as manufacturing, computer system design, military applications, inventory systems, and transportation networks. The improvement of simulation interfaces and interface customization are called for in discussions of future simulation systems (Banks, 1997). Banks (1999) asks for future simulation tools to provide end-user interfaces that are focused on the information and tasks the simulation user is responsible for. More recent simulation studies still maintain the need for rapid simulation and model development, through use of a effective user interface that provides for data entry and results analysis (Palaniappan, et al., 2006). Ease of use issues dominate a survey of simulation users regarding desired simulation software features (Hlupic, 2000). Yet characteristics of developed interfaces are often presented with no visible usability consideration (Robinson et al., 2001).. The published research discussing HCI aspects of simulation tends to use general discussions of usability benefits (Kuljis, 1996; Ören & Yilmaz, 2005; Pidd, 1996) or to review specific instances of interfaces designed for a selected task (P. Cohen et al., 1996; Herren, Fink, & Moehle, 1997). In particular, Ören and Yilmaz (2005) provides a rare recent focus on the elements of interactive simulation software, supported by usability quality principles from recognized sources (Mayhew, 1999; Shneiderman, 1998). They outline a set of 21 derived quality principles for simulation software grouped in four areas: usability, communicativeness, reliability, and evolvability. They further.
(15) 6 recommend application of the principles as a systematic approach for evaluation and design of simulation interfaces. This contrasts with the lack of usability support in many specific designs presented. In a less than rigorous approach to usability in simulation interface design Odhabi, Paul, and Macredie (1998) present development of a graphical user interface designed for simulation modeling. While the study recognizes the variety of front-ends used in simulation, from command line interfaces to direct manipulation, the selection of a graphical approach is made without support or experimentation, but simply because such interfaces are generally considered to better support novice users (Odhabi, et al., 1998).. The lack of integration of usability methods from discrete-event simulation development is not unique. A study of the relationship between usability methods and software engineering in general finds high levels of disconnect, claiming that most developers involved in user interface design do not use user-centered design approaches or tools (Seffah & Metzker, 2004). The study also suggests several obstacles that must be addressed in integrating HCI and software engineering, including clear and common definition of usability concepts; integration of usability methods into software development life cycles, address of gaps between specific usability and software engineering practices, development of computer-based usability tools, and provision for education on integrated approaches. In a similar discussion, Redish (2007) calls for expansion of usability testing to support complex systems, such as inventory analysis, resource allocation, health care, and intelligence analysis. Like simulation, such systems place a high burden on the user from the amount of information to consider, onerous data.
(16) 7 analysis and decision-making, difficulty in validation of results, lack of user domain knowledge, and interpretation of visualizations. Redish goes on to suggest expanded usability approaches and research, calling for usability practitioners to be more engaged in addressing such complex domains. Prior examinations There has been limited focused research directly tying HCI considerations to simulation design (Kuljis, 1996; Ören & Yilmaz, 2005; Pidd, 1996). Ören and Yilmaz (2005), the most recent study, is addressed in detail in Table 1 and the accompanying discussion. In a general examination of the interaction of HCI and simulation in several commercial discrete-event simulation systems, Kuljis (1996, p. 689) reviews how HCI aspects impact simulation development time, application consistency, ease of development, model completeness, and model validation. Kuljis, using a structured walkthrough of a typical user’s tasks, found “usability defects” in simulation-specific areas such as data input, user support, and result analysis. Further, it is suggested that the benefits of addressing the usability issues could include reduced development time, increased application consistency, ease of simulation development, and increased model completeness and validation. Kuljis concludes with some suggestions for improvements in commercial simulation tools, including pre-defined problem domains, facilities to create new domains, facilities for graphical representations of elements, and methods to set defaults for values, statistical data collection, and presentation of results. It is also noted that there is a lack of published empirical evidence to support claims that interface improvements will lead to significant impact on simulation use and results..
(17) 8 Pidd (1996, p. 681) points out that development in discrete-event simulation software has generally moved forward “hand-in-hand” with computer software, and simulation packages from vendors have grown in user interface capabilities. However, the issue, as Pidd (p. 684) points out, is not the lack of interface tools, but rather a lack of understanding that the nature of the user interface provided can change the simulation task. Because simulation developers are often not versed in HCI and usability theory, this aspect of simulation design is often neglected. Pidd provides a framework of classification for studying HCI in simulation, including a breakdown of simulation tools, individuals involved (modelers, programmers, project managers, customers, and users), and system features. Finally, Pidd also argues that the tendency of simulation developers to focus on graphics and visualization may distract from the impact of simplification and application of an overall user-centered design approach. A later related article (Pidd & Carvalho, 2006) presents a view of the current state of simulation, arguing that simulation tools must move in the same direction as other computing developments, and suggests a need to focus on component based models for discrete-event simulation.. Barriers and Issues. Work elements There are two major elements to this study – research-based development of a design process and experimentation to test study hypotheses. First, an extensive review of discrete-event simulation characteristics and appropriate usability methods was required. This research included applicable usability literature, such as examinations of usability assessment (Hollingsed & Novick, 2007; Nielsen, 1993), novice programming system.
(18) 9 usability (Galitz, 2007; Pane & Myers, 1996) or user interface elements (Myers, Hudson, & Pausch, 2000; Tidwell, 2011). This information is presented in this study to outline a process to allow simulation practitioners to use the information gathered and is summarized to guide their designs. Second, an experimental approach that both illustrates the application of usability methods and verifies the impact of these methods on simulation usage and solution sets was designed, developed, and deployed, with appropriate analysis of results. This study provided user-based tests with two alternate interfaces to a single simulation problem. A similar approach is used in two example studies, including an evaluation of three alternative interfaces to a database application from Medsker, Christensen, and Song (1995) and a usability evaluation of two alterative interfaces to a groupware application from Gutwin and Greenberg (1999). In this study, through application of a literature-supported user-centered design process with focus on simulation issues, the interfaces were developed to two expected usability levels (designated basic and improved). Live observed user testing of sample sizes appropriate to the study was conducted to ascertain both quantitative and qualitative measures of the impact of the varying interface usability levels on simulation interactions and solution quality.. Difficulty of problem This study has two primary elements, a developmental task and an experimental task. The first developmental task required literature review, synthesis, and summary, which was not inherently difficult, but did require rigorous research and organization in order to develop a well-grounded, publishable guideline as well as to drive the design of the.
(19) 10 experimental portion of the study. The experimental task was more onerous, requiring identification and development of a complex simulation core, as well as design and development of the two alternative illustrative simulation interfaces using the process outlined in the prior task. Thorough and rigorous usability evaluations and extensive user-based testing to determine issues involving simulation use and solution determination followed the development. Finally, a comprehensive results analysis and suggestions for follow-on research concluded the study.. Hypotheses. As previously stated, there are four primary research questions in this study: . What are user interface considerations for discrete-event simulation models?. . What are best practices for designing an interface to a simulation application?. . How is usability best evaluated for simulation interfaces?. . What is the actual effect of increased usability for specific interface elements on simulation operations, such as data input and solution analysis?. The first three research questions are answered through the development task of creating a literature-based process for usability design and evaluation targeted at simulation interfaces. The fourth research question was answered experimentally, to provide quantitative and qualitative evidence that the usability design and evaluation process actually results in the targeted effects of improvement in data input and solution analysis. This experiment also serves to confirm the focus and applicability of the development task results..
(20) 11. The following hypotheses were tested experimentally to provide answers to the fourth research question on actual effect of increased simulation operation usability (note that the hypothesis has been restated in terms of task failure rates instead of task success rates as appeared in the formal study proposal):. H(1). If a simulation-focused usability design process is applied to the data entry aspects of a simulation interface, there will be significant reduction in data entry time, interaction errors, and task failure rates.. H(2). If a simulation-focused usability design process is applied to the results analysis elements of a simulation interface, there will be significant reduction in analysis time, incorrect result reporting, and task failure rates.. H(3). If a simulation-focused usability design process is generally applied to a simulation interface, there will be significant increases in user satisfaction measures, including overall satisfaction, system usefulness, information quality, and interface quality.. The hypothesis discussion, related variables and methodology impact is further discussed in Chapter 3. (The format of the hypothesis discussion is drawn from a recent experimental study of social presence in asynchronous learning (M. S. Cohen & Ellis, 2007) and the previously discussed groupware usability assessment (Gutwin & Greenberg, 1999).).
(21) 12. Assumptions, Limitations, and Delimitations Assumptions . Sufficient computer-literate study participants are available for the simulation study.. . Sufficient skilled usability reviewers are available for usability reviews.. . All study participants will work to the best of their ability.. Limitations . Study participants are not experts in the simulation subject matter.. . The study examines a single type of discrete-event simulation.. . Usability inspection methods are subjective measures.. . There is disagreement about sample sizes appropriate for some methods of usability design and review.. . The simulation experiments gather only selected usability measures: time on task, data entry time, error rates, graded solution outcomes, and user impressions of ease-of-use.. Delimitations . Study participants needed to evidence basic computer literacy (word processing, e-mail use – etc.).. . Study participants had to prove capable of understanding the simulation problem.. . The simulation involved basic tasks, easily explained to novice users.. . Each simulation task experiment (introduction, data entry, solution review, and wrap-up) was limited to less than 30 minutes.. . Participants followed a script for data entry and solution exploration.. . The two simulation interfaces were deliberately designed with interface elements of differing usability levels; this is an artificial step in a normal design process, but was required for the study..
(22) 13 Definitions of Terms . C# - A structured object-oriented programming language developed for the Microsoft .NET platform, sharing similarities with Java and C++ (Liberty & Xie, 2008).. . Class Diagram – Representations of static elements of a system, including structure and interrelationships; depicts logical and physical design of a system (Maksimchuk & Naiburg, 2005).. . Cognitive Dimensions Analysis – Usability evaluation method employing evaluators to assess an interface against a set of 13 defined cognitive interface aspects (Green & Petre, 1996).. . Cognitive Walkthrough – Task-oriented exploration of system functionalities through step-by-step simulation of user behavior and observation of selected cognitive issues (Holzinger, 2005).. . Decision Support – Model-based procedures for support and improvement of decision making; simulation is one form of a decision support tool (Turban, Aronson, Liang, & Sharda, 2007).. . Direct Manipulation – Interface interaction involving visible objects and actions, rapid and reversible incremental actions, and replacement of typed commands with pointing at objects of interest (Shneiderman, Plaisant, Cohen, & Jacobs, 2009).. . Discrete-Event Simulation (DES) – Simulation and modeling of systems where the state variable is changed at a set of points in time (Banks, et al., 2010).. . Heuristic Evaluation – Usability engineering method that employs a small set of evaluators to examine and judge the compliance of a given interface with selected recognized usability principals (Nielsen & Mack, 1994).. . Human-Computer Interaction (HCI) (Also Computer-Human Interaction, CHI) – Interdisciplinary design science that combines experimental psychology datagathering methods and intellectual frameworks with tools developed from computer science (Shneiderman, et al., 2009).. . Mathematica – an interactive computer-based environment with a programming language providing for numerical, symbolic, procedural, and rule-based development; provides internal support for a wide range of graphics, mathematics, and statistical functions (Maeder, 2000)..
(23) 14 . Simulation – Evaluation of a mathematical model of a system through numerical (vs. exact analytic) means that generates data to estimate model characteristics (Law, 2007).. . Unified Modeling Language (UML) – A standardized modeling language made up of graphical notations to express various levels of system designs (Fowler, 2004).. . Usability – The ease-of use and acceptability of systems for selected classes of users and specific tasks in a given environment (Holzinger, 2005).. . Use Case – Modeling approach for business process implemented in a system; describes who interacts with a system, and the ways the system will respond (Maksimchuk & Naiburg, 2005).. Summary This study examined the effects of varying characteristics of user interface designs on the levels of usability and the solution quality of complex, multivariate discrete-event simulations. By selected variation of the usability of test application elements, and measurement of simulation interaction characteristics, the goal of demonstrating measurable effects of interface usability on both simulation use and solution quality was met. Ideally, this may bring more focus on including usability design in simulation development. Additional goals include identifying specific interface aspects that impact simulation, as well as providing an example of the use of usability design processes in a simulation development. For further support of the study, Chapter 2 outlines the literature support for the background, relevance, and approach for the study, followed by Chapter 3, which presents the steps for the methodology employed. The results of the study are presented in Chapter 4, followed by a discussion of conclusions, implications, recommendations, and a study summary in Chapter 5..
(24) 15. Chapter 2 Review of the Literature. Historical Overview The two primary elements examined in this study are usability and simulation, both areas with a long history of research in computer science. Simulation, including specifically discrete-event simulation, has been a part of operations management in manufacturing for over 50 years (Lawrence, 2003). For a historical perspective on simulation, Nance and Sargent (2002) trace the origins of simulation as a methodology for problem analysis. The article states simulation use predates the arrival of computers, with initial uses of a manual method called “artificial sampling” being introduced in 1777 as a method of estimating π. (Known as Buffon’s Needle Problem, the French naturalist Buffon first posed the problem in 1733, and proposed a solution in 1777 involving dropping needles on a grid of parallel lines and using the count of line and needle intersections as an estimator (Weisstein, 2005).). Nance and Sargent describe computer-based methods of continuous and Monte Carlo simulation being introduced during World War II, with the first use of discrete-event simulation in the late 1940s. They further state that as simulation became more tied to computer-based implementations and languages, advances in the methodologies were.
(25) 16 driven by external and internal influences. Internal factors are those derived from simulation research, including advances in modeling, functions, verification and validation, analysis, and theory. External influences come from advances in computer hardware and software, including influences from computer graphics, networks, the World Wide Web. Nance and Sargent also mention HCI techniques and technologies as an external influence. The historical perspective is updated with additional details in a more recent presentation (Goldsman, Nance, & Wilson, 2010). Functional areas being extended in current tools include model re-use, collaborative methods, as well as visual, web-based, parallel and distributed simulation; improvements in the simulation modeling life cycle are seen as increase the overall return and acceptance of simulation as a business practice by reducing effort and increasing value of results (Diamond et al., 2002).. As with simulation, the research in usability and HCI technologies began in earnest after World War II, with the beginnings of human factors and ergonomics. Myers (1998) reviews the history of HCI technologies, with the earliest reference being the idea of linked document references, a precursor of the hypertext concept, as discussed by Vannevar Bush in 1945. Myers continues, discussing the introduction of enabling technologies, such as direct manipulation interfaces, the mouse, and the concept of windows in the 1960s. Usability engineering, and methodological approaches to its use, are introduced in the 1970s (Mayhew, 1999). Mayhew also notes an early reference to a specific usability engineering methodology in 1985, and that texts on usability engineering begin to proliferate in the late 1980s through the 1990s. Nielsen (1993).
(26) 17 introduces the concept of discount usability engineering, with the goal of improving usability with a minimum of necessary tests or testers. Usability design and testing is more commonplace today in mainstream development. At Google, for instance, the focus on user experience is described as being “encoded” into the company’s culture, with usability staff on hand to consult on designs, perform various tests, gather and analyze data, and help with product localization (Au et al., 2008).. One recent study (Wania, Atwood, & McCain, 2006) has attempted to identify the focus of current usability research from analysis of the literature. The study maps current research showing how HCI authors cover topics from theory development to specific application to build usable systems, and from collaboration and group work to specific users and cognition issues. It is further suggested that research is trending toward design and evaluation methods in the context of use. Looking forward for HCI, Shneiderman (2007) discusses the need to expand interaction design and usability methods to enable creativity and exploration. In examples of such tools, mathematical and simulation tools are included. Shneiderman reviews many design aspects of such systems; aid in managing and comparing multiple designs, integration of search engines, and easier backtracking and historical comparison. Shneiderman also looks for an expansion of usability methods, including observation, long term case studies, data logging, and integration of multiple analysis methods to understand usage patterns, as well as continuing research in HCI to refine methods, theories, and study techniques that enable breakthrough designs for discovery and innovation..
(27) 18. Literature Specific to Topic Major focus areas for research The goals of an effective literature review are to understand what is known about a given subject area, to provide a foundation for intended research, to confirm the need for the research, to justify the contribution of the research, and to provide support for the goals and methodologies of the study (Levy & Ellis, 2006). To provide this research-based foundation for this study, the review is divided into two main areas; relevance of the subjects involved to address confirmation and justification, and support for the design of the experiment and assessment methods employed in the study. The areas of relevance to review include the importance of discrete-event simulation and of usability methods, the need for usability focus in simulation, and examples of simulation studies that call for improved interfaces but do not employ usability design or assessment methods. To support the design of the study, the following areas are subject to review: general experimental design guidelines, similar experimental studies, usability design and assessment methods, appropriate visual and user interface elements, interface needs in simulation, simulation development guidelines, and support for selected development tools.. Importance of Discrete-Event Simulation There is ample literature to support the widespread use of discrete-event simulation in various academic, industrial, and other applications. Simulation is presented among other approaches to decision support alongside various static, dynamic, and risk models,.
(28) 19 heuristic programs, and visual and data modeling methods, where it is recommended for problems too complex for more precise numerical optimization approaches (Turban, et al., 2007). Turban et al. also review advantages and disadvantages of simulation. Advantages include well-understood theories and approaches, time compression, ability to pose what-if questions, ability to handle a wide variety of problem types, and the ability to include real complexity through statistical modeling. Disadvantages include the lack of a guaranteed optimal solution, the overhead of the process, and the special skills required to develop.. Standard texts outline simulation methods, the design of models, data distributions, sensitivity analysis and reporting formats (Banks, et al., 2010; Fishman, 2001; Fishwick, 1995; Law, 2007; Ross, 2006). Law (2007) provides an overview of discrete-event simulation and steps for simulation studies, examples of modeling complex systems such as banks and job-shops, reviews of simulation software features, and probability and statistics that apply to simulation. Law also provides details for modeling systems and analyzing results, comparing alternatives and reducing variation, applying experimental designs, and simulating manufacturing systems.. Simulation can be applied to a wide range of problem domains. Specific industrial applications of simulation may include modeling automotive production lines, new manufacturing plant layouts, baggage handling systems, and communications networks (Lawrence, 2003). Fishman (2001) describes inventory systems, distribution systems, transportation networks, and health-care delivery systems as being amenable to discrete-.
(29) 20 event system modeling. Business process models are identified as particularly suitable subjects for discrete-event simulation, for a number of reasons: ease of modification, modeling complete processes, ease of modeling information flow, testing new process designs, capturing human and technical elements, showing dynamic change, and allowing for stochastic elements in designs (Hlupic, 2001). Introductions to and tutorials on simulation, tools, and applications are regularly presented at simulation conferences (Ingalls, 2002; Sanchez, 2006; Schriber & Brunner, 2007, 2010).. Schriber and Brunner provide recent tutorials on discrete-event simulation. software, where such simulations are described in a “transaction-flow world view”, that envisions simulations tracking discrete traffic elements (transactions) moving through a system from one point to another (flow) requesting and using resources. This describes the concept most common to discrete-event simulations, that of a collection of queuing systems. The Schriber and Brunner presentations go on to discuss the objects that make up a simulation, as well an overview of typical model execution, and how simulation is implemented specifically in three typical simulation tools.. Simulation has been an active area of research since the 1960s, but was inhibited by storage limitations, costly processor time, slow development iterations, and lack of textbooks (Nance & Sargent, 2002). Today simulation is widely used in military applications, where a high level architecture (HLA) has been developed to support reuse and interoperation of simulations (Dahmann & Morse, 1998). Simulation tools in industry have been shown to provide growth benefits to organizations employing the methods, including increased project completion, reduced cycle times, and earlier.
(30) 21 identification of wrong initiatives (Miller, Pulgar-Vidal, & Ferrin, 2002). Benefits and barriers to application of simulation in industry are also discussed in McLean and Leong (2001), who also point out that the benefits of simulation are offset by the costs, which include hardware, software, salaries, training, development and maintenance. Statistical support for simulation design can be found in general simulation texts discussed above, as well as in focused statistical distribution modeling guides (Dovich, 1990).. Relevance of Usability Methods Usability is an active research field, with HCI literature and research across several focus areas in usability design and evaluation, looking at theory and applications, as well as group, individual, and cognitive models of usability (Wania, et al., 2006). Standard texts, such as Shneiderman, Plaisant, Cohen and Jacobs (2009), present the wide variety of usability elements, such as theory, process, assessment, testing, tools, graphical environments, and multimedia. Schneiderman et. al. also present four “pillars of design” that outlines the key elements in successful interface development, which include use of user interface requirements, usability guideline documents and processes, user interface software tools, and expert reviews and usability tests all based on a foundation of academic research. There are also a number of usability motivations presented, including the need for usable interfaces to ensure effective life safety systems, to respond to entertainment applications, to enable creative and collaborative tools, to facilitate effective socio-technical systems for large numbers of people, and to reduce cost and increase performance of commercial and industrial tools (Shneiderman, et al., 2009). Significant literature focus is also placed on direct cost-justification of usability in the design process (Bias & Mayhew, 2005; Marcus, 2002). Marcus breaks this down into.
(31) 22 internal return on investment (ROI) such as increased productivity, less errors, and reduced training and support needs as well as external ROI factors such as increased sales, lower cost of customer-side support and training, and making changes to products earlier in design cycles through usability focus.. There are also more practical or applied views of the value of usability practices. In a recent essay, Brooks (2010) makes an argument for the need for explicit user and use models. He argues the need for such models as support for conceptual integrity in developing systems, becoming even more important as complexity increases. Brooks also states even wrong explicit assumptions about use models are better than none, because at least the wrong model will be questioned and examined, as opposed to one that is vague or missing. Krug (2010) also argues that even minimal usability focus has value. In application of his discount assessment methods for web usability, he states the processes work because all interfaces have usability issues, most serious issues are easily found, and directly involving and watching users makes interface developers stronger, as they are no longer designing for an abstract concept of their target user.. Lowgren (1995) looks at various perspectives on usability, including general theory and usability engineering, as well as subjective, flexible, and social aspects. In terms of general theory, Lowgren talks about a causal framework for usability in which the user’s motivation and knowledge combine with the ease of use of the system, the match between the system and the tasks, and the frequency of tasks, to produce a user reaction, which may be positive, resulting in continued use and learning, or negative, resulting in.
(32) 23 reduced or no use. This framework is posited as an approach to experimental definition focused on the user’s reaction. Usability engineering is the general approach for interface development, which Lowgren describes as a three step process, including user and task analysis, development of a usability specification, and iterative prototyping to develop the final interface. The subjective perspective looks at usability as a property of the interaction between a user and the system at a given time, which requires an iterative user-based process of contextual and participatory design. Flexibility in usability refers to extending the participatory design into a long term continuing design effort with tools that responds to changing situations the user may experience. Finally, Lowgren describes a social form of usability, sociality, which encompasses the design of systems for cooperative and collaborative environments. Lowgren suggests these perspectives as a framework for usability research, and also suggests maintaining a view of these different perspectives supports evolution and development of usability approaches.. Need for Usability in Simulation Design Because of the significant human-computer interaction components in simulation data entry, modeling, and analysis, there are regular calls for usability improvements and increased focus on ease-of-use in tools in panel discussions (Banks, 1997, 1999), industry reviews (Umeda & Jones, 1997), and surveys of simulation users (Hlupic, 2000). Hlupic’s survey results of academic and industrial users provide some support for usability enhancement for simulation tools. Significant number of academic respondents (55 total) found their simulation tools lacked flexibility (44%) or were difficult to learn (22%) yet, as Hlupic points out, only 6% cited a poor user interface. Some 59% of.
(33) 24 academic respondents cited software limitations that impeded simulation work. Industrial users also reported flexibility (22%) and learning difficulty (11%) issues, although only 25% reported issues completing simulation work. Hlupic draws the conclusion from the survey as a whole that increased flexibility, ease of use and learning, and features for experimental design and output analysis are key features. More recently, the SIMCHI 2005 (2005 International Conference on Human-Computer Interface Advances for Modeling and Simulation) conference was dedicated to the examination of a variety of topics in how simulation and HCI considerations interact (Ören & Yilmaz, 2005). In practice, extensive simulation is often performed without attention to interface design (Crawley, et al., 2001), yet this is recognized as a deficiency that should be addressed (Clarke, 2001; Papamichael, 1998).. As with simulation, integration of usability into general software development is also recognized as an area of concern; a recent tutorial outlined the challenges of integrating HCI and software development in both terminology and required design approaches (Juristo & Ferre, 2006). Similar to the challenge for a simulation developer, the tutorial states that because software engineering methodologies do not generally include usability concerns, a software developer that wants to integrate usability must consult several HCI books to investigate available methods, and then select a subset of the techniques described that fit the project in question. Holzinger (2005) echoes this need for software developers to be aware of usability methods, and to be able to decide which approaches best fit a given project. Holzinger calls for each software project to consider usability related requirements for learnability, efficiency, memorability (or prevention of re-.
(34) 25 learning), low error rates, and satisfaction, and presents a review of common usability inspection and test methods for design and development.. In applying usability practices to simulation design, it is important to balance usability design methods with other aspects of the overall design process. One recent study found a potential for usability methods to be misapplied, in that novice designers regularly (in approximately 70% of cases) disregarded usability fact-based measures in favor of other pseudo-evidence developed in design activities (Friess, 2008). Friess suggests that documentation of design decisions that include support for why design choices are made would help offset this effect. Friess also suggests that designer intuition may be undervalued in comparison to some formal methods. A similar concern is voiced in Greenberg and Buxton (2008) which suggests that usability tests and designs must be applied carefully, so as not to damage the design process and inhibit creativity and innovation. They suggest several approaches to ensure usability is applied appropriately, including using usability design only when appropriate, using scientific methods that can be replicated, and looking to other disciplines for additional design measures. Buxton (2007) focuses enabling early and iterative exploratory design by using sketch-based designs as a path to usability prototypes. Sketches are suggested to propose and suggest tentative early concepts, vs. prototypes that are intended to depict specific refined interface descriptions for early assessment.. Simulation Interface Aspects and Designs Numerous texts and studies look at the nature of simulations and their interfaces. Interfaces must provide access to core performance measures common to discrete-event.
(35) 26 models. Fishman (2001) outlines four measures: delay, buffer occupancy, throughput, and resource utilization. Delay is the time spent waiting for resources or events. Buffer occupancy describes the queuing of jobs, objects, or individuals as they wait for processing. The number of objects processed in a given amount of time is the achieved throughput, and the amount of time resources are in use describes utilization.. In addition to books and studies on discrete-event system mechanics, there are some discussions focused on simulation interface characteristics, some general (Diamond, et al., 2002), some with examples of applications developed to address specific domainrelated interaction needs (P. Cohen, 1991; P. Cohen, et al., 1996). Cohen (1991) addresses the potential for well designed user interfaces to provide new levels of ease of use for simulation systems. Cohen, using an early graphical user interface combined with natural language processing, attempted to provide ways for simulation-based decision makers to ask general what-if questions about the simulations and data available.. Even today, only a few papers present simulation interface requirements in relation to HCI and usability, either in how HCI concepts might be applied (Pidd, 1996), or in the specific usability concerns of different simulation tasks (Dawson, 2008; Kuljis, 1996; Kuljis & Paul, 2000; Ören & Yilmaz, 2005; Tabachneck-Schijf & Geenen, 2006). Ören and Yilmaz provide one of the most thorough published considerations of usability considerations for simulation. By reviewing key usability principles and applying those to simulation characteristics they propose a set of 21 recommended quality principles for simulation interface design, summarized in Table 1..
(36) 27 Table 1. Usability Principles for Simulation Software from Ören and Yilmaz (2005) Principle area Usability. Principles Least training. As little training required as possible. Minimum memory load. Users should not have to remember information from one interface part to another. Simplicity. Interface should not be distractive; should be uniform, unambiguous, and allow easy navigation. Familiarity. Language, terminology, metaphor, and inputs should be familiar. Separation of concerns. Interface should allow for focusing on simulation tasks. Functionality. Interface should be able to specify, process, analyze, and present results of problems. Communicativeness Restrained realitionship with users. Reliability. Notes. Do not use patronizing or insulting tone. Informativeness. Provide current system knowledge. Perceptiveness. Observe user actions and suggest actions. Explanation ability. Interface should justify decisions and explain results. Aesthetic and cultural acceptance. Information displays consistent with universal and local cultural and aesthetic norms. Access reliability. Control access by authorized users. Predictability. Interface should do what users expect. Consistency. Consistent reaction to user action in different contexts. Safety. Interface supports error tolerance, caution, and robustness. Built-in quality assurance. Filter and prevent (when possible) input and output errors.
(37) 28. Principle area Evolvability. Principles. Notes. Adaptability. Adapt to users with differing skills and preferences. Customizability. Easily tailored interfaces. Learning ability. System should remember usage and enhance user problem solving. Maintainability. System should be easily updated. Portability. Should be portable to different platforms. Another article looks at attempting to prevent knowledge transfer errors in probabilistic decision support systems, such as a discrete-event simulation (Tabachneck-Schijf & Geenen, 2006). In examining the information transfers in such systems, the following representations are presented: knowledge for expert interaction (the development of models), model evaluation (mapping the model to user language), data entry (by the user), dissemination of outcomes (into a user-compatible form), and explanation of outcomes (in user language). They evolve this into a set of heuristics for user-centered representations: preserving precision, user compatibility, natural language, invisible technology, and an efficient application or system. These considerations should shape other simulation interface and interaction heuristics.. More recently, a study looked at what the author termed a holistic usability framework for distributed simulation (Dawson, 2008). Dawson’s investigation has similar goals to this study, but takes a much different approach to simulation usability improvement. Dawson develops a usability framework for distributed simulation development that involves a set of measures for various dimensions of distributed simulation characteristics.
(38) 29 – interfaces, visualization, installation, training, etc. The approach does not measure usability directly from the system, but rather from assessing the attributes that can affect usability measures. The result is a survey that provides input on usability concerns for distributed simulation systems.. It is very common in the simulation literature for studies to recognize a need for usability or to claim the presence of user-friendly interactions, but to then provide interfaces and designs with no application of usability design, test or assessment (Bendre & Sarjoughian, 2005; Chen, Olson, & Morrison, 2002; Hastbacka, Westerlund, & Westerlund, 2007; Heilala, et al., 2007; Herren, et al., 1997; Hewitt & Herrmann, 2003; Kim, Halpin, & Abraham, 2001; Martens & Himmelspach, 2005; Odhabi, et al., 1998; Randell, 2002; Randell & Bolmsjo, 2001; Tebo, Mukherjee, & Onder, 2010; Valentin & Verbraeck, 2002; Verbraeck & Valentin, 2008; Wood & Harger, 2003). Similarly, specific reviews of simulation tools often speak obliquely of usability needs; Gray (2007) provides a review of an object-based discrete-event tool using a list of desirable system features that includes ease-of-use concerns with no reference to usability design or assessment. Another study looks at a template-based discrete-event tool, and discusses issues found during use of the system, including recommended changes, with no reference to formal usability reviews or methods (Grigorov, 2007). There is also an comparison of simulation tools for protein cell signaling that includes usability reviews by the authors, with no formal usability assessment or references (Manninen et al., 2006)..
(39) 30 The concept of using objects, templates, or “plugins” (Himmelspach & Uhrmacher, 2007) to build discrete-event systems is largely an attempt to reduce complexity through abstraction, and could also be supported by applied usability techniques. A similar approach with goals of flexibility and reuse is considered in articles regarding a project describing a building block approach to simulation (Valentin & Verbraeck, 2002; Verbraeck & Valentin, 2008), in which one article (Verbraeck & Valentin, 2008) includes significant discussion of user interface characteristics without addressing the usability of the individual blocks or of the assembled systems.. One parallel area that may help in both justifying and structuring usability processes for simulation is the examination of usability issues with medical decision support systems (Graham et al., 2008; Kushniruk, Borycki, Anderson, & Anderson, 2008). In these studies systems were assessed using think aloud subject-based tests for usability errors that could cause life threatening mistakes (Graham, et al., 2008). The authors suggest usability engineering approaches be used to identify issues early in design cycles to eliminate these serious consequences, including the development of simulated human interaction with the systems to further explore the problem space (Kushniruk, et al., 2008)..
(40) 31 Experimental Design and Similar Studies There are a variety of support materials for research guidelines, experimental design, and other appropriate documentation that helped structure this study. General texts on research methodologies (Bock, 2001; Leedy & Ormrod, 2010) provide suggestions for structuring research problems. Bock provides support for modern scientific studies, including discussion of the scientific method and its components – analysis, hypothesis, synthesis, and validation. The preliminary proposal for this study developed the analysis and hypothesis steps, the execution of the study itself provided synthesis, and the final analysis and report presented here comprises the validation step. Bock also provides guidelines for the design of experiment protocols that outline considerations for laboratory based testing and related test instruments. Leedy and Ormrod provide a more academic focus, including support for planning research study designs, including methods for ensuring internal and external validity. Internal validity refers to eliminating other possible explanations for observed results, where external validity refers to the ability to generalize the results of a study. Leedy and Ormrod highlight the Hawthorne effect as one element of concern in internal validity. A recent study (Macefield, 2007) specifically looks at the Hawthorne effect in usability testing, which suggests that participants in a human-centric study may perform at higher levels because they are aware they are being studied. Macefield reviews the effect and its origins in detail, and suggests ways for usability studies to defend against such issues, including application of verbal protocols, semi-structured interviewing, and elimination (or minimization) of extrinsic performance feedback..
(41) 32 There are also studies similar to the proposed project that lend credence to the approach presented (Gutwin & Greenberg, 1999; Medsker, et al., 1995). Gutwin and Greenberg provided an example of a very thorough study that tested two alternative interfaces to determine the effect of enhancing awareness of other user’s activities for a distributed groupware system. The study employed a complex design where individuals were asked to perform three tasks using first one interface, and then alternate with an interface enhancement. Half of the participants started with the advanced interface, half with the basic interface. The hypothesis was expressed for both the between-participants and within-participants studies. Measures included completion time, perceived effort, verbal efficiency (working in the group), preference, and strategy use. Participants were drawn from a student population, all had used e-mail and web browsers at least once per week, and all had no experience with the problem domain or the system being used. Stopwatches, videotape, and questionnaires were used for data gathering.. Finally, the study uses standard documentation approaches wherever possible. There is significant support for documenting software designs using UML (Unified Modeling Language) notation (Fowler, 2004; Maksimchuk & Naiburg, 2005; Phillips, Kemp, & Kek, 2001), allowing software designs to follow a common graphical representation. There are also standard formats for usability test reports (National Institute of Standards and Technology, 2001) that were reviewed and applied or adapted to support completeness in result reporting..
(42) 33 Usability Design and Assessment Methods There are numerous usability design and assessment methods available that were considered for use in the project and documented in the literature. These include standard usability testing guides for user-based tests (Dumas & Redish, 1999; National Institute of Standards and Technology, 2001; Rubin & Chisnell, 2008) as well as studies employing such usability tests (Weaver et al., 2002). Carter (2007) focuses on user-based testing, and discusses specific approaches to improve the talk aloud method, as well as the proper relationships between the tester and the user that will garner the best results. This is also examined in Molich and Wilson (2008) which discusses the most common issues in preparing, conducting, and concluding a usability test scenario. Among the many problems identified are over-direction by the facilitator, interference by third parties, lack of clear guidelines for facilitator intervention, and lack of post-test debriefing.. Hornebaek and Law (2007) discuss concerns over the correlations between measures gathered in standard usability tests, such as task completion time, error rates, satisfaction, perceived workload, product quality, are reviewed in a study of over 70 usability tests, which concludes there is medium to low correlation between usability measures, but encourages researchers to use standard instruments where possible and to consider possible correlation issues. Another suggested set of metrics, specific to assessment of visual analytics, is also available, which outlines specific measures to answer different types of hypotheses for testing visual systems (Scholtz, 2006). Tullis and Albert (2008) also present standard approaches for gathering, analyzing, and presenting performance.
(43) 34 metrics such as time-on-task, task success, errors, efficiency, and learnability as well as usability issue metrics that involve assessment of severity, frequency of use, business impact, and persistence. Extending this is a proposal for an automated framework for collecting summary statistics and visualization of mouse click events, which is intended to allow instrumentation for gathering usability data without directly programming the application to do so (Bateman, Gutwin, Osgood, & McCalla, 2009). A similar proposed effort at automating user logs for usability assessment, based on a data model that relates components, inputs, and tasks, has also been presented (Babaian, Lucas, & Topi, 2007).. There are also discussions of specific usability evaluation methods such as heuristic evaluations (Chattratichart & Lindgaard, 2008; Mankoff et al., 2003; Nielsen, 1993; Nielsen & Mack, 1994), cognitive dimensions assessments (Green & Petre, 1996), and cognitive walkthrough-based approaches (Green et al., 2000; Karoulis, Demetriades, & Pombortsis, 2000). Each source on evaluation generally weighs the strengths and weaknesses of the usability assessment technique presented, and this is supplemented by overall reviews and surveys (Ivory & Hearst, 2001; John & Marks, 1996) as well as specific comparison studies (Englefield, 2003), method summaries (Axup, 2002), and best practice discussions (Holzinger, 2005). Hollingsed and Novick (2007) review literature for usability inspection methods over 15 years of use, and evaluate the effectiveness of heuristic evaluations, cognitive walkthroughs, pluralistic usability walkthroughs, and formal usability inspections. Heuristic evaluations and cognitive walkthroughs are found to be in common use as inspection methods, with developers using the method appropriate to a given project. The conclusion is that inspection alone.
(44) 35 cannot provide a full assessment, and must be used with user-based tests to provide full defect exposure. Another study of variations in heuristics models presents a framework for comparison of heuristics including reliability, validity, effectiveness, and reliability, and also looks at performance of the heuristics when used by novices vs. experts (Chattratichart & Lindgaard, 2008).. An experimental application of the evaluation methods also reviews strengths and weaknesses of the approaches (Karoulis, Valsamidou, Demetriadis, & Timcenko, 2005). Karoulis et al. (2005) discusses the advantages and disadvantages of evaluation vs. userbased testing. Evaluation advantages include early application, easy preparation and performance, good assessment of problem severity, and high effectiveness for low cost; disadvantages include not finding all problems, requiring experienced evaluators, losing sight of user concerns, and difficulty in proposing solutions. User-based testing can find problems real users encounter, can find most issues, and is efficient for complex interfaces; it is however expensive and difficult, requires numerous representative users, subject bias is a concern, and it requires some level of product completion. (It should be noted that other literature suggests a combination of these approaches for an overall usability assessment (Holzinger, 2005; Usability Professionals' Association, 2000)) Karoulis et al. also includes an experimental application of cognitive walkthrough and heuristic evaluation; both methods provided good results, but the heuristic evaluation was more easily applied..
(45) 36 Significant portions of these sources and others (Krug, 2006, 2010; Medlock, Wixon, Terrano, Romero, & Fulton, 2002) discuss the effects of testing with low numbers of users, known as discount assessments. Medlock et al. provides a review of the literature related to sample size in usability evaluations, stating that 4 to 5 participants will uncover approximately 80% of detectable issues (a likelihood of detection greater than .31). If problems have a higher likelihood of detection (5 or greater), three participants will find 87.5% of issues. Krug (2006) suggests that the addition of more test subjects has significantly diminishing returns after the 4th or 5th participant. In all cases, where discount methods with low numbers of participants are used, it is recommended to iterate tests to ensure coverage.. There is also a range of literature sources that discuss whole usability-focused design methodologies, from gathering requirements through product deployment. Moggridge (2007) presents a series of studies of and interviews with interaction designers, and then describes a suggested process for prototyping “screen-based experiences”; three steps that includes low fidelity paper prototypes, high fidelity computer-based prototypes, and user testing with final prototypes. The approach in this three stage process is echoed in other process methodology studies (Hackos & Redish, 1998; Usability Professionals' Association, 2000). Other studies of similar overall process methodologies are provided in some sources (Constantine & Lockwood, 1999; Cooper, Reimann, & Cronin, 2007; Mayhew, 1999), in others there is more specific focus parts of the process: use case development (Ambler, 2005), sketch-based designs (Buxton, 2007), paper prototyping (Snyder, 2003), or process best practices (Bailey, 2005). Given the wide range of.
(46) 37 suggested processes, a study of user-centered design processes found that not all aspects of system development are covered by each approach in the literature, and that evaluation of the applicability of individual usability design practices is needed for individual projects (Iivari & Iivari, 2006). There are also efforts to automate some of these practices, such as a proposed automation of the paper prototyping process (Li, Cao, Everitt, Dixon, & Landay, 2010).. Mirel (2004) provides a discussion of interaction modeling and usability design approaches focused on complex problem solving applications. Mirel describes the contextual influences in understanding the problem solving work space, which has four components: the problem, the work domain, technology and data, and subjective elements. The problem describes the severity and nature of the task – its type, trigger events, and inquiry patterns. The work domain looks at the surrounding influences of roles, environment, and external pressures. The technology and data describe the infrastructure for the work, such as databases, software tools, and information sources. Finally the subjective elements include the cognitive abilities of the users and their preferences, skills, and motivation. Mirel states that interaction designers must address this full context space for a successful outcome. Further, in looking at the actions complex problem solving must address, Mirel identifies three core activities – data ordeals, using large volumes of multidimensional data; wayfinding, working through complex analysis and exploration; and sensemaking, processing data to draw relationships and develop meaning. In addition, for a complete address of a complex problem, the designer must consider what Mirel calls mainline and enabling tasks, the.
(47) 38 basic procedures involved in the work, as well as the patterns of inquiry that are used to solve problems.. The Mirel text is also expanded on in Redish (2007) which agrees with the need for focus on usability for complex systems. Redish states that the main point of the Mirel text is that usefulness is as important as usability in complex systems, and that the product developed must match the actual work and requirements. Redish looks at the aspects of complex systems that differ from normal subjects of usability tests, special considerations, and what should be and has been done to support complex system usability development. To facilitate usability tests for complex systems, Redish suggests a number of approaches, including use of usability studies outside of laboratories, possibly at conferences where developers and domain experts are present; building simulations of tasks; development of situational awareness assessment; and automating long-term use data capture.. Hilbert and Redmiles (2000) describes in detail the theory. and application of automated extraction of usability information from user interface events. Suggested metrics for capture include performance time, mouse travel, command frequency, command pair frequency, cancel and undo use, and physical device swapping.. Inspired by the Mirel text, Albers (2004) also looks at complex system issues, and concludes that the focus is presenting the right information in the right way at the right time, and that complex system designers must ensure content is communicated to users in a way that justifies the cost of complex system development, ensuring users understand where information can be found and when it is needed. In an article in a follow on.
Related documents