Task-Centric User Interfaces

(1)

Task-Centric User Interfaces

by

Benjamin J. Lafreniere

A thesis

presented to the University of Waterloo in fulfillment of the

thesis requirement for the degree of Doctor of Philosophy

in

Computer Science

Waterloo, Ontario, Canada, 2014

(2)

Author’s Declaration

I hereby declare that I am the sole author of this thesis. This is a true copy of the thesis, including any required final revisions, as accepted by my examiners.

(3)

Abstract

Software applications for design and creation typically contain hundreds or thousands of commands, which collectively give users enormous expressive power. Unfortunately, rich feature sets also take a toll on usability. Current interfaces to feature-rich software address this dilemma by adopting menus, toolbars, and other hierarchical schemes to organize functionality—approaches that enable efficient navigation to specific commands and features, but do little to reveal how to perform unfamiliar tasks.

We present an alternative task-centric user interface design that explicitly supports users in performing unfamiliar tasks. A task-centric interface is able to quickly adapt itself to the user’s intended goal, presenting relevant functionality and required procedures in task-specific customized interfaces. To achieve this, task-centric interfaces (1) represent tasks as first-class objects in the interface; (2) allow the user to declare their intended goal (or infer it from the user’s actions); (3) restructure the interface to provide step-by-step scaffolding for the current goal; and (4) provide additional knowledge and guidance within the application’s interface.

Our inspiration for task-centric interfaces comes from a study we conducted, which revealed that a valid use case for feature-rich software is to perform short, targeted tasks that use a small fraction of the application’s full functionality. Task-centric interfaces provide explicit support for this use.

We developed and tested our task-centric interface approach by creating AdaptableGIMP, a modified version of the GIMP image editor, and Workflows, an iteration on AdaptableGIMP’s design based on insights from a semi-structured interview study and a think-aloud study.

Based on a two-session study of Workflows, we show that task-centric interfaces can successfully support a guided-and-constrained problem solving strategy for performing unfamiliar tasks, which enables faster task completion and reduced cognitive load as compared to current practices.

We also provide evidence that task-centric interfaces can enable a higher-level form of application learning, in which the user associates tasks with relevant keywords, as opposed to low-level

commands and procedures. This keyword learning has potential benefits for memorability, because the keywords themselves are descriptive of the task being learned, and scalability, because a few keywords can map to an arbitrarily complex set of commands and procedures.

Finally, our findings suggest a range of different ways that the idea of task-centric interfaces could be further developed.

(4)

Acknowledgements

I feel incredibly lucky to have joined the Human-Computer Interaction lab only three years after its inception, and to have been a part of its growth into the thriving lab it is today. In particular, I want to thank Michael Terry, Edward Lank, Daniel Vogel, and the many other lab members who have together created an ideal place to pursue a PhD.

I want to thank Mike for his guidance and supervision, and for being a tireless champion for myself and my research. I can’t think of a better mentor and role model to have had these past years.

I want to thank my committee members: Edward Lank, Daniel Vogel, and Parmit Chilana for providing invaluable advice and guidance, and my external examiner Carl Gutwin for asking tough questions and providing inspiration for future research.

All of my coauthors and collaborators have my sincere thanks. In particular, I want to thank Andrea Bunt, who was a collaborator on all of the projects reported in this dissertation, and has also been a constant source of encouragement and good advice. In addition to Mike and Andrea, I want to thank everyone who contributed the AdaptableGIMP project, including Adam Fourney, Filip Krynicki, Leslie Ng, Matthew Lount, and Jordan Stinson.

In the HCI lab there are more people than I can easily name that have made these past years a memorable and enjoyable experience. I want to thank Adam Fourney, who has always been up for discussing research, and without fail has provided good suggestions; and Matthew Kay, who graduated soon after I started, but has had a big influence on my direction in HCI research.

I want to acknowledge the tireless efforts of the department and university’s administrative staff, and Wendy Rush in particular, for helping me with booking rooms, filing reimbursements, and the many other individually small but collectively huge tasks it takes to complete a PhD.

Outside of the university, I owe a debt of gratitude to the great people at Autodesk Research, where I did two summer internships. In particular, I want to thank Tovi Grossman, Justin Matejka, and George Fitzmaurice for giving me an alternative perspective on how to do research.

Finally, I want to thank my family and friends, who have always been supportive of me. I particularly want to thank Valerie Busch, who has been a pillar of support when things got the craziest, and Max (the cat) who directly oversaw the writing of most of this thesis from a chair beside mine (though he was asleep most of the time).

(5)

Over the course of my PhD I received financial support from the Natural Sciences and Engineering Research Council of Canada (NSERC), the Graphics, Animation and New Media Network of Centres of Excellence (GRAND NCE), and the University of Waterloo, to all of whom I am grateful.

Parts of Chapter 3 of this dissertation have been published in the proceedings of the Canadian Information Processing Society’s Graphics Interface (GI) 2010 conference.

(6)

List of Figures

Figure 1. The basic components of a task-centric interface in Workflows. ... 4

Figure 2. Research path showing research problems, activities, and main results. ... 11

Figure 3. Histogram showing active usage time for sessions ... 29

Figure 4. Histogram showing the number of command invocations per session ... 31

Figure 5. Conceptual comparison of task-centric interfaces to existing personalization approaches .. 42

Figure 6. The Ribbon interface from Microsoft PowerPoint 2007. ... 43

Figure 7. Conceptual comparison of task-centric interfaces with Ribbon and Perspectives. ... 44

Figure 8. Design space of task-support and learning techniques... 46

Figure 9. The task-centric interface provided by AdaptableGIMP. ... 49

Figure 10. Task set documentation on the AdaptableGIMP wiki ... 50

Figure 11. The basic components of a task-centric interface in Workflows. ... 62

Figure 12. Comparison of AdaptableGIMP and Workflows. ... 64

Figure 13. Before/after images for the four study tasks performed by participants. ... 70

Figure 14. Average per-participant task times... 75

Figure 15. Average cognitive load for the six axes of the NASA TLX by condition and session. ... 76

Figure 16. Self-rated preferences for the two study conditions.. ... 87

Figure 17. Creating a task set in AdaptableGIMP. ... 117

(12)

List of Tables

Table 1. The number of observed commands shared by different proportions of users. ... 32

Table 2. Task sets for the seven clusters of sessions... 34

Table 3. Frequently occurring activity tag keywords for each cluster ... 36

Table 4. Participant responses to 7-point Likert-scale statements ... 54

Table 5. Templates used to create workflows based on procedures found in web tutorials. ... 72

Table 6. Research questions and corresponding measures. ... 73

Table 7. Summary of qualitative codes for the two study conditions. ... 78

(13)

Chapter 1 Introduction

Feature-rich software applications for design and creation are an essential part of modern computing. Adobe Photoshop, with over five-hundred menu items, is the de-facto standard tool of the graphic design industry; Microsoft Word, with over a thousand commands and options, is a near-universal fixture across the business world; and Autodesk’s AutoCAD, with over a thousand commands, is used to design everything from teapots to skyscrapers. In a very literal way, these applications shape the world that we live in.

The feature-richness of these applications is a driver of their enduring success. A feature set consisting of hundreds or thousands of commands supports an exponentially larger set of tasks— activities undertaken to achieve specific desired goals. Supporting a wide range of related tasks in one place gives these applications enormous expressive power and broad appeal. Large feature sets also have potential advantages for usability: a given tool can be used for many different tasks, and the software’s interface can establish conventions that are re-used throughout the interface, which allows a user to transfer their knowledge across related tasks.

Unfortunately, rich feature sets also add significant complexity to an application’s interface, and this takes a toll on usability. Previous work has shown that users respond to feature-rich applications by learning a small, and personal, subset of functionality that is relevant to their individual needs (Draper 1984; Greenberg 1993; Linton et al. 2000; Matejka et al. 2009; Sutcliffe and Old 1987), and that once they have learned such a subset, they are hesitant to spend additional time on concerns, such as learning, that are unrelated to their primary tasks (Carroll and Rosson 1987; Carroll 1990; Mackay 1991; Rieman 1996). Irrespective of expertise, many users find feature-rich applications to be overwhelming, frustrating, and difficult to navigate (McGrenere and Moore 2000), and excess interface complexity can have a negative impact on performance, particularly for novice users (Cockburn, Gutwin, and Greenberg 2007).

Our view is that these disadvantages are not innate traits of feature-rich software, but are a result of current feature-centric approaches to organizing functionality, in which similar commands are grouped into menus, submenus, toolbars, palettes, and tabs. This organizational scheme allows the interface to fit a large number of commands, and aids in navigating to specific commands, but it does not reveal how to perform unfamiliar tasks, which may require commands found scattered throughout

(14)

the application. As a result, the user wishing to perform an unfamiliar task is forced to (1) identify relevant functionality by inefficiently hunting through the application’s interface, and (2) determine the required procedures to reach their intended goal through error-prone trial-and-error and

experimentation. In principle, the user could turn to external help resources for assistance, but past work has shown that users are typically hesitant to do so (Carroll and Rosson 1987; Rieman 1996; Andrade, Bean, and Novick 2009; Rettig 1991).

In this dissertation, we introduce the notion of task-centric user interfaces for feature-rich software that adapt “on the fly” to support specific tasks. By doing so, a task-centric interface can provide explicit support and guidance within the application for performing unfamiliar tasks.

In this chapter, we first describe task-centric interfaces and the prototype application that we have developed to investigate this idea. We then present a high-level summary of the research questions we set out to answer, and the contributions that we make in this dissertation. Next, we step back and define the scope of our investigation in greater detail, and provide a chapter-by-chapter summary of the research to be presented in this dissertation.

1.1 _{Task-Centric Interfaces}

A task-centric user interface is able to quickly adapt itself to support the user in performing specific intended tasks. By adapting the interface to the user’s intended task, the application reveals the relevant functionality and the required procedures to reach the desired goal in an interface that can be directly interacted with to perform the required actions.

In this dissertation, we introduce the concept of a task-centric interface and define four important properties of this style of user interface. Specifically, task-centric interfaces:

• Represent tasks as first-class objects that associate a task goal with relevant commands and functionality, and the procedures required to reach the goal;

• Can be made aware of the user’s intended task, either through explicit task declaration by the user, or implicit task inference by the system;

• Include a task adaptation mechanism that restructures the interface to provide explicit, step-by-step scaffolding for task completion; and

• Provide task knowledge in the interface, to help users understand how commands and features relate to achieving specific, meaningful goals in the application.

(15)

The choice of specific means to satisfying these four properties creates a design space for task-centric user interfaces.

Like adaptable and adaptive user interfaces, task-centric interfaces use customization to provide the user with a simplified interface. Unlike these approaches, the focus is on using customization to assist the user in completing specific tasks. One way of thinking of task-centric interfaces is that they take the idea behind the Ribbon interface in Microsoft Office, or Perspectives in the Eclipse IDE, to a logical extreme; instead of using customization to support large classes of related tasks, task-centric interfaces use customization to provide step-by-step scaffolding for achieving individual, specific goals in an application.

To investigate the potential impact of a task-centric interface design, we designed and evaluated AdaptableGIMP (Lafreniere et al. 2011) and Workflows, two modified versions of the GNU Image Manipulation Program (GIMP).

AdaptableGIMP provided a task-centric interface, and additionally sought to address how a community of users of an application could collectively create a body of task-centric knowledge around an application. Workflows was created as an iteration on AdaptableGIMP’s design, to further investigate the design of task-centric interfaces.

1.2 _{Workflows – A Prototype Task-Centric Interface}

Workflows supports an interaction in which the user enters keywords describing their intended goal (e.g. “black and white with color”), and the application responds with interface customizations that match the declared task (Figure 1). These interface customizations, which we call workflows, include step-by-step instructions for how to complete the task, embedded with actionable references to commands—the buttons can be clicked to directly invoke commands from within the workflow itself, without having to locate them in the full interface. This supports a problem-solving strategy for unfamiliar task that starts with declaring the intended task, and then using the commands and guidance provided in the resultant streamlined interface to perform the task.

(16)

Figure 1. The basic components of a task-centric interface in Workflows.

Using keyword search as a task-declaration mechanism allows the user to express their intended goal in their own terminology, with the system producing workflows that it believes to be most relevant. Keyword search also naturally handles providing access to a large number of workflows, which is important because the space of potential tasks that a user might wish to perform could be very large.

Task adaptation occurs when a workflow is loaded in response to a search. Because the commands included in the workflow can be directly invoked, the workflow acts as a task-specific customization of the application’s interface, organizing its low-level functionality for achieving a particular goal. At

(17)

the same time, the workflow provides step-by-step guidance on how to use the included commands to reach that goal.

The design of Workflows was developed over a number of iterations, starting with AdaptableGIMP, an earlier task-centric interface prototype that we developed.

1.3 _{Thesis Statement and Research Questions}

Our investigation of task-centric interfaces is structured around the following thesis statement: A task-centric interface is able to assist the user with performing unfamiliar tasks by providing explicit support for locating relevant functionality, and determining the procedures required to reach an intended goal. The expected advantages of a task-centric interface are improved task performance, reduced cognitive load, and enabling the user to learn and use the application at the higher conceptual level of tasks, instead of individual commands and features.

To defend this thesis statement, the work presented in this dissertation addresses the following research questions:

• How are users currently using feature-rich software?

• How does a task-centric interface affect the problem-solving strategies used to complete unfamiliar tasks in feature-rich software?

• How does a task-centric interface affect performance measures such as task time and cognitive load when performing unfamiliar tasks?

• What are the relative advantages and disadvantages of a task-centric interface over existing feature-centric interfaces for feature-rich software?

• How do task-centric interfaces change learning in feature-rich software? • What are users’ subjective reactions to a task-centric interface design? • What are the important components of a task-centric interface design?

Next, we provide a summary of the specific research contributions that we have made through the work presented in this dissertation. Later in this chapter we summarize the specific research activities we carried out to answer these questions.

(18)

1.4 _{Research Contributions}

In this dissertation, we make the following research contributions:

1. Based on a study of two-years’ worth of usage data gathered from a publicly-deployed, instrumented image editor, we provide evidence that a valid use case for feature-rich software is to perform short, targeted tasks using a small fraction of the available functionality.

2. We also replicate previous findings that individual users’ command vocabularies tend to be small and idiosyncratic, and that command frequencies follow a long-tailed distribution, when considered across a community of users.

3. We argue that current feature-centric paradigms for organizing functionality in feature-rich software are poorly suited to the type of use that we have identified above, because they do not support users in performing unfamiliar tasks.

4. To explicitly support this scenario of use, we present the concept of a task-centric interface— an alternative interface paradigm for feature-rich software that is able to quickly adapt itself as needed to support the user in performing specific tasks.

5. We present AdaptableGIMP, an initial task-centric interface design that we developed. Based on two user studies of AdaptableGIMP, we present evidence that users appreciate the features of a task-centric interface, and derive insights into how the design can be improved.

6. We present Workflows, an iteration on the task-centric interface design in AdaptableGIMP, and the main task-centric interface prototype that we have developed to evaluate the potential of task-centric interfaces.

7. Based on the results of a laboratory study held over two sessions, we find evidence that a task-centric interface design can provide advantages over current practices for using feature-rich software to perform unfamiliar tasks. In particular, we present evidence that:

a. A task-centric interface design can enable users to complete tasks significantly faster and with reduced cognitive load, both when performing an unfamiliar task for the first time, and re-performing that task at least two weeks later.

b. Users express a subjective preference for the task-centric interface design over current practices.

(19)

8. We present evidence for three main mechanisms by which task-centric interfaces provide the benefits listed above:

a. The task adaptation provided by the task-centric interface helps users to locate relevant functionality for a task.

b. The task knowledge provided by the task-centric interface helps users to form a plan for how to achieve their intended goal.

c. By explicitly supporting the process of performing an unfamiliar task, the task-centric interface reduces self-directed exploration of the interface, which is a common source of frustration and errors.

9. We present evidence that a task declaration mechanism based on keyword search allows users to engage in keyword learning to efficiently return to task-centric help resources. This

enables the user to learn the interface at a higher conceptual level of tasks, rather than learning lower-level functionality and procedures. Keyword learning also has potential advantages in terms of scalability, because a few keywords can map to an arbitrarily complex set of commands and procedure, and memorability, because the keywords themselves are descriptive of the task being learned.

10. Based on the results of all of the studies described above, we outline specific areas for future work to continue to develop the idea of task-centric interfaces.

Having presented a high-level summary of the contributions of the work, we now step back and describe the particular problem setting and scope of our investigation, and then give an outline of the specific research activities presented in this dissertation.

1.5 _{Research Scope}

We start by defining the scope of our work in this dissertation, including the particular problem, user group, scenario of use, and type of task that we are focusing on in this work. We also discuss the particular independent variable that we are manipulating in our investigation.

1.5.1 Non-Expert, Occasional Users

Our goal is to assist non-expert, occasional users of feature-rich software. This user group is characterized by the following attributes:

(20)

• They do not use the application frequently (e.g. daily or weekly), though they may still use it regularly (e.g. on a monthly basis).

• They may have a weak or incomplete understanding of the commands and features available in the application.

• They may not have a strong understanding of the vocabulary and terminology surrounding the application and the application’s domain.

• They are primarily interested in completing their primary task, with learning and expanding their skill set as secondary considerations.

In Chapter 3, we provide evidence to suggest that occasional use of feature-rich software by non-expert users is a common practice.

1.5.2 Unfamiliar Tasks

Our scenario of use is the user performing an unfamiliar task. This could be a new task that the user is performing for the first time or an occasional task that the user does not perform regularly enough to remember the full details of. In Chapter 3 we present evidence to suggest that this is a common scenario for certain feature-rich software applications.

Within this scenario, the specific problem that we are seeking to address is what Norman referred to as the “gulf of execution”, or the gap between a user’s goal for action and the means to execute that goal (D. A. Norman and Draper 1986).

1.5.3 Tasks Involving Ill-Defined Problem Solving

We focus on supporting users in performing tasks that involve an element of ill-defined problem solving. Ill-defined problems require a human to judge the results of individual operations, and then adjust their actions accordingly (Schön 1983). For example, selective colorization (retaining color in some parts of an image, while converting the rest to greyscale) can be considered to be an ill-defined task because the user must be “in the loop” to select the portion of the image to remain colored.

This type of task is common in image editing applications, where users often engage in creative activities that have a subjective or aesthetic element to the desired outcome. However, ill-defined tasks are not limited to this area. For example, exploratory data analysis also includes a strong

(21)

element of ill-defined problem solving, because though there are established processes for drawing insights out of data, human judgment is required throughout the process.

1.5.3.1 Relationship to Automation

A trend in feature-rich software applications has been to create automatic tools to perform tasks that once required human judgment. For example, Photoshop CS5 introduced several “content-aware” versions of fill and brushing tools that analyze the contents of an image to intelligently remove unwanted objects or regions within the image, tasks that once required human judgment throughout the process. Numerous techniques have also been investigated to completely automate tasks using macros (Bergman et al. 2005; Berthouzoz et al. 2011), machine learning (Berthouzoz et al. 2011; Grabler et al. 2009b; Yeh, Chang, and Miller 2009), mechanisms to automatically personalize scripts for the current user (Leshed et al. 2008), or the crowd (human workers) (Bernstein et al. 2010).

There are two points to make on the relationship between ill-defined tasks and task automation. First, there are certain tasks that, by their nature, cannot be automated because the desired result is too dependent on the aesthetic judgment of the user. Second, as more tasks become automated through macros, scripts, and higher-level commands, these become component operations for even higher level ill-defined tasks. As a result, it’s possible that task automation will lead to users spend a larger proportion of time on ill-defined tasks, as automatable tasks become “one click” operations.

Considering these points, task automation is not at odds with techniques to support ill-defined tasks, and developments in these two areas have the potential to benefit one another.

1.5.4 Evaluating an Alternative Problem-Solving Process

Our intervention to support the scenario outlined above (non-expert users performing unfamiliar, ill-defined tasks) is to introduce a new kind of interface designed to support an alternative problem-solving process to that currently supported in feature-rich software.

As a result, the goal of our evaluation is both to show that task-centric interfaces successfully support an alternative problem-solving process, and to demonstrate that this provides benefits over current practices. To achieve this, we employ a mix of quantitative and qualitative approaches to evaluate the impact of our task-centric interface design.

(22)

1.5.5_{Simplifying Assumptions}

To focus on investigating the potential of task-centric interfaces, we make a few additional simplifying assumptions in this dissertation.

First, we focus on investigating task-centric interfaces from the perspective of the end-user, leaving the question of how to create a suitable corpus of task-specific knowledge to future work. In Chapter 9, we discuss some potential ideas for how this could be accomplished.

Second, in AdaptableGIMP and Workflows we use keyword search as a “black box” component, assuming that off-the-shelf techniques can be used to match a user’s query with appropriate resources.

Finally, in our study of Workflows, we confine our investigation to a scenario in which a user wishes to perform an unfamiliar task for which a single appropriate workflow is available in the system. This is a logical place to start, because demonstrating benefits in this scenario is a precondition for the task-centric interfaces approach working more generally. In Chapter 9, we discuss how our investigation could be extended to other scenarios.

1.6 _{Summary and Outline}

In this section we give an overview of the research activities presented in this dissertation, and how they fit together to accomplish our research objectives. A diagram of the path of questions and research activities taken in this work is shown in Figure 2.

(23)

Figure 2. Research path showing research problems, activities, and main results. Bold text indicates research questions or activities. Italics indicate main results.

To start, the work presented in this dissertation builds upon work in a number of related research areas, which we outline in Chapter 2.

In Chapter 3, we present an analysis of usage data gathered over a 2-year period using ingimp, an instrumented version of the GIMP image editor. This provided an ecologically valid window into how feature-rich image editing software is used “in the wild” by actual users.

In Chapter 4, we present a detailed introduction to the idea of a task-centric interface. We describe the properties of a task-centric interface, and contrast it with related techniques that have been explored in research and in existing feature-rich software.

(24)

In Chapters 5 and 6 we present the design phase of the project. In Chapter 5 we present AdaptableGIMP, which included an initial iteration of our task-centric interface design, and also explored the idea of task-centric community customization, or the idea of providing a set of tools to allow a community of users to build up a body of task-centric knowledge to meet their needs. In this chapter, we focus on the task-centric interface component of AdaptableGIMP, and present a semi-structured interview study and a think-aloud study. The results of these studies prompted us to iterate on the task-centric interface design provided by AdaptableGIMP to create Workflows, which we describe in detail in Chapter 6.

In Chapter 7 we present a laboratory study that we conducted to understand how the task-centric interface provided by Workflows impacts the strategies employed by users to perform unfamiliar tasks. In particular, we compared use of Workflows with an unmodified version of GIMP and a web browser, representing current practices for completing unfamiliar tasks in feature-rich software. Participants performed the same tasks in two sessions separated by at least two weeks, allowing us to test the impact of our design for tasks that the user had never performed before, and for occasionally performed tasks for which the user has some familiarity. This also allowed us to gain insights into what users learned in the first study session.

In Chapter 8, we discuss important findings from our study of Workflows and our large-scale study of ingimp in greater detail. In particular, we tie the results of our study of Workflows back to the conceptual design of task-centric interfaces, discuss the impact of task-centric interfaces on learning, and highlight the potential of keyword learning to enable learning of feature-rich software at a higher conceptual level than individual commands. We also discuss how task-centric interfaces could lead to radically different interfaces for assisting users with performing short, targeted tasks.

Finally, in Chapter 9 we provide a set of high level conclusions for the research described in this dissertation, and outline specific directions for future work, concluding our discussion of task-centric interfaces and the main content of this dissertation.

In Appendix A, we present some additional information the AdaptableGIMP project. In particular, we discuss the idea of task-centric community customization, and present additional results from the semi-structured interview study from Chapter 5, and findings from a ten-month public deployment of the system.

(25)

In Appendix B, we present study materials that we developed for use in the study of Workflows presented in Chapter 7.

1.7 _Terminology

Before we continue, we briefly define a number of terms that are frequently used throughout this dissertation.

task – We use the term “task” to refer to an activity to be undertaken to achieve a desired goal. This is similar to the definition of task used in the task modeling literature (e.g. see the introduction to (Mori, Paternò, and Santoro 2002)), however in this work we frequently discuss tasks with the understanding that the user does not currently know how to reach the desired goal.

goal – The task modeling literature refers to goals as “either a desired modification of state or an inquiry to obtain information on the current state.” (Mori, Paternò, and Santoro 2002). Our use of “task” tends toward the former of these definitions, with the desired modification of state typically being in a document or the state of an application.

commands and tools – We use the term “command” to refer to discrete capabilities of an application that are available to the user, typically presented through toolbars, toolboxes, and menus. In places, we substitute the term “tools” for commands when referring to capabilities that have a modal component (e.g. the Paintbrush tool in an image editing application, which can be selected once and then invoked multiple times).

features (of software) and functionality – We also use the terms “features” and “functionality” to refer to the capabilities that the application makes available to the user. In comparison to the terms

“commands” and “tools”, these terms are more expansive, additionally encompassing settings, options, dialogs, and other capabilities of the system.

(26)

Chapter 2 Background Literature

The ultimate goal of this work is to better support individuals as they perform unfamiliar tasks in feature-rich software. In this chapter we review what is currently known about how users respond to feature-rich interfaces, and present existing techniques that have been proposed to assist users of these applications.

2.1 _{Learning and Using Feature-Rich Software}

A number of projects have been undertaken to understand how individuals use and respond to rich feature sets in software. The work we review in this section serves to motivate our approach, and also establishes a backdrop of what is currently known about how users currently learn and use feature-rich software.

2.1.1 Large-Scale Analyses of Use of Feature-Rich Software

We start by reviewing work that has investigated use of feature-rich software at a macro-level, using large-scale analyses of log data. Macro-level analyses of feature-rich software are able to characterize population-wide trends that may not be apparent in small-scale studies, and also have high ecological validity as compared to laboratory studies.

Large scale analyses of feature-rich software has examined use of the UNIX command line

(Greenberg 1993; Draper 1984; Hanson, Kraut, and Farber 1984), text editing applications (Whiteside et al. 1982; Cook et al. 1995; Linton, Joy, and Schaefer 1999; Kay and Thomas 1995), and software development environments (Murphy, Kersten, and Findlater 2006).

A robust finding from these studies is that users’ command vocabularies—the set of unique commands used by a user—are relatively small as compared to the set of available commands, and also tend to idiosyncratic, with low overlap between users (Greenberg 1993; Cook et al. 1995; Linton et al. 2000).

Another robust finding is that the frequency with which individual commands are used, across the entire community of users, follows a long-tailed distribution in which a very small number of commands accounts for the vast majority of observed command use, with the remaining commands

(27)

used by a small percentage of the user population each (Cook et al. 1995; Greenberg 1993; Kay and Thomas 1995; Cook et al. 1995; Linton, Joy, and Schaefer 1999; Whiteside et al. 1982; Draper 1984).

We note two holes in existing large-scale studies of feature-rich software. First, existing work has focused on domains where text is the main unit of concern (the UNIX command line, text editors, and IDEs). In particular, we are unaware of any work that has investigated large-scale use of applications that follow a primarily direct-manipulation paradigm. Second, existing large-scale analyses of usage data have focused on analyzing use of commands and features (e.g. command frequencies, command vocabularies). What is missing is work that characterizes use of feature-rich software at the level of sessions and tasks.

This motivated us to conduct an analysis of data from ingimp, an instrumented version of the GIMP image editor, which we present in Chapter 3. The results of our study show that the major findings discussed above also apply to direct manipulation applications in the image editing domain. We also found that ingimp users tend to use the application to perform short, targeted tasks using a small subset of the full functionality available. The findings from this study directly motivated a number of aspects of our task-centric interface design.

2.1.2 The Impact of Rich Feature Sets

Given that large-scale studies have shown that users only make use of a small amount of the available functionality in feature-rich software, a logical next question is what effect this unused functionality is having on users.

Quantitatively, past work has found that the number of interface elements negatively impacts task time, particularly for novices users, whose visual search time is a linear function of the number of “relevant” items present in an interface (e.g. within a given menu or toolbar) (Cockburn, Gutwin, and Greenberg 2007). As expertise increases, the impact of excess complexity on performance is

lessened. However, for all but perhaps the most expert users, command selection time is always some function of the number of relevant commands available (Cockburn, Gutwin, and Greenberg 2007; K. L. Norman 1991).

Qualitatively, McGrenere and Moore found that many users find rich feature sets to be

overwhelming, frustrating, and difficult to navigate (McGrenere and Moore 2000); and Lazar et al, found that “Missing / hard-to-find or unusable features” was the second most commonly cited cause of frustrating experiences with computers in the workplace (Lazar, Jones, and Shneiderman 2006).

(28)

Though the McGrenere and Moore study did find evidence that unused functionality had a negative effect, there was no broad agreement among participants on how they wanted to see excess

functionality handled. Only 25% of participants wanted unused functionality to be removed entirely, and 45% wanted unused functions to be tucked away. Moreover, 51% of participants agreed that it is important to continually discover new functions of the application (McGrenere and Moore 2000).

In concert with the findings from the previous section, these findings highlight the challenge of designing interfaces for rich feature sets. Unused functionality has a negative impact on usage, but you cannot simply trim down the interface to include the “most important” commands, because this set of commands varies from user to user. In our task-centric interface design we address this problem through task adaptation, which customizes the interface for specific tasks.

2.1.3 Learning to Use Feature-Rich Software

The work reviewed in the previous two sections focuses on the high-level impact of rich feature sets on users. In this section, we discuss what is currently known about how users go about learning and performing tasks in feature-rich software.

This is not a new question. Some of the earliest research into the usability of software, conducted in the 1980s, examined how individuals learn to use early word processing systems (Carroll et al. 1985; Mack, Lewis, and Carroll 1983). In the introductory chapter to The Nurnberg Funnel, a collection of a series of studies on this topic carried out by researchers at the IBM Watson Research Center, John Carroll summarizes the zeitgeist of this period in the history of computing:

In the late 1970s, access to computing equipment was expanding rapidly to people who were not programmers or other computer professionals. These people often experienced serious and

frustrating difficulties when they attempted to use computers. Tasks they had been able to do had to be relearned. Competent secretaries, accountants, and lawyers were—at least temporarily— returned to varying levels of incompetence until they could master the system. (Carroll 1990)

As a response, Carroll and others set out to understand the difficulties that individuals had with learning software and how to better help users to learn this increasingly prevalent technology. This early work provided some of the first insights into how users react when faced with new software, and has had an enduring impact on research in this area.

A robust finding from this early work is that users of software systems are task-focused; continuous progress toward goals is paramount, and time spent on other concerns, including learning how to

(29)

perform a task, are minimized to the greatest extent possible (Carroll and Rosson 1987; Rieman 1996; Carroll et al. 1987; Carroll 1990; Mackay 1991).

In particular, Carroll and Rosson coined the term the “paradox of the active user” to refer to the phenomenon that users could ultimately save time by spending an initial period learning a system, but that this is counter to how they behave in practice (Carroll and Rosson 1987); and Rieman concluded that “Task-free exploration is unlikely to occur, not only because systems are uninteresting, but because the users’ fundamental goal is to complete their work-related tasks. To that end, they will use their knowledge of the available resources to choose the fastest, most focused learning strategies available.” (Rieman 1996). Finally, in a study of software customization behavior, Mackay found that users were hesitant to take time away from their primary tasks to spend time customizing because it meant trading off short-term productivity for an unknown long-term benefit (Mackay 1991).

Our idea for task-centric interfaces was motivated by these findings in a number of ways. First, task-centric interfaces provide assistance with performing specific tasks, as opposed to more general assistance with using the system. Second, the task adaptation component of task-centric interfaces organizes commands and features for the task at hand, directly supporting the user in completing their current task. Finally, the task knowledge provided by the task-centric interface is introduced in the context of a task, so users can engage in just-in-time learning as needed.

2.1.4 Problem-Solving Strategies

Past work has also been undertaken to understand the specific problem-solving strategies that users employ when faced with unfamiliar tasks in feature-rich software. This work is relevant to our work, because we are positing a new way for users to solve problems and achieve goals in feature-rich software.

Studies have shown that users adopt a range of different problem-solving strategies to perform tasks in feature-rich software, but that they favor an exploratory approach consisting of self-guided exploration of the interface, and trial-and-error experimentation, to attempt to make progress toward their goals (Rieman 1996; Novick, Andrade, and Bean 2009; Andrade, Bean, and Novick 2009; Trudel and Payne 1995; Carroll 1990).

Novick, Andrade, and Bean showed that this exploratory strategy can be effective for completing tasks, but can lead to suboptimal solutions, and its effectiveness is heavily dependent on the

(30)

two common difficulties faced by users when following a trial-and-error approach: hidden

affordances, where the interface does not provide sufficient affordances to allow the user to proceed with the task; and false affordances, where the system includes affordances that mislead the user because they appear to be related to the task, but are actually not.

Trudel and Payne found that users engaging in exploratory problem-solving exhibited maladaptive behavior characterized by a large number of mode changes and goal-shifts, and argue that exploratory problem-solving can lead to poorer learning outcomes if the user does not reflect on the results of their explorations (Trudel and Payne 1995).

Finally, both work by Rieman, and by Novick, Andrade, and Bean, suggest that users’ choice of problem-solving strategies are dependent on their level of experience with the system (Rieman 1996; Novick, Andrade, and Bean 2009). In particular, Rieman found that inexperienced users were more likely to select whatever learning method is most readily available, while more experienced users were more selective, choosing the strategy that would help them acquire the skills they needed in the least possible time.

In summary, past work has shown that users tend to favor and adopt exploratory problem-solving strategies, but that these strategies are associated with a number of problems: exploratory learning can lead to suboptimal results; it can lead to problems caused by false or hidden affordances; and it can lead to less learning if the user does not reflect on their actions. Moreover, these difficulties appear to be more pronounced for novice users, who are less selective about their choice of problem-solving strategies.

2.1.5_{The Challenge Posed by Feature-Rich Software}

The main points from this section can be summarized as follows:

• Command frequencies across large numbers of users tend to follow a long-tailed

distribution, and individual users’ command vocabularies are small and idiosyncratic. This indicates that users will be faced with excess interface complexity from unused features. • Rich feature sets and excess interface complexity have a negative effect on user

experience.

• Users are task focused, which makes them hesitant to spend time on activities that do not directly advance their primary task, including learning.

(31)

• Users adopt a range of problem-solving strategies, but appear to favor exploratory and trial-and-error approaches.

• Evidence for the effectiveness of this exploratory problem-solving strategy is mixed. It can enable users to complete tasks, but may lead to suboptimal solutions; its effectiveness appears to be heavily dependent on the affordances provided by the interface; and it may lead to poorer learning outcomes if the user does not reflect on the results of their explorations. Moreover, these difficulties may be more pronounced for novice users. Overall, these findings argue that feature-rich software is particularly challenging for novice users, and for users performing unfamiliar tasks. Moreover, current help resources appear to be

incompatible with users’ task focus and preference for exploratory problem-solving strategies. This suggests that it is worthwhile to investigate alternative ways to support users in performing tasks in feature-rich software. In the next section, we review existing work that has been done in this area.

2.2 _{Support Techniques for Feature-Rich Software}

Research into how to improve use of feature-rich software has focused on three main areas: interface personalization, task automation, and tutoring. We review each area in turn.

2.2.1 Interface Personalization

Interface personalization approaches provide a mechanism for dealing with interface complexity by customizing the interface to better suit the individual user. A wide range of different approaches to personalization have been explored:

Level-structured interfaces offer the user access to a small number of predefined feature subsets (Shneiderman 2002). Examples include the Training Wheels interface (Carroll and Carrithers 1984), which blocked invocation of commands for advanced tasks, and software that offer “Basic” and “Advanced” modes.

Adaptable interfaces give the user a mechanism to customize the interface to suit their needs (McGrenere, Baecker, and Booth 2007; MacLean et al. 1990), typically by adding or removing commands from toolbars or menus. The advantage of this approach is that it gives the user control over how the interface is customized. However, empirical studies of customization behavior have found that users are often hesitant to deviate from their primary task to spend time to learn and use customization facilities (Mackay 1990; Mackay 1991) (with gamers being a notable exception (Dyck

(32)

et al. 2003)), or to re-customize when their work habits change (McGrenere, Baecker, and Booth 2002). Mackay also found that users desired to customize based on “patterns of behavior” rather than lists of commands and features (Mackay 1991), which provides some motivation for our task-centric approach.

Adaptive interfaces model the user’s interests, preferences, and usage and automatically initiate changes to the interface (Findlater et al. 2009; Gajos et al. 2008; Gajos et al. 2006). This approach has the advantage of not requiring the user to take time to customize. However, studies have shown that system-initiated changes to the interface can lead users to feel a lack of control, transparency, and predictability (Fischer 1993; Höök 2000).

Finally, mixed-initiative approaches have been investigated, which combine the elements of adaptable and adaptive strategies (Bunt, Conati, and McGrenere 2007; Oppermann 1994; Fischer 1993; Bunt, Conati, and McGrenere 2004).

Arcing back to the goal of this dissertation, there are a number of reasons why existing approaches to interface personalization are not well suited to supporting users in performing unfamiliar tasks.

First, existing personalization approaches have focused on providing fast navigation to individual commands and features. They do not address how an interface might be optimized for performing specific tasks.

A partial exception to this is the Perspectives feature in the Eclipse IDE. Perspectives can be considered to be a level-structured interface based on tasks; loading a perspective (e.g. “Debug”) rearranges the layout of views and editors in the interface to suit it to the various activities involved in debugging. While this has clear conceptual similarities with our task-centric approach, the

customizations provided by Perspectives are suited to broad classes of tasks instead of specific tasks (e.g. debugging encompasses many specific activities, including setting a breakpoint, advancing the instruction pointer, etc.), and customizations do not provide specific guidance on how to perform tasks (that is, they do not provide task knowledge). Thus, Perspectives does not address the goal of supporting users in performing unfamiliar tasks, which is our focus.

Second, to provide benefits, existing personalization approaches typically require the user to be familiar with the application’s functionality (so the user can effectively customize the interface), or the system to have a model of the user’s past behavior (so the system can perform reasonable

(33)

customizations). In the situation of a user performing an unfamiliar task, neither of these is likely to be the case.

There is some work on interface personalization that circumvents these requirements by taking advantage of the efforts or usage of other users of an application. In the area of adaptive interfaces, OWL (Linton et al. 2000) and CommunityCommands (Matejka et al. 2009) provide adaptive recommendations of commands based on the usage data of similar users in an organization, and for adaptable interfaces, Kahler proposed the notion of collaborative tailoring, or the idea of providing explicit support for sharing customizations with other users (Kahler 2001a). However, we are unaware of any work that has applied such an approach to assisting the user with performing unfamiliar tasks.

In summary, existing work on interface personalization has focused on customizing interfaces on the granularity of individual users, to support efficient access to individual commands. We are unaware of work that has explored interface customization as a method to assist with performing specific unfamiliar tasks.

2.2.2 Task Automation

A second major approach to assisting users of feature-rich software has been task automation. The classic examples of task automation in feature-rich software are wizards, macros, and scripts. Wizards “[guide] a person along an efficient path to success, autonomously completing those steps of the task that do not require the person’s attention.” (Dryer 1997) In their most common form, wizards take the form of a linear dialog composed of multiple steps that prompt the user for information necessary to further the task. Scripts and macros execute a pre-defined set of operations, typically without input from the user.

Wizards and scripting capabilities are common in feature-rich applications. However, these techniques have a number of limitations. First, they are most effective for tasks that have

algorithmically-derived solutions, making them poorly suited to ill-defined tasks. Second, they tend to adapt poorly to unforeseen conditions. Finally, they are laborious to create and maintain, often requiring advanced skills.

To address these drawbacks, a number of alternative approaches to task automation have been investigated. Numerous approaches have been investigated to provide automation of traditionally difficult to automate tasks. These include using machine learning and image recognition (Berthouzoz

(34)

et al. 2011; Grabler et al. 2009b; Yeh, Chang, and Miller 2009), automatically personalizing scripts for the current user (Leshed et al. 2008), and using the crowd (human workers) to complete a task (Bernstein et al. 2010).

To address the problem that scripts are laborious to create, a number of systems have adopted a by demonstration approach where the task is performed in the system to record a script (Grabler et al. 2009b; Leshed et al. 2008; Bergman et al. 2005; Berthouzoz et al. 2011; “Adobe Labs Tutorial Builder” 2012). The use of data from multiple recordings to make scripts more general has also been explored (Bergman et al. 2005; Berthouzoz et al. 2011).

Typically, research on automation techniques has focused on minimizing the input required from the user. However, some of these projects have explored how to automate parts of a task, while keeping the user “in the loop”. DocWizards (Bergman et al. 2005) and CoScripter (Leshed et al. 2008) provide scripts that can prompt the user to take actions. Tutorial Builder (“Adobe Labs Tutorial Builder” 2012) creates text/image tutorials that include a “Show me in Photoshop” button that

executes the corresponding action in-application. Tutorial-Based Applications (Laput et al. 2012) convert web-based tutorials into interactive macros that allow the user to adjust how an individual step is applied, and have the results automatically apply through the rest of the tutorial.

By expanding the set of tasks that can be completed with minimal input from the user, task

automation techniques do have the potential to help users with performing unfamiliar tasks. However, as discussed in Chapter 1, there will always be ill-defined tasks that cannot be completely automated, and task automation may in fact increase the proportion of these tasks faced by users. Thus, these approaches do not represent a general-purpose technique for task support.

Moreover, with task-centric interfaces, we are positing a new way of working with feature-rich software that includes both a mechanism for helping the user to complete a given task, as well as a mechanism for accessing assistance for a task. That is, the scope of our investigation is different from work on task automation techniques, which do not address how users would locate relevant wizards, scripts, or macros.

2.2.3 In-Application Tutoring Systems

A third major approach that has been explored to assist users with using feature-rich applications is in-application tutoring systems, which provides instruction on how to complete tasks within the context of the application’s interface.

(35)

Early work in this area by Tuck and Olsen referred to this as a guided task paradigm for help delivery. Tuck and Olsen’s system translated scripts describing tasks into in-application

demonstrations, with arrows and explanatory text to indicate the interactions with the interface that would be required at each step (Tuck and Olsen 1990).

A number of in-application tutoring systems in this style have been developed. Stencils (Kelleher and Pausch 2005) presents instructions on a semi-transparent coloured layer over the application’s interface, limiting the user’s interaction with the interface to components that are revealed through holes in the translucent layer. Google SketchUp’s “self-paced tutorials” (“Google SketchUp Training” 2012) present instruction in-context in documents. Sketch-Sketch Revolution (SSR) (Fernquist, Grossman, and Fitzmaurice 2011) uses many of the same techniques as previous systems, and additionally applies a number of techniques designed to allow the user to experience success, and keep the user in a flow state (Csikszentmihalyi 2008) while they learn.

Research on these systems has explored a number of novel interaction techniques for tutoring the user in how to perform tasks in-application. However, these techniques do not support the user while they are in the process of performing their own tasks. Instead, the focus is on teaching the user how to use the system while performing an artificial task, with the idea that they could then apply what they have learned to their own goals. In principle, many of these techniques could be adapted to provide support while the user was advancing their own goals, and this has been suggested in the form of Guides, a user interface agent that monitor the user’s interaction with the system and presents information to assist their progress (Dryer 1997). However, implementing this support is difficult in practice, because it requires that the system have an accurate model of the user’s specific intentions as they interact with the system.

In contrast, our work on task-centric interfaces investigates a mechanism for assisting the user with completing their own unique tasks. This is more compatible with the task-focus of users, discussed earlier in this chapter.

2.2.4 The Need for Techniques to Support Unfamiliar Tasks

The main points from this section can be summarized as follows:

• Existing interface personalization techniques have not been designed to assist users with performing unfamiliar tasks.

(36)

• Exiting approaches to in-application task support have focused on either (1) providing more sophisticated techniques for automating a given task, or (2) teaching the user how to perform a task in an artificial situation. Neither of these techniques offers a general-purpose approach to help the user with performing unfamiliar tasks.

• Existing work on in-application task support has not addressed mechanisms for locating task support for a user’s current goal.

We conclude that the space of general-purpose techniques to support users in performing

unfamiliar tasks in feature-rich software is largely unexplored. Moreover, the idea of designing such a technique around interface personalization on the granularity of tasks is novel, and well-motivated by our current understandings about how users learn and use feature-rich software. Our work on task-centric interfaces in this dissertation is intended to contribute to filling these gaps, and exploring this space for design.

In the next chapter, we begin our investigation of task-centric interfaces by returning to another gap in current research that we identified earlier in this chapter. While there have been a number of large-scale analyses of usage of feature-rich software, none have investigated large-large-scale use of

applications that follow a primarily direct-manipulation paradigm. Moreover, existing large-scale analyses have focused on use of commands or features (e.g. overall command frequencies, users’ command vocabularies), and have not characterized use of feature-rich software at the level of sessions and tasks. In the next chapter, we present the results of a study conducted to fill this gap.

(37)

Chapter 3 Characterizing Large-Scale Use of a

Direct-Manipulation Application in the Wild

Our investigation of task-centric interfaces starts with a study that we conducted to understand how individuals make use of feature-rich image editing software “in the wild”, outside of a laboratory setting. In particular, in this chapter we present an analysis of usage logs collected from a public deployment of ingimp (Terry et al. 2008), an instrumented version of the open source GNU Image Manipulation Program. Our analysis includes data from more than 200 users and 4000 sessions, collected over a two-year period from May 2007 to May 2009.

As discussed in Chapter 2, existing large-scale analyses of feature-rich software have focused on applications in primarily text-based domains, and have not characterized use of feature-rich software at the level of sessions and tasks.

The analysis presented in this chapter extends previous work by studying a primarily direct-manipulation application, and providing characterizations of sessions and tasks. Specifically, the key findings from our analysis are:

• The majority of users of ingimp use the software to perform relatively short, targeted tasks (averaging ~6 minutes of active usage per session.)

• Users mostly work on documents of modest complexity, both in terms of resolution and number of layers per document.

• Users perform tasks using only a small proportion of the full functionality of the

application (an average of only ~6 unique commands are used in any particular session, of the nearly 500 available.)

• Users’ command vocabularies tend to be small (averaging only ~27 commands), and have little overlap with one another (the vast majority of observed commands are used by no more than 10% of users each.)

• Users of ingimp perform a range of different high-level tasks, including some relatively sophisticated image editing tasks.

(38)

These findings directly motivated us to develop the idea of task-centric interfaces. Our finding that the majority of users use the application for short, targeted tasks establishes this as a valid use case for feature-rich software that should be designed for; and our finding that users only require a small percentage of the available functionality to perform tasks suggests task-centric customization as a potentially effective solution.

The rest of this chapter is structured as follows. We start by describing the ingimp project and the ingimp dataset that we analysed. Next, we provide a basic summary of ingimp users, including the characteristics of their sessions (e.g. length, frequency of use) and the characteristics of the documents that they worked on. We then present a summary of command usage across the ingimp user community. Finally, we characterize the specific kinds of tasks performed by ingimp users, through use of an unsupervised clustering technique. We conclude with a discussion of the implications of our findings for task-centric interfaces.

Note that the content presented in this chapter is drawn from the published paper Characterizing Large-Scale Use of a Direct Manipulation Application in the Wild (Lafreniere et al. 2010) which reports on work carried out by myself in collaboration with Andrea Bunt, John Whissell, Charles Clarke, and Michael Terry.

3.1 _{The ingimp Dataset}

ingimp is an instrumented version of the open source GNU Image Manipulation Program (Terry et al. 2008) and an experiment in open instrumentation, or the idea of openly collecting and publicly disseminating rich application usage data. All collected data was made publicly available on the project’s website (www.ingimp.org) for the duration of the project.

ingimp was designed to collect the following information:

• Activity tags – optional user-supplied keywords describing how the user intends to use the application (prompted for at the start of each session)

• System characteristics, including operating system, CPU, number of monitors, screen resolution, and time zone

• Document summaries, including the resolution and number of layers in images • Command invocations that appear on the undo stack

Task-Centric User Interfaces