Procedure

Individual participants completed the exercises on the Drexel University Campus.

The overall procedure can be seen in Figure 6. Participants were asked to fill out a pre-test questionnaire. They were then given the design task. Those in the patterns group were individually given an overview/tutorial on patterns. They were then given the 39 patterns in aIRPLane. Those in the guidelines group were also individually given an overview/tutorial on guidelines. They were then given the 10 guidelines. Those in the control group did not receive anything other than the design task. After completing the design task participants were asked to fill out a post-test questionnaire. All the resulting designs were then evaluated by two experts. Both experts were Professors of HCI with a significant number of referred publications and decades of experience conducting related research, teaching, and designing and evaluating interactive systems.

Figure 6. Experimental Procedure

Pre-test Questionnaire

Participants were asked to fill out a pre-test questionnaire (Appendix B) that gathered demographic information and information about the participants’ educational experience including the number of HCI courses completed.

Design Task

Participants were given a design task (Appendix C) and informed that they were hired to design interfaces for an information retrieval system. The design task described the basic functionality that needed to be included in the system interfaces. The design task also included a short scenario which described how a student may interact with a system. The scenario was provided in order to guide the students towards sketching the required interfaces

No exposure to pattern language or any other design technique

Exposure to pattern language

No exposure to pattern language or any other design technique

Exposure to pattern language

in a particular order. This was done in an effort to make the evaluation of the interfaces easier. The scenario followed a logical flow of information seeking behavior that does not seem unreasonable.

A few well known and highly cited models of information seeking behaviors were reviewed in order to reinforce the logical flow of the steps in the scenario, including Marchionini’s (1995) information seeking in electronic environments, Dervin’s (1983;

Dervin & Nilan, 1986) sense making model, Belkin’s (1980) model of Anomalous States of Knowledge (ASK), and Kuhlthau’s (1991) model which addresses the affective states of information seekers. These models reinforce that the logical flow of steps in the scenario are not unreasonable.

Control Condition

The participants in the control condition were given the design task and asked to sketch the interfaces described in the design task and scenario on the interface template.

Guidelines Condition

The participants in the guidelines condition were given the design task and a brief background on designing with user interface guidelines. They were then given Nielsen and Molich’s (Molich & Nielsen, 1990; Nielsen, 1994a;

http://www.useit.com/papers/heuristic/heuristic_list.html) guidelines in the last format cited here. The participants were then asked if they had any questions about guidelines. If there were any questions they were addressed. The participants in the guidelines condition then sketched the interfaces described in the design task and scenario on the interface template.

Guidelines were chosen as the other design technique because they have been compared to and distinguished from patterns in the HCI literature (Granlund et al., 2001), (Henninger & Ashokkumar, 2005). (See previous Chapter for a more detailed discussion.)

Stewart & Travis (2003) point out that Nielsen’s (1994a), Shneiderman & Plaisant’s (2005), and Smith and Mosier’s guidelines are the three most influential sets of guidelines. Nielsen and Molich’s (http://www.useit.com/papers/heuristic/heuristic_list.html) guidelines have been selected as the set of guidelines to be given to those in the guidelines condition because of the reasons mentioned above and because Molich & Nielsen’s (1990) claim that almost all usability problems fit into one of the categories addressed by their guidelines. Recently Nielsen (2005) addressed the claims that usability guidelines from years past may be obsolete and concluded that the majority of the guidelines from 20 years ago are still valid.

Patterns Condition

The participants in the patterns condition were given the design task and the 39 patterns in aIRPLane. They were given a brief overview on the use of patterns in HCI. The participants were then given instructions and asked to sort the patterns into piles of related patterns. They were told that there is no one right way to sort the patterns, but that there are multiple ways the patterns could be sorted because there are multiple ways the patterns are related. See Appendix F for the instructions for the sorting exercise. The participants were also asked to label or name the resulting piles of patterns using post-it notes. After

completing the sorting exercise the participants were then asked to sketch the interfaces described in the design task and scenario on the interface template.

Post- task Questionnaire

After sketching the interfaces all the participants were given a questionnaire to complete. The questionnaire can be found in Appendix I. The questionnaire gathered information about the users’ perceived impressions.

Evaluation of Designed Interfaces

The quality elements upon which the evaluators judged the interfaces were chosen based on definitions of usability (Nielsen, 2003) and measurements used in the empirical studies of patterns in HCI (Chung et al., 2004; Saponas et al., 2006). There is no agreed upon definition of usability, or HCI for that matter, but most would agree that usability is one of the core concepts of HCI. Definitions of usability range from a few words to multiple perspectives within a definition. Table 10 contains various definitions of usability.

Table 10. Definitions of Usability

Author Usability Definition

Bevan, 1995 “ease of use and acceptability of a product for a particular class of users carrying out specific tasks in a specific environment” (p. 156) Gould & Lewis, 1985

“any system designed for people to use should be easy to learn (and remember), useful, that is, contain functions people really need in their work, and be easy and pleasant to use” (p. 300)

Mayhew, 1999 “a measurable characteristic of a product’s user interface that is present to a greater or lesser degree” (p. 1)

Nielsen, 2003 “usability is a quality attribute that assesses how easy user interfaces are to use… the word usability refers to methods for improving ease-of-use during the design process”

Rosson & Carroll, 2002 “the quality of a system with respect to ease of learning, ease of use, and user satisfaction” (p. 9)

Two broad dimensions of usability identified by Mayhew (1999) include how easy it is to learn the user interface and how easy it is to use the user interface. Nielsen (2003) describes the components of usability as: learnability, efficiency, memorability, errors, satisfaction, and utility. Rosson & Carroll (2002) acknowledge that it is difficult to pinpoint what makes a system usable. Rosson & Carroll describe usability as “the quality of a system with respect to ease of learning, ease of use, and user satisfaction” (p. 9).

Ease of use is described on some level in each of these definitions of usability. For this reason, ease of use will be used as one element in evaluating the quality of the resulting

designs. In addition, some of the elements used by both Saponas et al. (2006) and Chung et al. (2004) will be used in order to allow for comparison to their results. Below is a list of the quality elements used in this study and their corresponding definitions. The definitions of level and detail, completeness, and overall quality were taken from Saponas et al.

• Ease of Use – How easy is it to use the interface?

• Level of Detail – How low-level, readily implementable, and non-vague are the elements of the design?

• Completeness – How complete is the design? Does the design contain all the necessary parts that it will need to work?

• Overall Quality – Overall how good of a solution is the design?

These elements have been chosen above others because we are concerned with the systems design process as a whole. Therefore we are concerned with creating interfaces (or paper prototypes) which may be handed off to someone else that may then actually

implement the designed interfaces. These definitions were expanded to include examples after the first training session with the evaluators.

Evaluator Training Sessions

During the first training session it was explained to the evaluators that the interfaces would be evaluated, on a 7-point likert-type agreement scale, using the four quality elements described above (ease of use, completeness, level of detail, and overall quality). The

evaluators were given the definitions of each of these quality elements. Any questions about the definitions were addressed at this point. We then walked through an evaluation of four interfaces from a pilot study using the think aloud protocol to share our ideas about why we were assigning a particular rating to each quality element. Any questions were again addressed.

The two evaluators were then presented with three different sets of interfaces from the pilot study, which consisted mainly of four interfaces: a general search interface, an advanced search interface, a results interface, and a help interface. Each evaluator was asked to individually assign a subjective rating to each set of interfaces for each quality element on a 7-point likert-type agreement scale. After each evaluator individually completed the evaluation, the evaluators disclosed their ratings and justified their reasons for assigning a particular rating. During this portion of the session it became apparent that there were

particular interface elements that had an impact on the various quality ratings. These interface elements were discussed in detail. The evaluators agreed that explicitly listing some of these examples below the definitions of the quality elements would help them in their evaluations.

Therefore the definitions of the quality elements were expanded to include examples of interface elements that exemplified the quality elements.

Before the second session the evaluators were sent the expanded definitions which included more detailed definitions and examples (Appendix K). During the second session the evaluators were again presented the expanded definitions of the quality elements. The evaluators were then asked to individually evaluate four sets of interfaces from the pilot study (different from those used previously) using the expanded definitions. Each evaluator was asked to individually assign a subjective rating to each set of interfaces for each quality element on a 7-point likert-type agreement scale. After completing the individual assessment the evaluators disclosed their ratings and justified their reasons for assigning a particular rating. After this discussion the evaluators were given the 52 sets of interfaces to evaluate individually. (They were both given the interfaces in the same order.)