Expert Reviews

A natural starting point for evaluating new or revised interfaces is to ask col-leagues or customers for their feedback. Such informal demos with test subjects can provide some useful feedback, but more formal expert reviews have proven to be far more effective (Nielsen and Mack, 1994). These methods depend on having experts (whose expertise may be in the application or user-interface domains) available on staff or as consultants. Expert reviews can then be con-ducted on short notice and rapidly.

Expert reviews can occur early or late in the design phase. The outcome can be a formal report with problems identified or recommendations for changes.

Alternatively, the expert review may result in a discussion with or presentation to designers or managers. Expert reviewers should be sensitive to the design team's ego involvement and professional skill, so suggestions should be made cautiously: It is difficult for someone just freshly inspecting an interface to understand fully the design rationale and development history. The reviewers note possible problems for discussion the designers, but solutions gener-ally should be left for the designers to produce. Expert usually take from half a day to one week, although a lengthy training period may be required

to explain the task domain or operational procedures.Itmay be useful to have the same as well as fresh expert reviewers as the project progresses. There are a variety of expert-review methods from which to choose:

• Heuristic evaluation. The expert reviewers critique an interface to determine conformance with a short list of design heuristics, such as the eight golden rules (see section 2.3.4). Itmakes an enormous difference ifthe experts are familiar with the rules and are able to interpret and apply them.

• Guidelines review. The interface is checked for conformance with the organi-zational or other guidelines document. Because guidelines documents may contain a thousand items, it may take the expert reviewers some time to mas-ter the guidelines and days or weeks to review a large inmas-terface.

• Consistency inspection. The experts verify consistency across a family of inter-faces, checking for consistency of terminology, fonts, color schemes, layout, input and output formats, and so on within the interface as well as in the training materials and online help. Software tools can help automate the process, as well as produce concordances of words and abbreviations.

• Cognitive walkthrough. The experts simulate users walking through the inter-face to carry out typical tasks. High-frequency tasks are a starting point, but rare critical tasks, such as error recovery, also should be walked through.

Some form of simulating the day in the life of the user should be part of the expert-review process. Cognitive walkthroughs were developed for inter-faces that can be learned by exploratory browsing (Wharton et a1., 1994), but they are useful even for interfaces that require substantial training. An expert might try the walkthrough privately and explore the system, but there also should be a group meeting with designers, users, or managers to conduct the walkthrough and provoke discussion. Extensions to cover web-site naviga-tion incorporate richer descripnaviga-tions of users and their goals plus linguistic-analysis programs to estimate the similarity of link labels and destinations (Blackmon et a1., 2002).

• Formal usability inspection. The experts hold a courtroom-style meeting, with a moderator or judge, to present the interface and to discuss its merits and weaknesses. Design-team members may rebut the evidence about problems in an adversarial format. Formal usability inspections can be educational experiences for novice designers and managers, but they may take longer to prepare and more personnel to carry out than do other types of review.

Expert reviews can be scheduled at several points in the development process, experts are available and when the design team is ready for feed-back. The number of expert will depend on the magnitude of the project and on the amount of resources allocated.

An expert-review report should aspire to comprehensiveness, rather than making opportunistic comments about specific features or presenting a random

collection of suggested improvements. It might use a guidelines document to structure the report, then comment on novice, intermittent, and expert features and review consistency across all displays. Another strategy would be to use a theory or model, such as the object-action interface model (see Section 2.5), to organize a report. An evaluation of the task objects and actions (nouns and verbs) is a good starting point, followed by comments on the corresponding interface objects and actions.

If the report ranks recommendations by importance and expected effort level, managers are more likely to implement them (or at least the high-payoff, low-cost ones). In one expert review, the highest priority was to shorten a three-to-five minute log-in procedure that required eight dialog boxes and passwords on two networks. The obvious benefit to already over-busy users was apparent, and they were delighted with the improvement. Common middle-level recom-mendations include reordering the sequence of displays, providing improved instructions or feedback, and removing nonessential actions. Expert reviews should also include required small fixes such as spelling mistakes, poorly aligned data-entry fields, or inconsistent button placement. A final category includes less vital fixes and novel features that can be addressed in the next ver-sion of the interface.

Comparative evaluation of expert-review methods and usability-testing methods is difficult because of the many uncontrollable variables. However, the studies that have been conducted provide evidence for the benefits of expert reviews (Jeffries et aI., 1991; Karat, Campbell, and Fiegel, 1992). Different experts tend to find different problems in an interface, so three to five expert reviewers can be highly productive, as can complementary usability testing.

Expert reviewers should be placed in the situation most similar to the one that intended users will experience. The expert reviewers should take training courses, read manuals, take tutorials, andtrythe interface in as close as possible to a realistic work environment, complete with noise and distractions. However, expert reviewers may also retreat to a quieter environment for detailed review of each screen.

Another approach, getting a bird's-eye view of an interface by studying a full set of printed screens laid out on the floor or pinned to walls, has proved to be enormously fruitful in detecting inconsistencies and spotting unusual patterns.

The bird's-eye view enables reviewers to quickly see if the fonts, colors, and ter-minology are consistent, and to appreciate whether the multiple developers have adhered to a common style.

Expert reviewers may also use software tools to speed their analyses, espe-cially with large interfaces. Sometimes string searches on design documents, help text, or program code can be valuable, but more specific interface-design analyses-such as web-accessibility validation, privacy-policy checks, and download-time reduction-are growing more effective. These tools usually pro-vide specific instructions for improvements.

The danger with expert reviews is that the experts may not have an adequate understanding of the task domain or user communities. Experts come in many flavors, and conflicting advice can further confuse the situation (cynics say, every PhD., there is an equal and opposite Ph.D."). To strengthen the pos-sibility of successful expert review, it helps to choose knowledgeable experts who are familiar with the project and who have a long-term relationship with the organization. These people can be called back to see the results of their inter-vention, and they can be held accountable. However, even experienced expert reviewers have difficulty knowing how typical users, especially first-time users, will behave.