Evaluating Interface Designs

(1)

(2)

INTRODUCTION

Experienced designers know that extensive testing is a necessity. If feedback is the “breakfast of champions”, then testing is the “dinner of the gods”.

The determinants of the evaluation plan include atleast :

• Stage of design(early, middle, late)

• Novelty of project(well defined versus exploratory) • Number of expected users

• Criticality of the interface(life-critical system versus

museum-exhibit support)

• Cost of product and finances allocated for testing. • Time available.

(3)

Expert Reviews

▣ A natural starting point for evaluating new or revised interfaces is to ask colleagues or customers for their feedback.

▣ Such informal demos can provide some useful feedback, but more formal expert reviews have proven to be far more effective.

▣ Expert reviews can occur early or late in the design phase. The

outcome can be a formal report with problems identified or recommendations for changes.

▣ The reviewers note possible problems for discussion with the

(4)

Expert Reviews

▣

Expert

reviews usually takes from

half a day to one week,

although a lengthy training period may be required

to

explain the task domain or operational procedures.

▣

There are a

variety of expert-review methods

from which to

choose :

▣

Heuristic evaluation

▣

Guidelines review

▣

Consistency Inspection

▣

Cognitive walkthrough

▣

Formal Usability Inspection

(5)

Expert Reviews

❖

Heuristic Evaluation :

The expert reviewers analyse an interface to determine conformance(how well a system meets a specified standard) with a short list of design heuristics, such as eight golden rules, game speed..

❖

Guidelines Review :

▣ The interface is checked for conformance with the organizational or other guidelines document.

(6)

Expert Reviews

▣ Consistency Inspection :

The experts verify consistency across a family of interfaces such as terminology, fonts, color schemes, layout, input and output formats and so on within the interface as well as in the training materials and online help.

Cognitive walkthrough:

High-frequency tasks are a starting point, but rare critical tasks, such as error recovery should be walked through.

(7)

Expert Reviews

▣

Formal Usability Inspection :

The experts hold a

courtroom-style meeting

, with a moderator

or judge, to present the interface and to discuss its merits and

weaknesses.

Design team members may rebut(invalidate) the evidence about

problems in an adversarial format.

The

adversarial system

(or

adversary system

) is a legal

system where two advocates represent their parties' positions

before an impartial person or group of people, usually a

jury

or

judge

, who attempt to determine the truth of the case.

[

(8)

▣

Metaphors of Human Thinking

:

▣

They consider metaphors for five aspects of human

thinking:

▣

Habit, stream of thoughts, awareness and associations,

utterances and thought and knowing.

(9)

The number of expert reviews will depend on the magnitude(size) of the project and on the amount of resources allocated.

Expert reviews also include required small fixes such as spelling mistakes, poorly aligned data entry fields or inconsistent button placement.

Expert reviewers should also take training courses, read manuals and try the interface in as close as possible to a realistic work

environment.

Another approach : bird’s eye view study about full set of

printed screens laid out on the floor or pinned to walls and it detects unusual patterns.

(10)

▣ The bird’s eye view enables reviewers to quickly see if the fonts, colours, and terminology are consistent.

▣ Expert reviewers may also use software tools to speed up their analyses, especially with large interfaces.

▣ Web-accessibility validation, privacy-policy checks and

download-time reduction are some design-analysis tools which are growing more effective.

▣ Disadvantage :

The danger with expert reviews is that the experts may not have an adequate understanding of task domain or user

communities.

(11)

▣

To strengthen the possibility of

successful expert

review

, choose

knowledgeable experts who are

familiar with the project and who have a long term

relationship with the organization.

(12)

USABILITY TESTING AND LABORATORIES

•

The

usability-test

report

provide

supportive

confirmation of progress and specific recommendations

for changes

in the design phase.

•

This not only

speed up the projects but also provides

dramatic cost savings.

•

Usability tests are designed to

find flaws in interfaces

(13)

Usability Testing and Laboratories

•

Usability Laboratories

:

-- a typical modest Usability laboratory would have

two 10- by 10- foot areas, divided by a half-silvered

mirror: one for the participants to do their work and

another for the testers and observers.(designers,

managers, customers)(fig 4.1)

•

IBM

was the early leader in developing usability

laboratories.

•

Microsoft

started later with 25-usability test labs.

•

Consulting community

that do testing for hiring is also

(14)

Usability Testing and Laboratories

• Usability laboratory staff members participate in early task analysis or

design reviews, provide information on software tools or literature references and help to develop the set of tasks for the usability test.

• Two to six weeks before the usability test , the detailed test plan is

developed. It contains the list of tasks plus subjective satisfaction and debriefing questions.

• The number, type and sources of participants are also identified.

• After changes are approved, participants are chosen to represent the

(15)

▣

Participants can read and sign a statement like this :

-- I have freely volunteered to participate in this

experiment.

-- I am aware that I have the right to withdraw consent

and to discontinue the participation at any time.

Effective technique

: invite users to

think aloud

what

they are doing.

(16)

▣ After a suitable time period for accomplishing the task list – the participants can be invited to make general comments or

suggestions or to respond to specific questions.

▣ Videotaping participants performing task is often valuable for later review and for showing designers or managers the

problems that user encounters.

(17)

Variant forms of usability testing : ▪ Paper mockups

▪ Discount usability testing

▪ Competitive usability testing

▪ Universal usability testing

▪ Field test and portable labs

▪ Remote usability testing

▪ Can-you-break this test

▪ Paper mockups(manipulate) : this assess user reactions to

(18)

▪ Discount usability testing:

This is used in formative evaluation (while designs are changing substantially) . It identifies problems that guide redesign, and more extensive usability testing be used as a summative

evaluation (near the end of the design process). It provides

evidence for product announcements and clarifies training needs.

eg : with 4 min of instruction, every participant successfully programmed the videorecorder.

▪ Competitive usability testing :

(19)

Usability Testing and Laboratories

⚫ Universal usability testing :

This approach tests interfaces with highly diverse users, hardware, software platforms and networks.

Eg : web-based information services/e-government services.

Trials with small and large displays, slow and fast networks and a range of OS or internet browsers will do much to raise the rate of customer success.

⚫ Field Tests and Portable Lab:

(20)

⚫ Portable usability laboratories with videotaping and logging facilities have been developed to support more thorough field testing.

⚫ A different kind of field testing is to supply users with test versions of new software or consumer products and be asked to comment.

⚫ Remote Usability Testing :

Usability test can also be conducted online without incurring the complexity and cost of bringing participants to lab.

(21)

⚫ The downside is that there is less control over user behaviour and less chance to observe their reactions, although usage logs and phone interviews are useful supplements.

⚫ Can-you-break-this tests :

The users try to find fatal flaws in the system or otherwise destroy it, has been used in other projects and should be considered

seriously.

(22)

EVALUATION DURING ACTIVE USE

❖ A carefully designed and thoroughly tested interface is a wonderful asset, but successful active use requires constant attention from dedicated managers, user service personnel and maintenance staff.

❖ Interface refinements provide higher levels of service.

❖ Interviews and focus group discussions.

❖ Continuous user-performance data logging

❖ Online or telephone consultants, Online suggestion box or email trouble reporting.

❖ Discussion groups, wikis and newsgroups

(23)

Evaluation During Active Use

▣ Interviews and focus group discussions

▣ Interviews with individual users can be productive because the interviewer can pursue specific issues of concern.

▣ After a series of individual discussions, focus group discussions are valuable to ascertain the universality of comments.

▣ Interviewing can be costly and time-consuming, so usually only a small fraction of the user community is involved.

(24)

Continuous user-performance data logging(sorting):

▣ Logging data provide guidance in new hardware, changes in operating procedures, improvements to training, plans for system expansion and so on.

▣ Eg : if a frequency of error message is recorded, the highest frequency error is a candidate for attention.

▣ Staff should examine whether there is an error in the code or whether users are avoiding use of some facility.

▣ If logging data is available for each command, each help screen and each database record , changes to the

(25)

▣ A major benefit of user-frequency data is the guidance that they provide to system maintainers in optimizing

performance and in reducing costs for all participants.

▣ Commercial services such as Nielsen NetRatings and

knowledge Networks are making a success of providing

clients with log data and analysis of web visits from their panels of users.

▣ The purchasers of data are interested to know what kinds of people buy books, visit new sites, or seek healthcare

(26)

Online or telephone consultants, Online suggestion box or email trouble reporting.

▪ They can provide effective and personal assistance to users who are experiencing difficulties.

▪ They can suggest solutions to problems , improvements and potential

extensions.

▪ Many organizations provide toll-free numbers via which the user can reach a knowledgeable consultant. Others charge for consultation by the minute.

▪ America online provides live chat rooms for discussion of user problems.

Users can type their questions and get responses promptly.

(27)

Online suggestion box or email trouble reporting :

Electronic mail can be employed to allow users to send messages to the

maintainers or designers.

Online suggestion box encourages users to make productive comments, since

writing a letter may be seen as requiring too much effort.

Web-based error reporting schemes are growing in popularity.

Netscape’s quality Feedback system, Microsoft’s Dr. Watson, and

Bugtoaster are programs that automatically file reports after a crash.

Suggestion boxes and complaint facilities are becoming common in websites for organizations that are eager to provide high levels of support.

Microsoft’s feedback page request suggestions in these categories :

(28)

Discussion groups, wikis and newsgroups :

❑ Many interface designers and website managers offer users discussion groups or newsgroups to permit posting of open messages and questions.

❑ America online, Yahoo groups and Microsoft Networks provides discussion groups.

❑ Discussion groups usually offers list of item headlines, allowing users to scan items for relevant topics.

❑ New items can be added by anyone but usually someone moderates the discussion to ensure that useless, offensive or repetitious items are

removed.

(29)

Tools for automated Evaluation:

▣ Software tools can be effective in evaluating user interfaces for desktop applications, web sites, and mobile devices.

▣ Tools to check spelling or concordance of terms benefit interface designers.

▣ The inclusion of more sophisticated evaluation procedures can allow interface designers to assess whether a menu tree is too deep or

contains redundancies, whether widget

labels have been used consistently, whether all buttons have proper transitions associated with them,…

(30)

▣ The World Wide Web consortium provides a Markup Validation Service for HTML checking that pinpoints the problems.

▣ Eg : Line 13, column 8, required attribute “TYPE” not specified.

▣ The U.S. National Institute Of Standards and Technology(NIST) Web Metrics TestBed provides extensive testing tools, such as a static analyzer(WebSAT) for web pages or sites, a categorical analyzer to check whether categories extracted from web pages fit the designer’s intents, an instrumenter(WebVIP) to quickly instrument existing web pages to capture logs of interactive use.

▣ Some of the easily applicable results for page design :

▣ Few web sites did not always have the fastest user performance, Download speeds for the web pages can be improved by using website optimization services that count the number of items in the page, the number of bytes in each image, and size of the source code.

▣ Run-time logging software captures the users’ patterns of activity. Frequency of each error message, menu-item selection, dialog-box appearance, form-field usage are of great benefit to maintenance personnel.