Chapter 14:
Usability testing and field
studies
The aims:
• Explain how to do usability testing through
examples.
• Outline the basics of experimental design.
• Discuss the methods used in usability
testing.
• Discuss the role of field studies in
evaluation.
Usability testing
• Basic elements:
– A set of typical tasks.
– Controlled environmental settings. – Users.
– Data recording equipments.
• The data
– performance times – errors.
– User satisfaction.
Experimental study
• controlled evaluation of specific aspectsof interactive behaviour
• evaluator chooses hypothesis to be tested
• a number of experimental conditions are considered which differ only in the value of some controlled variable. • changes in behavioural measure are
attributed to different conditions
Example
• There are many different input
devices: keyboard, button, knob,
mouse, touch screen, voice
activation, etc.
• Questions:
– does the optimality of an input device depend on the task being performed? – Is performance with a given input
device influenced by the age of the user?
Example
• Three factors:
– Input devices – Task demands – User capacities• How can we categorize these
factors?
Categorize input devices
• Direct devices: no translation is required between the activity performed by the person and the action of the device – Touch screen – Light pen – Voice activation • Indirect devices: require a translation between the activity of the person and the action of the device – Mouse – Trackballs – Joysticks – Rotary encoders Requires translation between: rotary and linear movement; hand and screen, Learning time Control time …. Can adjust control-display ratio More precise Given tactile feedback …Indirect
Arm fatigue Limited resolution Slow entry Finger or arm may obscure screen Direct hand-eye coordination No memory loading Minimal training …. Direct disadvantages Advantages Devices
Hypothesis
• prediction of outcome– framed in terms of IV and D
E.g. the optimality of an input device is depending on the task being performed
• null hypothesis:
– states no difference between conditions – aim is to disprove this
E.g. the optimality of an input device is independend from the task being performed
Variables
• independent variable (IV)characteristic changed to produce different conditions
e.g. interface style, number of menu items
• dependent variable (DV)
characteristics measured in the experiment e.g. time taken, number of errors.
Experimental factors
• Subjects– who – representative, sufficient sample
• Variables
– things to modify and measure
• Hypothesis
– what you’d like to show
• Experimental design
– how you are going to do it
Experimental design
• within groups design– each subject performs experiment under each condition.
– transfer of learning possible
– less costly and less likely to suffer from user variation.
• between groups design
– each subject performs under only one condition
– no transfer of learning – more users required
Analysis of data
• Before you start to do any statistics:– look at data – save original data
• Choice of statistical technique depends on
– type of data – information required
• Type of data
– discrete - finite number of values – continuous - any value
Analysis - types of test
• parametric– assume normal distribution – robust
– powerful
• non-parametric
– do not assume normal distribution – less powerful
– more reliable
• contingency table
– classify data by discrete attributes – count number of data items in each group
Analysis of data (cont.)
• What information is required?
– is there a difference? – how big is the difference? – how accurate is the estimate?
Experiments & usability
testing
• Experiments is to discover new
knowledge
• Usability testing is to check that the
system is usable by the intended user
population for their tasks.
• Experiments may also be done in
usability testing.
Usability testing & research
Usability testing• Improve products • Few participants • Results inform design • Usually not completely replicable • Conditions controlled as much as possible • Procedure planned • Results reported to developers Experiments for research • Discover knowledge • Many participants • Results validated statistically • Must be replicable • Strongly controlled conditions • Experimental design • Scientific reported to scientific community
Usability lab with observers
watching a user & assistant
Portable equipment for use in
the field
Testing conditions
• Usability lab or other controlled space. • Emphasis on:– selecting representative users; – developing representative tasks.
• Tasks usually last no more than 30 minutes.
• The test conditions should be the same for every participant.
• Informed consent form explains
procedures and deals with ethical issues.
Some type of data
• Time:• to complete a task.
• time away from the product.
• Error
• Number and type of errors per task. • Number of errors per unit of time. • Number of navigations to online help or
manuals.
• Number of users making a particular error. • Number of users completing task successfully.
Usability engineering
orientation
• Aim is improvement with each
version.
• Current level of performance.
• Minimum acceptable level of
performance.
• Target level of performance.
How many participants is
enough for user testing?
• Depends on:
– schedule for testing; – availability of participants; – cost of running tests.
• Typically 5-10 participants.
• Some experts argue that testing
should continue until no new insights
are gained.
Field studies
• The aim is to understand what users do naturally and how technology impacts them.
• Field studies can be used in product design to:
- identify opportunities for new technology; - determine design requirements;
- decide how best to introduce new technology;
Data collection & analysis
• Observation & interviews
– Notes, pictures, recordings – Video
– Logging
• Analyzes
– Categorized
– Categories can be provided by theory
• Grounded theory • Activity theory
Key points
• Testing is a central part of usability testing. • Usability testing is done in controlled conditions.• Usability testing is an adapted form of experimentation.
• Experiments aim to test hypotheses by manipulating certain
variables while keeping others constant.
• The experimenter controls the independent variable(s) but not
the dependent variable(s).
• There are three types of experimental design:
different-participants, same- different-participants, & matched participants.
• Field studies are done in natural environments.
• Typically observation and interviews are used to collect field studies data.
• Categorization and theory-based techniques are used to
