Product Development and Usability Testing
PRODUCT DEVELOPMENT, EFFICACY TESTING, AND COMPARISON TESTING
OF AN ASSISTIVE WALKER
Given that therapists have accepted the job as consultants and members of the ergonomic evalu-ation team, they fi rst review the literature and construction of walkers and refamiliarize them-selves with the types of clients who use them.
They review the accoutrements that users may want, such as baskets, pouches for carrying small items, and drink holders. They examine the balance characteristics of walkers. Some are bal-anced at the center handle; these walkers are designed for clients with hemiplegia and thus with limited use of one hand. Wheeled walkers may be especially benefi cial during the early rehabilita-tion process, but it is diffi cult to know whether one with front wheels only or one with three wheels will best serve a client. Other important features are the weight, portability, and stability of the walker and the height, shape, and size of the grip handles. Some clients may want a walker with an attached seat.
Given that the New Equipment Company has an idea for a new walker design, the team decides to start there. They plan for three iterations of the usability process. During the fi rst iteration (product development), several variations of the new walker design will be constructed and evaluated.
This is prototype or pilot testing, which involves the evaluation of a newly developed trial product by the end-users who represent the target market.
Both the walker design and the testing process are evaluated. The information gained from the pilot test is used in the second iteration of the usability process, in which the best walker design (as deter-mined during the pilot test) is evaluated (effi ca-cy testing). This phase involves a more formal process of performance testing in a controlled setting to determine the effectiveness of the new walker. The fi nal phase (comparison or fi eld testing) involves a fi eld study to determine user acceptance and performance. This testing is con-ducted in a setting similar to the environment in which the walker will be used (see Figure 10-2, phase 3).
Each phase is considered part of the usability testing. Usability testing means that the product
is evaluated by obtaining information from repre-sentative users, often while they use the product.
To reiterate, the goals of usability testing are to develop a product that accomplishes the purpose for which it was designed, is easy and safe to use, and will be used.
Creating a product that will be utilized involves other factors, such as aesthetics, that infl uence whether a person chooses to use the product. In addition, the best design is one that does not require the user to study an instruction manual;
instead, the design should guide the user’s actions so that use of the product is intuitive.
First Iteration: Product Development
The goal of product development is to produce several design alternatives and to select one for additional evaluation. The fi rst step is to identify the SMEs, users, and investigators (see Figure 10-2). This group could include product developers, medical personnel who have prescribed walkers for clients, therapists and nurses who work closely with clients who use walkers, family members of clients who use walkers, and the clients them-selves. A target group of clients should be identi-fi ed, because the needs of various groups, such as those with hemiplegia and those with cerebral palsy, differ. For example, a client who has prob-lems with balance and coordination may not want wheels on his or her walker, and a client who quickly becomes fatigued may need an attachable seat that folds while he or she is walking. Identi-fi cation of a target group should be based on demographics; knowledge, skills, and experience;
attitude; lifestyle; cognitive and physical abilities;
and cultural background. In the case study, New Equipment Company wants to create several walkers for different populations; therefore more than one target group would be identifi ed and involved in testing. It may be that New Equipment Company has already had their marketing depart-ment identify the groups that will have the largest population that could benefi t from, and would be likely to purchase, their walkers over the next 25 years.
The second step is the interactive process be-tween the investigators and the SMEs and users (see Figure 102). During this interaction, defi
-Ch010-A04853.indd 161
Ch010-A04853.indd 161 6/8/2007 3:08:14 PM6/8/2007 3:08:14 PM
ciencies in existing walker designs are identifi ed and consequent research questions are developed.
Positive aspects of existing walker designs may also be identifi ed and incorporated into the design objectives (see Figure 10-2, step 3). If the produc-ers of the equipment have different aims, these also need to be identifi ed. Such aims could include high sales, marketability, production loca-tion or costs, and user educaloca-tion or manual development.
Design objectives are developed as a result of the observations, replies to questionnaires, and discussions among SMEs, users, and investigators (Box 10-4). Design objectives should include any items considered important to enable full, practi-cal use of the walker. The development of design objectives should answer the question, “What
design features are important for a walker to be used by this target population?” The purpose of a walker is to assist people in walking by allowing them to stabilize themselves by putting some of their weight on the walker handles. Thus, the fi rst objective should be stability. Secondary character-istics of the design are those that are important to a user but that may not infl uence the primary purpose of the product. An example is making the walker easily collapsible for placing into a car or storage area (Figure 10-3). Tertiary items include attractiveness and convenience. Conve-nience characteristics of a walker might include baskets or pouches for personal items and attach-able trays to hold food or drinks.
Labeling design objectives as primary, second-ary, and tertiary does not mean one level is more important than another. Secondary and terti-ary items are important because they infl uence whether the product will be accepted and used. A BOX 10-4 Design Objectives for Product
Development
Primary Walker Lightweight Adjustable height Adjustable width Stability
User
Appropriate weight distribution
Ability to maintain erect posture during use
Secondary Comfort Ease of use Ease of adjustment Ease of storage Portability
Optimum grip height Shape
Size
Tertiary Attractiveness Convenience
FIGURE 10-3 Being able to fold a walker for storage may be important to a person who does not need to use it to walk for short distances or on a daily basis.
Ch010-A04853.indd 162
Ch010-A04853.indd 162 6/8/2007 3:08:14 PM6/8/2007 3:08:14 PM
product may help clients accomplish a task but be so diffi cult, inconvenient, or unattractive to use that people choose to do without it. The impor-tance of individual design objectives should be determined by the combined interaction of the SMEs, users, and investigators.
While design objectives are being defi ned, a task and function analysis should be accomplished (see Figure 10-2, step 4). Information gained from establishing the design objectives should be used in conducting the task and function analysis and vice versa. The task and function analysis is based on input from users and SMEs. The investigator who conducts the assessment should observe the user performing a typical task and break the task into its component parts. These components should be described using action phrases. The design objectives and the information gained from the task and function analysis are used to develop performance criteria (see Figure 10-2, step 5).
Representative tasks are identifi ed on the ba-sis of criticality, frequency, and diffi culty. The selected tasks can be used as independent vari-ables (the different walkers are also independent variables). For this situation, the tasks chosen could include walking and maneuvering around items that block the user’s path; entering, using, and exiting a restroom; and using a small set of stairs. The fi rst task is used to test the walker prototypes. Performance criteria are developed from the selected task or tasks.
The sixth step is to establish subjective and objective measurements. Because the fi rst itera-tion is the development phase, the investigator may decide to use only one task to select the new design for the walker. Similarly, the investigating team may choose to use only the design objectives deemed most important. The breadth and depth of the evaluation during the product-development stage are determined by the investigator or inves-tigating team. Consideration of costs and benefi ts assist the investigator in making the determina-tion. For example, if construction of the walkers for additional testing is expected to be expensive, the testing should be thorough. If construction and possible alterations are relatively inexpensive, the prototype study may be smaller in terms of breadth and depth (or complexity).
Design objectives (dependent measurements) require both objective and subjective measure-ments. Dependent measurements for the sample situation are listed in Box 10-5. In addition to the measurements listed, the base and depth of the walker should be measured to determine walker stability. Many manufacturers list the weight capacity of walkers. If more information is required, however, material strength can be deter-mined through consultation with an engineer familiar with the materials and construction of walkers. Subjective measurement techniques may include interviews, questionnaires, rankings, Likert scale ratings, or ratings by means of tech-niques such as magnitude estimation (Box 10-6).1,11 Group interviews, rather than open-ended individual interviews, are often used to promote discussion.20 Forced-choice rankings, especially useful in the comparison of several designs, re-quire the user to rank the designs in order of preference. Observations and ratings by the inves-tigator can be helpful, but the invesinves-tigator must take care not to bias the results.
BOX 10-5 Dependent Measurements for Product Development
Objective Walker weight Height adjustment
Percentage of the target population that can use the walker
Distance between walker legs
Biomechanical analysis of weight distribution Material strength
Subjective Perceived stability Perceived comfort Perceived pain or strain Perceived exertion Perceived ease of use Perceived ease of adjustment Perceived portability
Forced-choice rankings
Ch010-A04853.indd 163
Ch010-A04853.indd 163 6/8/2007 3:08:14 PM6/8/2007 3:08:14 PM
Subject training and a walk-through of the testing process constitute the seventh step (see Figure 10-2, step 7). Enough training should be done to eliminate a learning (or practice) effect.
Subjects should not continue to improve with time, regardless of experimental condition.
The eighth step is the actual assessment; in this case it involves a comparison study of several prototype walkers. Subjects perform one or more of the reference tasks, and the investigator collects and analyzes objective and subjective informa-tion. On the basis of the analysis, one design is usually selected for the next phase, effi cacy test-ing. In the example, the walking task is evaluated in a nonexperimental, formal-informal, three-dimensional, and performance-oriented context.
Nonexperimental means no statistical controls, even though contrasting conditions are used (one walker design compared with another). The com-parison study contains both formal and informal elements: A formal procedure and questionnaire are used in addition to an informal interview session. The process is three-dimensional because prototypes of the walkers are used in a realistic task or series of tasks.
The goal of the evaluation (to identify one walker for additional testing) can be met with a relatively small number of participants. Subjects receive a detailed briefi ng, undergo a medical screening, and sign an informed-consent form.
Each hospital or nursing facility usually has a human-use committee that determines the require-ments for briefi ng, screening, and the format and contents of the consent form.
The experimental design is a repeated-mea-surements design, counterbalanced for order. The term repeated-measurements design means that each subject serves as his or her own control and completes the task under each of the experimental conditions (various walker designs). Counterbal-ancing for the order in which each walker is used can be accomplished by using a balanced Latin-square design. This means each treatment condi-tion (each walker design) is immediately preceded and followed once by each of the other condi-tions.31 (This is often the preferred method to counterbalance a design without having to conduct tests of all possible ordering combinations.) Another method of controlling for order effects is to randomize the order of administration.
Analysis of the subjective data can be accom-plished by the use of nonparametric statistical analysis.26,31 Nonparametric statistical analysis is a useful tool for usability studies that collect sub-jective data and use small sample sizes. Paramet-ric statistical analysis can be used for objective data when proper experimental design and suffi -cient population sampling are used. Considerable debate exists about using parametric statistics with subjective data.2,15,29
The results should clearly indicate the pre-ferred design on the basis of user preference and performance data. The investigator may give a weighting factor to items considered to be of primary importance. For example, object load, adjustability, use by the greatest percentage of the target population, and biomechanical advantage may be weighted more than convenience and aesthetics.
As a result of the fi rst iteration, design 1 is selected for additional testing. As seen in Table 10-1, the design is selected because it has the largest height range and is considered the most stable, adjustable, and portable. Its use caused the least pain and strain, and it was ranked the pre-ferred walker. Note that design 1 was selected despite being the most diffi cult to use.
BOX 10-6 What Is Magnitude Estimation?
Magnitude estimation is an experimental tech-nique used in psychophysical experiments. Mag-nitude estimations involve having a subject compare a current sensation with a reference sensation. For example, a subject might be asked to handle a box of a particular weight and then be asked to judge other weights as weighing more or less than the reference weight. Another example would be the comparison of tactile pres-sures administered by a monofi lament as being of greater or lesser sensation.
Ch010-A04853.indd 164
Ch010-A04853.indd 164 6/8/2007 3:08:14 PM6/8/2007 3:08:14 PM
Second Iteration: Effi cacy Testing (Controlled Setting)
The goal of effi cacy testing is to determine whether the walker improves the user’s ability to walk and maneuver through the activities of daily living—
that is, it answers the question of whether the walker is effective for completing the tasks the user needs to complete. Therefore, testing consists of having subjects use the walker, as opposed to not using a walker, while performing several rep-resentative tasks. If the investigator believes that walkers have been shown to be effective ambula-tion tools and that such an evaluaambula-tion would be superfl uous, this phase can be eliminated. If this phase is eliminated, usability testing begins with a comparison between the new design and exist-ing designs (usability [comparison] testexist-ing; see Figure 10-2, phase 3).
Identifi cation of the SMEs and users was accomplished in the beginning of phase 1 (pilot testing); the experimental subjects now are added to the group as SMEs (see Figure 10-2, phase 2, step 1). The interaction among SMEs, users, and
the investigator should focus on the results of the pilot test accomplished during phase 1.
The design objectives for the walker most likely will remain the same as those identifi ed in the development phase (see Figure 10-2, phase 2, step 3; Box 10-4). However, additional objectives can be identifi ed in the pilot testing and in the interac-tions among the subjects, SMEs, and users.
The task and function analysis should be re-evaluated (see Figure 10-2, phase 2, step 4). The representative tasks can be altered on the basis of information gained during the development phase.
For the second iteration of the process (effi cacy testing), all three representative tasks are used to ascertain whether the new walker meets the functional goals. The tasks identifi ed during the task or functional analysis are walking and maneu-vering around items that block the user’s path;
entering, using, and exiting a restroom; and using a small set of stairs. Each task is completed in a controlled laboratory setting. In each task, per-formance criteria should provide information essential to successful performance and include TABLE 10-1 Hypothetical Results from Product Development
Design 1 Design 2 Design 3
Load 6 lb 7 lb 16 lb
Height 17-37 in 32-37 in 30-38 in
Weight distribution Good Good Good
Material construction 350-lb capacity 375-lb capacity 500-lb capacity
Posture Good Good Good
Stability 17.5* 12.2 14
Comfort 14.5 15.8 16
Pain or strain 12.1* 14.2 14.6
Ease of use 10.3 12.2 16.8*
Ease of adjustment 18.7* 16 14.8
Portability 16.5* 13.9 9.8
Ranking 1.25* 2.25 2.5
*Signifi cantly different from other two walkers (P < .05).
Note: All ratings (except ranking) used a Borg-type scale with anchored subjective ratings of 0 to 20.8 The lower number indicates less and the higher number indicates more of the given quality. Rankings were 1 to 3.
Ch010-A04853.indd 165
Ch010-A04853.indd 165 6/8/2007 3:08:14 PM6/8/2007 3:08:14 PM
objective and subjective data (see Figure 10-2, phase 2, step 5). When the same criteria are used for product development, effi cacy testing, and comparison testing, performance standards can be developed and product improvement can be mon-itored. Additional dependent measurements in the example include time to complete each ele-ment of the task, time to complete the entire pro-cedure, heart rate, and perceived exertion (Table 10-2).7-9
In the example, the same objective and subjec-tive measurement techniques used during the development phase are used during effi cacy testing (see Figure 2, phase 2, step 6; Box 10-5). The fi rst task is walking and maneuvering around items that block the user’s path and involves the following procedures: rising from
an easy chair, turning right, walking 5 feet and maneuvering to the left of a chair that blocks the path, walking 4 feet and maneuvering right to avoid a child’s toy, walking another 5 feet, and sitting in a kitchen chair.
In addition to the primary task of walking, important secondary tasks should be included in the testing procedure. For example, if the walker is used to enable someone to move between a desk and a fi ling cabinet, such a task pattern should be incorporated into the testing pro-cedure.
Again, subjects should be trained in each task used in the test procedure (see Figure 10-2, phase 2, step 7). Because more than one task is being studied (walking, maneuvering in a restroom, and using stairs), the order of the tasks should be bal-anced to control for order effects, such as transfer of learning or a conditioning effect. Training of test subjects in testing procedures also decreases the likelihood that learning effects will infl uence the study results.
After training, the actual assessment (experi-mental evaluation) takes place. Task performance should be evaluated by timing and accuracy data.
In the example, effi cacy testing is experimental, formal, three-dimensional performance testing.
As with any research method, consistency in experimental testing must be ensured in subject training, measurement techniques, and data com-pilation. Two excellent resources on these topics are Winer and colleagues31 for laboratory studies and Cook and Campbell10 for fi eld studies.
During effi cacy testing, the number of subjects
During effi cacy testing, the number of subjects