Ideas arc tested now, not code. The "testers" (reviewers) include marketers, product managers, senior designers, and human factors analysts. Members of the Testing Group are rarely involved at this stage. (See Chapter 13 for useful planning-stage tasks for testers.)
The reviewers read drafts of the planning documents. Then they gather data, using comparative product evaluations, focus groups, or task analyses. These arc commonly described as planning and design tools, but they are also testing procedures: each can lead to a major overhaul of existing plans.
The reviewers should evaluate the requirements document (and the functional definition based on it) in terms of at least six issues:
• Are these the "right" requirements? Is this the product that should be built?
• Are they complete? Does Release 1 need more functions? Can some of the listed requirements be dropped?
• Are they compatible? Requirements can be logically incompatible (i.e., contradictory) or psycho logically incompatible. Some features spring from such different conceptualizations of the product that if the user understands one of them, she probably won't understand the othcr(s).
• Are they achievable? Do they assume that the hardware works more quickly than it does? Do they require too much memory, too many I/O devices, too fine a resolution of input or output devices? • Are they reasonable? There are tradeoffs between development speed, development cost, product performance, reliability, and memory usage. Are these recognized or do the requirements ask for lightning speed, zero defects, 6 bytes of storage, and completion by tomorrow afternoon? Any of these might be individually achievable, but not all at once, for the same product. Is the need for a priority scheme recognized?
• Are they testable? How easy will it be to tell whether the design documents match the requirements? If you go to a requirements review, evaluate the document in advance in terms of the questions above. Dunn (1984), Gause & Weinberg (1989), and ANSI/IEEE Standard 830 describe other problems to consider and questions to ask when reviewing requirements.
Having considered the general issues of interest to reviewers, consider the data collection tools: compara- tive product evaluations, focus groups, and task analyses.
36
TESTING DURING THE PUNNING STAGES
COMPARATIVE PRODUCT EVALUATIONS
C
OMPARATIVE PRODUCT EVALUATIONSIn the comparative product evaluation, the reviewer asks what will make this product different from others already on the market. What does the competition do better? Which of their features must be built into this product?
The reviewer uses working copies of competing products, demonstration versions, or published descriptions if that's all he can get He lists their features, their strengths and weaknesses, and anything about them noticed (praised or panned) in product reviews. He may categorize them in terms of the market segment to which they appeal, or the specific application for which they're best suited. He derives detailed profiles of competing products' capabilities, adding obvious "next steps" since these will probably come to market soon. He writes a similar profile for the planned product. How does it compare? Why would anyone want to buy it?
Initially, this evaluation leads to expansion of the requirements document and functional definition. The reviewer is tempted to design the ultimate product, packing into it the hundreds of good ideas gleaned from the competition. Unfortunately, it costs too much to include them all. Further, it's impossible to put them all into one cohesive product. Many features reflect fundamentally different conceptions of a product's task. They just don't work well together. Even with compatible features, as the feature set grows, so does the product' s complexity. At some point the product is so feature-laden that it's too hard to use, even though each feature is a good one in its own right. (Read Rubenstein & Hersh, 1984, and Norman, 1988.)
Some reviewers ignore problems of feature compatibility and complexity. They just generate a long list of competitors' good ideas. This can be a useful reference. However, before these are all tossed in as requirements, someone must prune the list. The reviewer may draft a much shorter list for this purpose, or he might submit the full list for review. Focus groups and task analyses can provide bases for much of the pruning from this list.
FOCUS GROUPS
A product is targeted toward specific market segments. The reviewer wants to know how they'll respond to it. The reviewer chooses a small group he considers representative of a market segment. Group members don't know each other. He asks them to discuss one or very few topics. He decides the topics, the scope, and focus of the discussion. He may moderate the discussion or he may hire a moderator. He does not participate as a discussant except, possibly, to ask the occasional question. His goal is to gauge current market reaction to an idea, not to convince these people of anything.
Focus groups can give feedback at many levels of generality. The reviewer might want an overview of what the group wants from this type of product, how they'll use it, and what features are most important. Or, he might focus on only one feature or one product application. He might use the group to generate ideas, before much detailed planning of the product has been done, or he may use it later, to test their reactions to details of the product plan.
37
T
ASK ANALYSESThe product automates or partially automates some task, probably a complex task. The analyst observes people doing their work, interviews them, tries to figure out all aspects of the task that the product will help them do. The analyst asks, what exactly is this task? How do people do it now, without the product? What order are subtasks done in? Why? When is what information needed and why? What are the bottlenecks in the work flow and why haven't they been solved? A task analysis is a design tool, vital for designing the user interface.
The task analysis might not be done until after the requirements seem settled. However, the results often challenge product requirements. They lead the analyst to simplify, combine, or eliminate features. New ones are born of the analyst's conceptualization of the job actually done by the users.
For more information on task analyses, see Bailey (1989), Card, Moran, & Newell (1983), Helander (1991, especially Chapter 38), Norman & Draper (1986, Section IV), and Rubenstein & Hersh (1984). See Baecker & Buxton (1987, e.g., Chapter 6) for interesting examples.