We write test plans for two very different purposes. Sometimes the test plan is a product; sometimes it's a tool. It's too easy, but also too expensive, to confuse these goals. The product is much more expensive than the tool.
T
HE TEST PLAN AS A PRODUCTA good test plan helps organize and manage the testing effort. Many test plans are carried beyond this important role. They are developed as products in themselves. Their structure, format, and level of detail are determined not only by what's best for the effectiveness of the testing effort but also by what a customer or regulating agency wants. Here are some examples:
• Suppose your company makes a software-intense product for resale by a telephone company. (Call accounting programs and PBX phone systems are examples of such products.) Telephone compa nies know that they must support products they sell for many years. Therefore, they will scrutinize
f your test plan. They will demand assurance that your product was thoroughly tested and that, if they
need to take over maintenance of the software (e.g., if you go bankrupt), they'll be able to rapidly figure out how to retest their fixes. The test plan's clarity, format, and impressiveness are important sales features.
• If you sell software to the military, you also sell them (and charge them for) Mil Spec test plans. Otherwise, they won't buy your code.
• If you develop a medical product that requires FDA inspection, you'll create a test plan that meets very detailed FDA specifications. Otherwise, they won't approve your product.
• A software developer might choose to leverage the expertise of your independent test agency by having you develop a test plan, which the developer's test group will then execute without further help. You must write a document that is very organized and detailed, or your customer won't know how to use it.
Each of the above test plans is useful for finding bugs. However, it's important to note that in each case, if you could find more bugs in the time available by spending more time thinking and testing and less time writing an impressively formatted test plan, you would still opt for the fancy document (test plan) because the customer or the regulating agency requires it.
170
T
HE TEST PLAN AS A TOOLThe literature and culture of the traditional software quality community prepare readers and students to create huge, impressive, massively detailed test planning documents. Our major disagreement with the traditional literature is that we don't believe that creating such detailed documents is the best use of your limited time—unless you are creating them as products in their own right.
Look through standards like ANSI/IEEE 829 on test plan documentation. You'll see requests for test design specifications, test case specifications, test logs, test-various-identifiers, test procedure specifica- tions, test item transmittal reports, input/output specifications, special procedure requirements specifica- tions, intercase dependency notes, test deliverables lists, test schedules, staff plans, written lists of respon- sibilities per staffer, test suspension and resumption criteria, and masses of other paper.
Listen carefully when people tell you that standards help you generate the masses of paper more quickly. They do, but so what? It still takes a tremendous amount of time to do all this paperwork, and how much of this more-quickly generated paper will help you find more bugs more quickly?
Customers of consumer software ask for something that adds the right numbers cor- rectly, makes the right sounds, draws the right pictures, and types the text in the right places at the right times. They don't care how it was tested. They just care that it works. For these customers and many others, your test plan is not a product. It is an invisible tool that helps you generate test cases, which in turn help improve the product.
When you are developing a test plan as a tool, and not as a product, the criterion that we recommend for test planning is this:
A test plan is a valuable tool to the extent that it helps you manage your testing project and find bugs. Beyond that, it is a diversion of resources. . . _________________ __ ______________________________ __^_
As we'll see next, this narrowed view of test planning still leaves a wide range of functions that good testing documentation can serve.
DETAILED OBJECTIVES OF TEST PUNNING AND DOCUMENTATION
Good test documentation provides three major benefits, which we will explore in this section. The benefits are: • Test documentation facilitates the technical tasks of testing.
• Test documentation improves communication about testing tasks and process.
• Test documentation provides structure for organizing, scheduling, and managing the testing project. Few organizations achieve all potential benefits of their test plans. Certainly, anyone who writes a test plan gains at least some education about the test-relevant details of the product. But not every test group reviews test plans effectively or uses other project members' review feedback effectively. And many consult test plans only as technical documents, never using one to control a testing project or monitor project progress.
As a tester, you will spend many, many hours developing test plans. Given the investment, it's worth considering the potential benefits of your work in more detail. You may as well make the most of it.