How do you plan usability testing so that all aspects of the software are reasonably tested while staying within an acceptable budget? The answer to that question, of course, depends in part on the complexity of the sys- tem or unit being tested. If budget and time allow, it is advisable to test software in stages, as each segment is completed. If individual components have been tested throughout the development process, then the final series of tests need only test the integrated operation of the parts.
Additionally, you may designcomponent tests, which are intended to test the usability of an interactive component, something that requires user in- put and that responds to this input in a user-perceivable way. This kind of feedback testing can help improve the user experience, reduce operational errors, and improve software consistency. Again, if you have tested a soft- ware system at this level as the user interface was being designed, you will have collected a significant body of important testing and operational knowledge before total system testing begins.
How many individual users should test your software? Again, system complexity and initial test results should dictate the number of individual testers. For example, if three or five (or some reasonable number) of users have difficulty navigating from the opening screen to screens that support the assigned tasks, and if these users are sufficiently representative of the target market, then you likely have enough information to tell you that the user interface needs more design work.
A reasonable corollary to this might be that if none of the initial testers have a problem navigating through their assigned tasks, and none uncover any mistakes or malfunctions, then perhaps the testing pool is too small. After all, is it reasonable to assume that usability tests of a reasonably com- plex software system will uncover no errors or required changes? Recall principle 6, from Table 2.1: Examining a program to see if it does not do what it is supposed to do is only half the battle; the other half is seeing whether the program does what it is not supposed to do. There’s a subtle difference in this comparison. You might find that a series of users determine that a pro- gram does, in fact, seem to do what it is supposed to do. They find no errors or problems in working through the software. But have they also proven that the program isn’t doing anything it is not supposed to do? If things appear to be running too smoothly during initial testing, it probably is time for more tests.
We don’t believe there is a formula that tells you how many tests each user should conduct, or how many iterations of each test should be re- quired. We do believe, however, that careful analysis and understanding of the results you gather from some reasonable number of testers and tests can guide you to the answer of when enough testing is enough.
Summary
Modern software, coupled with the pressure of intense competition and tight deadlines, make user testing of any software product crucial to suc- cessful development. It stands to reason that the targeted software user can be a valuable asset during testing. The knowledgeable user can determine whether the product meets the goal of its design, and by conducting real- world tasks can find errors of commission and omission.
Depending on the software target market, developers also may benefit from selecting random users—persons who are not familiar with the pro- gram’s specification, or perhaps even the industry or market for which it is intended—who can uncover errors or user interface problems. For the same reason that the developers don’t make good error testers, expert users may avoid operational areas that might produce problems because they know how the software is supposed to work. Over many years of software development we have discovered one unavoidable testing truth: The soft- ware the developer has tested for many hours can be broken easily, and in a 154 The Art of Software Testing
C07 08/29/2011 14:50:12 Page 155
short time, by an unsophisticated user who attempts a task for which the user interface or the software was not designed.
Remember, too, that a key to successful user (or usability) testing is ac- curate and detailed data gathering and analysis. The data-gathering pro- cess actually begins with the development of detailed user instructions and a task list. It ends by compiling results from user observation or post- test questionnaires.
Finally, the testing results must be interpreted, and then developers must effect software changes identified from the data. This may be an itera- tive process wherein the same testing users are asked to complete similar tasks after identified software changes have been completed.
C08 08/17/2011 1:8:14 Page 157
8
Debugging
I
n brief, debugging is what you do after you have executed a successful test case. Remember that a successful test case is one that shows that a programdoes not do what it was designed to do. Debugging is a two-step process that begins when you find an error as a result of a successful test case. Step 1 is the determination of the exact nature and location of the suspected error within the program. Step 2 consists of fixing the error.As necessary and integral as debugging is to program testing, it seems to be the one aspect of the software production process that programmers enjoy the least, for these reasons primarily:
Your ego may get in the way. Like it or not, debugging confirms that programmers are not perfect; they commit errors in either the design or the coding of the program. You may run out of steam. Of all the software development activities, debugging is the most mentally taxing activity. Moreover, debugging usually is performed under a tremendous amount of organizational or self-induced pressure to fix the problem as quickly as possible. You may lose your way. Debugging is mentally taxing because the error you’ve found could occur in virtually any statement within the program. Without examining the program first, you can’t be abso- lutely sure, for example, that the origin of a numerical error in a pay- check produced by a payroll program is not a subroutine that asks the operator to load a particular form into the printer. Contrast this withthe debugging of a physical system, such as an automobile. If a car stalls when moving up an incline (the symptom), you can immedi- ately and validly eliminate as the cause of the problem certain parts of the system—the AM/FM radio, for example, or the speedometer or the trunk lock. The problem must be in the engine; and, based on our overall knowledge of automotive engines, we can even rule out certain engine components such as the water pump and the oil filter.
You may be on your own. Compared to other software development activities, comparatively little research, literature, and formal instruc- tion exist on the process of debugging.Although this is a book about software testing, not debugging, the two processes are obviously related. Of the two aspects of debugging, locating the error and correcting it, locating the error represents perhaps 95 percent of the problem. Hence, this chapter concentrates on the process of finding the location of an error, given that a successful test case has found one.