The last point to realize about program debugging is that in addition to its value in removing an error from the program, it can have another valuable effect: It can tell us something about the nature of software errors, some- thing we still know too little about. Information about the nature of software errors can provide valuable feedback in terms of improving future design, coding, and testing processes.
Every programmer and programming organization could improve im- mensely by performing a detailed analysis of the detected errors, or at least a subset of them. Admittedly, it is a difficult and time-consuming task, for it implies much more than a superficial grouping such as ‘‘x percent of the errors are logic design errors,’’ or ‘‘x percent of the errors occur in IF statements.’’ A careful analysis might include the following studies:
Where was the error made? This question is the most difficult one to answer, because it requires a backward search through the documen- tation and history of the project; at the same time, it also is the most valuable question. It requires that you pinpoint the original source and time of the error. For example, the original source of the error might be an ambiguous statement in a specification, a correction to a prior error, or a misunderstanding of an end-user requirement. Who made the error? Wouldn’t it be useful to discover that 60 percent of the design errors were created by one of the 10 analysts, or that programmer X makes three times as many mistakes as the other pro- grammers? (Not for the purposes of punishment but for the purposes of education.) What was done incorrectly? It is not sufficient to determine when and by whom each error was made; the missing link is a determination of exactly why the error occurred. Was it caused by someone’s inability to write clearly? Someone’s lack of education in the programming lan- guage? A typing mistake? An invalid assumption? A failure to con- sider valid input? How could the error have been prevented? What can be done differently in the next project to prevent this type of error? The answer to this 172 The Art of Software TestingC08 08/17/2011 1:8:16 Page 173
question constitutes much of the valuable feedback or learning for which we are searching.
Why wasn’t the error detected earlier? If the error was detected during a test phase, you should study why the error was not unearthed during earlier testing phases, code inspections, and design reviews. How could the error have been detected earlier? The answer to this offers another piece of valuable feedback. How can the review and testing processes be improved to find this type of error earlier in fu- ture projects? Providing that we are not analyzing an error found by an end user (that is, the error was found by a test case), we should realize that something valuable has happened: We have written a suc- cessful test case. Why was this test case successful? Can we learn something from it that will result in additional successful test cases, either for this program or for future programs?We repeat, this analysis process is difficult, and costly, but the answers you may discover by going through it can be invaluable in improving subsequent programming efforts. The quality of future products will increase while the capital investment will decrease. It is alarming that the vast majority of programmers and programming organizations do not employ it.
Summary
The main focus of this book is on software testing: How do you go about uncovering as many software errors as possible? Therefore, we don’t want to spend too much time on the next step—debugging—but the simple fact is, errors found by successful test cases lead directly to it.
In this chapter we touched on some of the more important aspects of software debugging. The least desirable method, debugging by brute force, involves such techniques as dumping memory locations, placing print statements throughout the program, or using automated tools. Brute-force techniques may point you to the solution for some errors uncovered dur- ing testing, but they are not an efficient way to go about debugging.
We demonstrated that you can begin debugging by studying the error symptoms, or clues, and moving from them to the larger picture (inductive debugging). Another technique begins the debugging process by consider- ing general theories, then, through the process of elimination, identifies
the error locations (deductive debugging). We also covered program backtracking—starting with the error and moving backwards through the program to determine where incorrect information originated. Finally, we discussed debugging by testing.
If, however, we were to offer a single directive to those tasked with debugging a software system, we would say, ‘‘Think!’’ Review the numer- ous debugging principles described in this chapter. We believe they can lead you in the right direction, toward accurate and efficient debugging. But the bottom line is, depend on your expertise and knowledge of the program itself. Open your mind to creative solutions, review what you know, and let your knowledge and subconscious lead you to the error locations.
In the next chapter we take on the subject of extreme testing, tech- niques well suited to help uncover errors in extreme programming envi- ronments such as agile development.
C09 08/26/2011 12:10:53 Page 175
9
Testing in the Agile
Environment
I
ncreased competition and interconnectedness in all markets have forced businesses to shorten their time-to-market while continuing to provide high-quality products to their customers. This is particularly true in the soft- ware development industry where the Internet makes possible near-instant delivery of software applications and services. Whether creating a product for the masses or for the human resources department, one fact remains im- mutable: The twenty-first century customer demands a quality application delivered almost immediately. Unfortunately, traditional software develop- ment processes cannot keep up in this competitive environment.In the early 2000s, a group of developers met to discuss the state of light- weight and rapid development methodologies. At the gathering they com- pared notes to identify what successful software projects look like; what made some projects succeed while others limped along. In the end, they created the ‘‘Manifesto for Agile Software Development,’’ a document that became the cornerstone of the Agile movement. Less a discrete methodol- ogy, the Agile Manifesto (Figure 9.1) is a unique philosophy that focuses on customers and employees, in lieu of rigid approaches and hierarchies.