Testing & Debugging

A detailed treatment of the technical aspects of testing is beyond the scope of this text (see for ex- ample [BN03] for more information). Even so, there are some general issues about testing we would like you to keep in mind.

4.1. DEVELOPMENT CYCLE 95

First of all, you should be aware of the fact that after you have developed a compilable piece of code, the work is not done yet. You might even say that it has just begun. What comes next is the very important and often time-consuming task oftestinganddebugging the software, which makes up a large portion of the overall development cycle. Testing is performed with the aim to check whether the tested system meets its specification. Detected deviations from the specification may result in debugging the program code (if its cause was an implementation error), but may even instigate a complete redesign of the project in case of a design flaw.

It is immediately apparent that testing is important, even more so in safety-critical applications. However, it is also a fact that barring the use of formal verification at all stages (including a formally proven specification!) in conjunction with either automatic code generation or exhaustive testing, testing and debugging does not guarantee the absence of bugs from your software. It only (hopefully) removes bugs that show up in the tests, preferably without introducing any new bugs in the process. The higher the test coverage, the more bugs are found. On the other hand, the longer the testing and debugging phase, the longer thetime-to-market, which has a direct impact on the financial gain that can be expected from the product. Figure4.3 roughly sketches the relationship between debugging time and the percentage of errors remaining in the code.

0 10 20 30 40 50 60 70 80 90 100

remaining % bugs in code

time

Figure 4.3: Relationship of debugging time and percentage of errors still in the code.

As you can see, in the initial stages of the testing phase, a lot of bugs are found and removed in a short amount of time. After these easy to find bugs have been removed, however, it grows more and more difficult to find and eliminate the remaining errors in the code. Since80/20 rulesare very popular, there is one for the debugging process as well: The final 20% of the bugs cost 80% of the money spent for debugging. In the light of these figures, it is only natural for companies to enforce a limit on the time spent for debugging, which in turn influences the percentage of bugs remaining in the system. This limit depends on the target field of application, with safety-critical systems putting the highest demands on the testing and debugging process (using formal verification methods and automatic testing).

Testing and debugging is not just done on the final product, but should be performed in the early stages of implementation as well. As we have seen, the sooner a bug is caught the better. In con- sequence, modular design is important, because it facilitates testing. Testing concerns should also be considered during and incorporated into the design of the product. Both bottom-up and top-down testing are feasible strategies. In both cases, the application (which is on the highest level, on thetop) is broken into modules, which are again composed of sub-modules and so on. Inbottom-up testing,

the components on the lowest level, which are not broken down any further, are tested first. After that, the module which is formed by them is tested, and so on until the finalintegration testof the whole application, which tests the interworking of the modules. Intop-down testing, the sub-modules of a module are emulated by so-called stubs, which are dummy implementations with the sole purpose of providing adequate behavior to allow testing the module. Testing then starts at the top and moves down until the lowest level is reached. The top-down strategy has the advantage that the application itself can be tested at an early stage. Since a design error on a high level most likely affects the levels below and can even instigate a complete redesign, finding such errors soon saves a lot of time. How- ever, this approach requires the implementation of stubs and does not remove the need to do additional integration tests after the sub-modules become available. The bottom-up strategy does not need stubs, but high-level modules can only be tested after all sub-modules are available and tested. Note that the usage of stubs allows any module, on any level, to be tested at an early stage. So a hybrid approach could be implemented, testing low-level modules as soon as they are finished, while in the meantime testing crucial high-level modules with stubs.

Note that in any of the strategies, it is not sufficient to test the modules stand-alone. Integration tests must be performed to see if the modules correctly work together, and if any sub-module is changed, it and all modules affected by the change must be tested as well.

Finally, do not underestimate the value of good code documentation for avoiding and also finding bugs. Good documentation of the code forces the software engineer to think about what he or she is doing, about expectations placed upon the hardware and software. Not only does this help the software engineer focus on what needs to be done, it also facilitates debugging because the initial expectations can be compared to the real outcome step by step until the error is found.