There are two general categories of testing: pre-implementation and post- implementation testing. The first encompasses those activities that occur prior to placing the application system in an operational status. The objective of pre-implementation testing is to determine that the system functions as specified and that defects in the sys- tem are removed prior to placing the system into production. The second type of testing occurs after the system goes into operation and is normally considered part of systems maintenance.
The cost of removing system defects prior to the system going into production includes:
■■ Building the defect into the system ■■ Identifying the defect’s existence ■■ Correcting the defect
■■ Testing to determine that the defect has been removed
Defects uncovered after the system goes into operation generate the following costs: ■■ Specifying and coding the defect into the system
■■ Detecting the problem within the application system ■■ Reporting the problem to the project manager and/or user ■■ Correcting the problems caused by the defect
■■ Operating the system until the defect is corrected ■■ Correcting the defect
■■ Testing to determine that the defect no longer exists ■■ Integrating the corrected program(s) into production
Testing should include the cost to test plusthe cost of undetected defects. Few organi- zations consolidate all the named costs as testing costs; therefore, an organization rarely knows the true cost of testing. Testing is normally considered to be that process used to
find defects and ensure that the system functions properly. However, as illustrated, the cost of building and correcting defects may far exceed the cost of detecting those defects. The National Institute of Standards and Technology has estimated that testing, includ- ing the correction of defects prior to the application going into production, accounts for at least half of the total system development effort.
The high cost of system defects poses the following two challenges to organizations: how to quantify the true cost of removing defects, and how to reduce the cost of testing.
Quantifying the Cost of Removing Defects
Quality Assurance Institute surveys indicate that there are in the range of 20 to 60 defects in many application systems per 1,000 source statements. These surveys indi- cate that approximately two-thirds of the defects per 1,000 lines of source code occur in the requirements and design phases of application systems. Thus, while the defects are normally caught in the test phase of the system development life cycle, they occur early in the development process. (Note that as development processes mature, the number of defects produced is reduced.)
The causes of the defects built into application systems include:
■■ Improperly interpreted requirements.IT personnel misinterpret what the user wants, but correctly implement what the IT people believe is wanted.
■■ Users specify wrong requirements.The specifications given to IT personnel are erroneous.
■■ Requirements are incorrectly recorded.IT personnel fail to record the specifi- cations properly.
■■ Design specifications incorrect.The application system design does not achieve the system requirements, but the design as specified may be correctly implemented.
■■ Program specifications incorrect.The design specifications are incorrectly interpreted, making the program specifications inaccurate, but the program can be properly coded to achieve the correct program specifications. ■■ Program coding error.The program is not coded according to the program
specifications.
■■ Program structural or instruction error.The programming capabilities are improperly utilized, resulting in defects attributable to misuse of a program instruction or the method in which the instruction is used.
■■ Data entry error.The system and/or program information is incorrectly entered into the computer.
■■ Testing error.The test either detects an error where there is no error or fails to detect an existing error in the application system.
■■ Error correction mistake.In the process of correcting an error, the corrected condition contains a defect.
■■ Corrected condition causes another defect.In the process of correcting a defect, a defect occurs in the unchanged portion of the software.
The areas associated with the test process can usually be readily identified. It is the estimation of the costs associated with these areas that is difficult to obtain. However, until the total cost of testing is known, the cost of uncovering and correcting defects will be unknown.
There are two methods for developing a more realistic estimate of testing. The first is to ask IT personnel to identify all the preceding conditions and allocate their time and effort accordingly. Although this concept works in theory, in practice it is difficult to record the time and effort associated with incurring defects until that defect is actu- ally known. Because the point of uncovering defects may be many weeks or months after the actual day they were built into the system, it may be difficult to go back and recover these costs.
The second, and more practical, approach is to record the number of defects encoun- tered as a result of testing. As each defect is uncovered, it should be noted, as well as the point in the system development life cycle process where it was uncovered.
The actual cost to redesign and correct the system should then be recorded. These are the costs required for correcting the programs by some recompilation and change of documentation. The costs are then multiplied by a factor that represents the totality of the error and problems associated with the defect as follows:
■■ Defects corrected during design requirements phase.The cost to correct will be the total cost associated with the correction of the defect.
■■ Defects corrected during building the software. The cost to correct require- ment defects should be multiplied by a factor of 10 because the additional costs are associated with removing the defective components of the software. ■■ Defects corrected after the system goes into production.The cost to correct
will be approximately 100 times the cost to correct the same defect prior to placing the software into production.