The formal machine tests after code integration

Further assume that the defect removal effectiveness of the two broad phases is the same. Define:

MP =Major problems found during reviews/inspections and unit testing (from phase 1); these are the problems that if not fixed, will result in testing defects or defects in the field.

PTR =Problem tracking report after code integration: errors found during formal machine tests.

m =MP/PTR, m>1 (Note: The higher the value of m, the more effective the front end.)

Q = Number of defects in the released software�defects found in the field (customer usage).

TD =Total defects for the life of the software = MP + PTR + Q. By definition of effectiveness:

Equation 6.1

Equation 6.2

By the assumption that the two phases have the same effectiveness:

Addison Wesley: Metrics and Models in Software Quality Engineering, Second Edition6.3 Defect Removal Effectiveness and Quality Planning

Thus,

Equation 6.3

Then,

Therefore, Equation 6.4

By the same token, it can be shown that:

Equation 6.5

Furthermore, from the definition of m:

Addison Wesley: Metrics and Models in Software Quality Engineering, Second Edition6.3 Defect Removal Effectiveness and Quality Planning

Therefore, Equation 6.6

Equations (6.4) through (6.6) can be useful for quality planning. The equations can be applied to absolute numbers as well as to normalized rates (e.g., defects per KLOC). Given the number of MP and m, or PTR and m, one can estimate the number of defects that remained in the product by Equations (6.4) and (6.5).

Also, assuming we use the lifetime defect rate (TD) of a predecessor product to approximate the TD of the product being developed, given a target product quality level to shoot for, Equation (6.6) can determine the value of m that we need to achieve in order to reach the target. Choosing a specific value of m determines how much focus a project should have on front-end defect removal. Once the m target is set, the team can determine the defect removal techniques to use (e.g., formal inspection, function verification by owner, team verifications, rigorous unit testing, etc.). For example, if we use the data from the example of Figure 6.4 (TD = 34.6 defects/KLOC, Q = 0.81 defects/KLOC for life of customer use), then the value of m should be:

This means that if the effectiveness is the same for the two phases, then the number of defects to be removed by the first phase must be at least 6.5 times of the number to be removed by testing in order to achieve the quality target. Note that the equations described in this section are valid only under the assumptions stated. They cannot be generalized. Although Equations (6.4) and (6.5) can be used to estimate product quality, this special case DRM is still not a projection model. The equal effectiveness assumption cannot be verified until the product defect rate Q is known or estimated via an independent method. If this assumption is violated, the results will not be valid.

I l@ve RuBoard

Addison Wesley: Metrics and Models in Software Quality Engineering, Second Edition6.3 Defect Removal Effectiveness and Quality Planning

I l@ve RuBoard

6.4 Cost Effectiveness of Phase Defect Removal

In addition to the defect removal effectiveness by phase per se, the cost of defect removal must be considered for efficient quality planning. Defect removal at earlier development phases is generally less expensive. The closer the defects are found relative to where and when they are injected, the less the removal and rework effort. Fagan (1976) contends that rework done at the I0, I1, and I2 inspection levels can be 10 to 100 times less expensive than if work done in the last half of the process (formal testing phases after code integration). According to Freedman and Weinberg (1982, 1984), in large systems, reviews can reduce the number of errors that reach the testing phases by a factor of 10, and such reductions cut testing costs, including review costs, by 50% to 80%. Remus (1983) studied the cost of defect removal during the three major life-cycle phases of design and code inspection, testing, and customer use (maintenance phase) based on data from IBM's Santa Teresa (California) Laboratory. He found the cost ratio for the three phases to be 1 to 20 to 82.

Based on sample data from IBM Rochester, we found the defect removal ratio for the three phases for the AS/400 similar to Remus's, at 1 to 13 to 92. Caution: These numbers may not be interpreted

straightforwardly because defects that escaped to the later testing phases and to the field are more difficult to find. When we invest and improve the front end of the development process to prevent these more difficult defects from escaping to the testing phases and to the field, the ratios may decrease.

Nonetheless, as long as the marginal costs of additional front-end defect removal remains less than testing and field maintenance, additional investment in the front end is warranted.

Our sample study also revealed interesting but understandable findings. The cost of defect removal is slightly higher for I0 inspection than for I1 and I2 (Figure 6.6). The main reason for this is that external interfaces are being impacted and more personnel are involved in the I0 inspection meetings. The cost for creating and answering a problem trouble report during testing (i.e., problem determination cost) is

correlated with the testing phase, defect origin, and defect severity (1 being the most severe and 4 the least) (Figure 6.7).

Figure 6.6. Cost of Defect Removal by Inspection Phase

Figure 6.7. Cost of Creating and Answering a Problem Trouble Report by Several Variables Addison Wesley: Metrics and Models in Software Quality Engineering, Second Edition6.4 Cost Effectiveness of Phase Defect Removal

In his work on software inspection, Gilb (1993, 1999) conducted thorough analysis with ample data. The findings corroborate with those discussed here and support the general argument that software

inspection not only improves the quality of the product, but also is beneficial to the economics of the project and the organization.

Although front-end defect removal activities in the form of reviews, walk-throughs, and inspections are less expensive than testing, in general practice, these methods are not rigorous enough. Fagan's inspection method is a combination of a formal review, an inspection, and a walkthrough. It consists of five steps:

1. overview (for communications and education) 2. preparation (for education)

3. inspection (to find errors and to walk through every line of code)