If a tracking system is established, then gather data about defect origins (in addition to where defects were found) and start implementing phase-specific effectiveness metrics to

guide specific improvements.

The combination of where-found and defect-origin data is useful for many occasions, not just for phase effectiveness calculation. For example, defect cause analysis including phase origin during the testing phase of the project often provides clues for further actions before shipping the product. However, tracking both wherefound and phase-origin data for all defects does require additional resources.

If resources are severely constrained and data tracking for the front-end of the process (e.g., requirements, design reviews, and code inspections) is not available, then use the test effectiveness metric. But in terms of improvement actions, a focus on the entire development process is important and strongly recommended. Indeed, even for organizations with more mature metrics programs, tracking and data for the front-end of the development process normally are less rigorous than the testing phases. For small organizations starting a metrics program with minimum resources and with the intent to keep the number of metrics to the minimum, using the test effectiveness indicator for measurement while maintaining the overall process focus on requirements, design, coding, and testing is a good strategy. With or without metrics, the importance of a strong focus on requirements, designs, and reviews can never be overstated. Good requirements-gathering and analysis techniques can reduce the volume of requirements changes and secondary requirements significantly.

Addison Wesley: Metrics and Models in Software Quality Engineering, Second Edition6.5 Defect Removal Effectiveness and Process Maturity Level

Good design and programming practices with effective defect prevention and removal are crucial for the stability and predictability of the back end of the development process. Projects that bypass design and code inspections may seem to be ahead of schedule�until testing begins. When testing begins, a deluge of unexpected errors could bring the project to a standstill, and the development team can get locked into a cycle of finding bugs, attempting to fix them, and retesting that can stretch out for months.

I l@ve RuBoard

Addison Wesley: Metrics and Models in Software Quality Engineering, Second Edition6.5 Defect Removal Effectiveness and Process Maturity Level

I l@ve RuBoard

6.6 Summary

Effective defect removal during the development process is central to the success of a software project.

Despite the variations in terms of terminology and operational definitions (error detection efficiency, removal efficiency, early detection percentage, phase defect removal effectiveness, phase defect containment effectiveness, etc.), the importance of the concept of defect removal effectiveness and its measurement is well recognized. Literature and industry examples substantiate the hypothesis that effective end defect removal leads to improved quality of the end product. The relative cost of front-end defect removal is much lower than the cost of formal testing at the back front-end and the maintenance phase when the product is in the field.

To measure phase defect removal effectiveness, it is best to use the matrix approach in which the defect data are cross-tabulated in terms of defect origin and the phase in which the defects are found. Such an approach permits the estimation of phase defect injection and phase defect removal. In general, the shorter the time between defect origin and defect discovery, the more effective and the less expensive the development process will be. The special case of the two-phase defect removal model even provides a link between the relative effectiveness of front-end defect removal and the estimated outcome of the quality of the product.

Based on recent studies, defect removal effectiveness by the level of process maturity has been

assessed and comparison baselines have been established. Organizations with established data on their defect removal effectiveness can make comparisons to the baselines and assess their maturity level with regard to this important parameter.

In quality planning it is important that, in addition to the final quality goals, factors such as the defect model, the phase defect removal targets, the process and specific methods used, the possible effectiveness of the methods, and so forth be examined. Inclusion of these factors in early planning facilitates achievement of the software's quality goals.

It should be noted that defect removal effectiveness and defect removal models are useful quality

planning and management tools. However, they are not equipped for quality or reliability projections; they are not predictive models. In the next several chapters, we discuss the parametric models that were developed to perform such tasks.

I l@ve RuBoard

Addison Wesley: Metrics and Models in Software Quality Engineering, Second Edition 6.6 Summary

I l@ve RuBoard

References

1. Daskalanatonakis, M. K., "A Practical View of Software Measurement and Implementation Experiences within Motorola,"IEEE Transactions on Software Engineering, Vol. SE-18, 1992, pp.

998�1010.

2. Dunn, R. H., "The Quest for Software Reliability," Handbook of Software Quality Assurance, G. G.

Schulmeyer and J. I. McManus, Eds., New York: Van Nostrand Reinhold, 1987, pp. 342�384.

3. Curtis, B., "Quantitative Process Management and Process Capability Baselines," SEPG 2002 QM Tutorial, Phoenix, Arizona, SEPG 2002, February 18�21.

4. Diaz, M., and J. King, "How CMM Impacts Quality, Productivity, Rework, and the Bottom Line,"

CrossTalk, The Journal of Defense Software Engineering, Vol. 15, No. 3, March 2002, pp. 9�14.

5. Fagan, M. E., "Design and Code Inspections to Reduce Errors in Program Development,"IBM Systems Journal, Vol. 15, No. 3, 1976, pp. 182�211.

6. Freedman, D. P., and G. M. Weinberg, Handbook of Walkthroughs, Inspections, and Technical Reviews, Boston, Mass.: Little, Brown and Company, 1982.

7. Freedman, D. P., and G. M. Weinberg, "Reviews, Walkthroughs, and Inspections,"IEEE Transactions on Software Engineering, Vol. SE-10, No. 1, January 1984, pp. 68�72.

8. Gilb T., and D. Graham, Software Inspection, Reading, Mass.: Addison-Wesley, 1993.

9. Gilb, T., "Optimizing Software Engineering Specification Quality Control Processes (Inspections)"

SPIN Team Leader Course, Washington, D.C. SPIN Chapter, June 15�18, 1999.

10. Jones, C., Programming Productivity, New York: McGraw-Hill, 1986.

11. Jones, C., Software Assessments, Benchmarks, and Best Practices, Boston: Addison-Wesley, 2000.

12. Knight, K. C., and E. A. Myers, "Phased Inspections and Their Implementation," ACM SIGSOFT Software Engineering Notes, Vol. 16, No. 3, July 1991, pp. 29�35.

13. Kolkhorst, B. G., and A. J. Macina, "Developing Error-Free Software,"IEEE AES Magazine, November 1988, pp. 25�31.

14. Parnas, D. W., and D. M. Weiss, "Active Design Reviews: Principles and Practices,"Proceedings of Eighth International Conference on Software Engineering, London, England, IEEE Computer Society, August 1985.

15. Remus, H., "Integrated Software Validation in the View of Inspections/Review," Proceedings of the Symposium on Software Validation, Darmstadt, Germany, Amsterdam: North Holland, 1983, pp. 57�64.

16. Remus, H., and S. Zilles, "Prediction and Management of Program Quality," Proceedings of the Fourth International Conference on Software Engineering, Munich, IEEE Computer Society, 1979, pp.

341�350.

17. Ryan, J., "This Company Hates Surprises: The IBM Federal Systems Division Leaves No Stone Unturned in Its Quest to Produce Error-Free Software for NASA's Space Shuttle,"Quality Progress, September 1987, pp. 12�16.

18. Software Productivity Research, Quality and Productivity of the SEI CMM, Burlington, Mass.:

Software Productivity Research, 1994.

Addison Wesley: Metrics and Models in Software Quality Engineering, Second Edition References

I l@ve RuBoard

Addison Wesley: Metrics and Models in Software Quality Engineering, Second Edition References

I l@ve RuBoard