Metrics & Models in Software Quality Engineering.pdf

(1)

Metrics and Models in Software Quality Engineering, Second Edition, is the definitive book on this essential topic of software development. Comprehensive in scope with extensive industry examples, it shows how to measure software quality and use measurements to improve the software development process. Four major categories of quality metrics and models are addressed: quality management, software reliability and projection, complexity, and customer view. In addition, the book discusses the fundamentals of measurement theory, specific quality metrics and tools, and methods for applying metrics to the software development process.

New chapters bring coverage of critical topics, including: In-process metrics for software testing

Metrics for object-oriented software development Availability metrics

Methods for conducting in-process quality assessments and software project assessments Dos and Don'ts of Software Process Improvement, by Patrick O'Toole

Using Function Point Metrics to Measure Software Process Improvement, by Capers Jones In addition to the excellent balance of theory, techniques, and examples, this book is highly instructive and practical, covering one of the most important topics in software development--quality engineering. I l@ve RuBoard

(6)

I l@ve RuBoard

• Table of Contents

Metrics and Models in Software Quality

Engineering, Second Edition

By Stephen H. Kan

Publisher: Addison Wesley Pub Date: September 20, 2002 ISBN : 0-201-72915-6 Pages : 560

Copyright

Foreword to the Second Edition Foreword to the First Edition Preface

Themes of This Book Organization of This Book

Suggested Ways to Read This Book Acknowledgments

Chapter 1. What Is Software Quality? Section 1.1. Quality: Popular Views Section 1.2. Quality: Professional Views Section 1.3. Software Quality

Section 1.4. Total Quality Management Section 1.5. Summary

References

Chapter 2. Software Development Process Models Section 2.1. The Waterfall Development Model Section 2.2. The Prototyping Approach

Section 2.3. The Spiral Model

Section 2.4. The Iterative Development Process Model Section 2.5. The Object-Oriented Development Process Section 2.6. The Cleanroom Methodology

Section 2.7. The Defect Prevention Process

Section 2.8. Process Maturity Framework and Quality Standards Section 2.9. Summary

References

Chapter 3. Fundamentals of Measurement Theory

Section 3.1. Definition, Operational Definition, and Measurement Section 3.2. Level of Measurement

Section 3.3. Some Basic Measures

(7)

Section 3.4. Reliability and Validity Section 3.5. Measurement Errors Section 3.6. Be Careful with Correlation Section 3.7. Criteria for Causality Section 3.8. Summary

References

Chapter 4. Software Quality Metrics Overview Section 4.1. Product Quality Metrics

Section 4.2. In-Process Quality Metrics

Section 4.3. Metrics for Software Maintenance Section 4.4. Examples of Metrics Programs Section 4.5. Collecting Software Engineering Data Section 4.6. Summary

References

Chapter 5. Applying the Seven Basic Quality Tools in Software Development

Section 5.1. Ishikawa's Seven Basic Tools Section 5.2. Checklist

Section 5.3. Pareto Diagram Section 5.4. Histogram Section 5.5. Run Charts Section 5.6. Scatter Diagram Section 5.7. Control Chart

Section 5.8. Cause-and-Effect Diagram Section 5.9. Relations Diagram

Section 5.10. Summary References

Chapter 6. Defect Removal Effectiveness Section 6.1. Literature Review

Section 6.2. A Closer Look at Defect Removal Effectiveness Section 6.3. Defect Removal Effectiveness and Quality Planning Section 6.4. Cost Effectiveness of Phase Defect Removal Section 6.5. Defect Removal Effectiveness and Process Maturity

Level

Chapter 7. The Rayleigh Model Section 7.1. Reliability Models Section 7.2. The Rayleigh Model Section 7.3. Basic Assumptions Section 7.4. Implementation

Section 7.5. Reliability and Predictive Validity Section 7.6. Summary

References

Chapter 8. Exponential Distribution and Reliability Growth Models Section 8.1. The Exponential Model

Section 8.2. Reliability Growth Models Section 8.3. Model Assumptions

Section 8.4. Criteria for Model Evaluation Section 8.5. Modeling Process

(8)

Section 8.6. Test Compression Factor

Section 8.7. Estimating the Distribution of Total Defects over Time Section 8.8. Summary

References

Chapter 9. Quality Management Models Section 9.1. The Rayleigh Model Framework Section 9.2. Code Integration Pattern

Section 9.3. The PTR Submodel

Section 9.4. The PTR Arrival and Backlog Projection Model Section 9.5. Reliability Growth Models

Section 9.6. Criteria for Model Evaluation Section 9.7. In-Process Metrics and Reports Section 9.8. Orthogonal Defect Classification Section 9.9. Summary

References

Chapter 10. In-Process Metrics for Software Testing Section 10.1. In-Process Metrics for Software Testing Section 10.2. In-Process Metrics and Quality Management

Section 10.3. Possible Metrics for Acceptance Testing to Evaluate Vendor-Developed Software

Section 10.4. How Do You Know Your Product Is Good Enough to Ship?

Chapter 11. Complexity Metrics and Models Section 11.1. Lines of Code

Section 11.2. Halstead's Software Science Section 11.3. Cyclomatic Complexity Section 11.4. Syntactic Constructs Section 11.5. Structure Metrics

Section 11.6. An Example of Module Design Metrics in Practice Section 11.7. Summary

References

Chapter 12. Metrics and Lessons Learned for Object-Oriented Projects

Section 12.1. Object-Oriented Concepts and Constructs Section 12.2. Design and Complexity Metrics

Section 12.3. Productivity Metrics

Section 12.4. Quality and Quality Management Metrics Section 12.5. Lessons Learned from OO Projects Summary

References

Chapter 13. Availability Metrics

13.1 Definition and Measurements of System Availability Section 13.2. Reliability, Availability, and Defect Rate Section 13.3. Collecting Customer Outage Data for Quality

Improvement

Section 13.4. In-process Metrics for Outage and Availability Summary

References

(9)

Chapter 14. Measuring and Analyzing Customer Satisfaction Section 14.1. Customer Satisfaction Surveys

Section 14.2. Analyzing Satisfaction Data Section 14.3. Satisfaction with Company Section 14.4. How Good Is Good Enough Section 14.5. Summary

References

Chapter 15. Conducting In-Process Quality Assessments Section 15.1. The Preparation Phase

Section 15.2. The Evaluation Phase Section 15.3. The Summarization Phase

Section 15.4. Recommendations and Risk Mitigation Section 15.5. Summary

References

Chapter 16. Conducting Software Project Assessments Section 16.1. Audit and Assessment

Section 16.2. Software Process Maturity Assessment and Software Project Assessment

Section 16.3. Software Process Assessment Cycle

Section 16.4. A Proposed Software Project Assessment Method Section 16.5. Summary

References

Chapter 17. Dos and Don'ts of Software Process Improvement Section 17.1. Measuring Process Maturity

Section 17.2. Measuring Process Capability

Section 17.3. Staged versus Continuous�Debating Religion Section 17.4. Measuring Levels Is Not Enough

Section 17.5. Establishing the Alignment Principle Section 17.6. Take Time Getting Faster

Section 17.7. Keep It Simple � or Face Decomplexification Section 17.8. Measuring the Value of Process Improvement Section 17.9. Measuring Process Adoption

Section 17.10. Measuring Process Compliance

Section 17.11. Celebrate the Journey, Not Just the Destination Section 17.12. Summary

References

Chapter 18. Using Function Point Metrics to Measure Software Process Improvements

Section 18.1. Software Process Improvement Sequences Section 18.2. Process Improvement Economics

Section 18.3. Measuring Process Improvements at Activity Levels Section 18.4. Summary

References

Chapter 19. Concluding Remarks Section 19.1. Data Quality Control

Section 19.2. Getting Started with a Software Metrics Program Section 19.3. Software Quality Engineering Modeling

Section 19.4. Statistical Process Control in Software Development Section 19.5. Measurement and the Future

(10)

References

A Project Assessment Questionnaire I l@ve RuBoard

(11)

I l@ve RuBoard Copyright

Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and Addison-Wesley was aware of a

trademark claim, the designations have been printed with initial capital letters or in all capitals. The author and publisher have taken care in the preparation of this book, but make no expressed or implied warranty of any kind and assume no responsibility for errors or omissions. No liability is assumed for incidental or consequential damages in connection with or arising out of the use of the information or programs contained herein.

AS/400 and eServer iSeries are registered trademarks of International Business Machines Corporation. Lotus Notes and Lotus 1-2-3 are trademarks or registered trademarks of Lotus Development Corporation. The publisher offers discounts on this book when ordered in quantity for bulk purchases and special sales. For more information, please contact:

U.S. Corporate and Government Sales (800) 382-3419

[email protected]

For sales outside of the U.S., please contact: International Sales

(317) 581-3793

[email protected]

Visit Addison-Wesley on the Web: www.awprofessional.com Library of Congress Cataloging-in-Publication Data

Kan, Stephen H.

Metrics and models in software quality engineering / Stephen H. Kan�2nd ed p. cm.

1. Computer software�Quality control I. Title QA76.76Q35 K35 2003

005.1'068'5--dc21 2002027737

transmitted, in any form, or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the prior consent of the publisher. Printed in the United States of America. Published

simultaneously in Canada.

For information on obtaining permission for use of material from this work, please submit a written request to:

Pearson Education, Inc.

(12)

Rights and Contracts Department 75 Arlington Street, Suite 300 Boston, MA 02116

Fax: (617) 848-7047

Text printed on recycled paper

1 2 3 4 5 6 7 8 9 10�MA�0605040302 First printing, September 2002

Dedication

To my mother, Mui Leong

I l@ve RuBoard

(13)

I l@ve RuBoard

Foreword to the Second Edition

For more than 50 years software has been a troublesome discipline. Software's problems are numerous and include cancelations, litigation, cost overruns, schedule overruns, high maintenance costs, and low levels of user satisfaction. The problems with software occur more often than not. My company's research indicates that more than half of large software projects will encounter some kind of delay, overrun, or failure to perform when deployed.

But software does not have to be as troublesome as it has been. Some complex software projects do achieve their delivery schedules and cost targets, and behave properly when used. Throughout my career in software I've been interested in what distinguishes successful software projects from failures and disasters.

It happens that the main factors leading to software success are easily identified when side-by-side comparisons of similar projects are performed, where one set was successful and the other set was troublesome. The successful software projects achieve excellence in software quality control, and they are able to do this because of excellence in software quality measurements.

Although it might be thought that excellent software quality control is expensive, it turns out to yield a very positive return on investment. When canceled software projects and disasters are studied by means of "autopsies," they all have similar patterns: Early phases of troubled projects are handled carelessly without adequate requirements analysis or design reviews. After rushing through the early phases and seeming to be ahead of schedule, problems begin to mount during coding and testing. When testing begins in earnest, serious problems are detected so that schedules and cost targets cannot be achieved. Indeed, some software projects have so many serious problems�termed bugs or defects�that they are canceled without completion.

By contrast, successful projects are more thorough at the start. The requirements are carefully analyzed and the designs are formally inspected. This takes some time and adds upfront costs, but once coding and testing begin, the value of careful quality control allows the projects to move rapidly to a successful conclusion.

Stephen Kan and I both learned the importance of software quality control and good software quality metrics at IBM. Even though Stephen and I worked in different IBM labs during different decades, we have some common viewpoints. We have both been convinced by empirical evidence that without excellent software quality control, large system development is hazardous and likely to fail. We also both know that effective software quality control is built on a foundation of careful measurements using accurate metrics.

Stephen and I both had access to IBM's internal quality data�one of the best and largest collections of software data in the world. But one company's internal information is not sufficient to be convincing unless other organizations replicate the findings. Stephen's excellent book goes beyond IBM; it offers valuable information from many companies and government agencies. He cites studies by Hewlett-Packard, Motorola, NASA, and other organizations that have solved the problems of how to build software successfully.

I first encountered Stephen Kan's Metrics and Models in Software Quality Engineering book in 1995 when the first edition was published. I found that the book contained solid research, and that it covered a broad range of topics and very clearly showed the relationships among them.

This new edition keeps all the virtues of the earlier work and adds substantial new information, such as a chapter on measuring quality within the object-oriented paradigm. Stephen Kan's new edition contains an excellent combination of solid scholarship and breadth of coverage. This is the single best book on software quality engineering and metrics that I've encountered.

(14)

Capers Jones

Chief Scientist Emeritus

Software Productivity Research, Inc. (an Artemis company)

I l@ve RuBoard

(15)

I l@ve RuBoard

Foreword to the First Edition

Quality management and engineering have enjoyed a diversity of applications over the past few years. For example:

A system of teaching hospitals conservatively estimates $17.8 million saved on an investment of $2.5 million in quality management over a five-year time period.

The U.S. Air Force Military Airlift Command improved capacity so much through quality improvement problem solving during the Gulf War that they avoided having to deploy civilian aircraft (thus

avoiding the suspension of next-day mail delivery, among other conveniences).

The U.S. Bureau of Labor Statistics reduced the time needed to produce the monthly Consumer Price Index (CPI), compiled by 650 people in five departments, by 33 percent with no loss in accuracy.

The University of Pennsylvania saved more than $60,000 a year from one project focused on reducing mailing costs.

The examples go on and on, from industries such as telecommunications, health care, law, hospitals, government, pharmaceuticals, railways, and schools. The variety of terrains where the seeds of TQM successfully take hold is almost baffling.

As the rest of the world moves headlong into quality improvement at revolutionary rates, the software developers and engineers�consumed in debates over metrics and process models, over methodologies and CASE tools�often lag far behind. Some of the reasons include unique challenges in defining user requirements, the "guru" mentality prevalent in many software organizations, and the relative immaturity of the application of software engineering. Whereas the first two are fascinating topics in their own right, it is the third challenge at which Stephen Kan's book is squarely aimed.

Imagine designing an airplane using a small fraction of the aeronautical engineering knowledge available. Imagine designing an automobile while ignoring mechanical engineering. Imagine running a refinery with no knowledge of chemical process engineering. It is not surprising that the first recommendations from consultants offering credible software quality solutions would be to apply proven software engineering methods first. Just as these methods only slowly find acceptance in software development communities, so also have the methods of quality engineering.

One reason for this slow adoption lies with the relative lack of literature that gives clear descriptions of the fundamentals in these fields while illustrating actual use within leading-edge software development organizations. Stephen Kan's book, Metrics and Models in Software Quality Engineering, represents a laudable move to fill this need.

Dr. Kan provides a uniquely comprehensive reference to the field of software quality engineering. He has managed a delightful balance between the technical details needed and practical applications of the models and techniques. This book is peppered with industry examples, not only from Kan's own employer, the Malcolm Baldrige National Quality Award-winning IBM Rochester, but from NEC's Switching Systems Division, Hewlett-Packard, Motorola, NASA Software Engineering Laboratory, and IBM Federal Systems Division. Concepts and theory are illustrated by software industry examples, which make the reading that much richer.

Dr. Joseph Juran, one of the key founders of modern quality management and engineering, describes "life behind the quality dikes." As society becomes more reliant on technology, failures of that technology have increasing adverse impacts. Quality helps to insulate society from these dangers. This role of quality in software development certainly rivals that of business competitiveness, and gives another compelling reason to read, understand, and apply the ideas within this book.

(16)

Brian Thomas Eck, Ph.D. Vice President

Juran Institute, Inc. Wilton, Connecticut

I l@ve RuBoard

(17)

I l@ve RuBoard Preface

Looking at software engineering from a historical perspective, the 1960s and earlier could be viewed as the functional era, the 1970s the schedule era, the 1980s the cost era, and the 1990s and beyond the quality and efficiency era. In the 1960s we learned how to exploit information technology to meet

institutional needs and began to link software with the daily operations of institutions. In the 1970s, as the industry was characterized by massive schedule delays and cost overruns, the focus was on planning and control of software projects. Phase-based life-cycle models were introduced, and analysis, like the mythical man-month, emerged. In the 1980s hardware costs continued to decline, and information technology permeated every facet of our institutions and became available to individuals. As competition in the industry became keen and low-cost applications became widely implemented, the importance of productivity in software development increased significantly. Various software engineering cost models were developed and used. In the late 1980s, the importance of quality was also recognized.

The 1990s and beyond is certainly the quality era. With state-of-the-art technology now able to provide abundant functionality, customers demand high quality. Demand for quality is further intensified by the ever-increasing dependence of society on software. Billing errors, large-scale disrupted telephone services, and even missile failures during recent wars can all be traced to the issue of software quality. In this era, quality has been brought to the center of the software development process. From the standpoint of software vendors, quality is no longer an advantage factor in the marketplace; it has become a

necessary condition if a company is to compete successfully.

Starting in the mid 1990s two major factors emerged that have proved to have an unprecedented impact on not only software engineering but also on global business environments: business reengineering for efficiency and the Internet. Software development has to be more efficient and the quality level of the delivered products has to be high to meet requirements and to be successful. This is especially the case for mission-critical applications. The adverse impact of poor quality is much more significant and at a much wider scale; the quality "dikes" that software is supposed to provide are never more important. These factors will continue to affect software engineering for many years to come during this new millennium.

Measurement plays a critical role in effective and efficient software development, as well as provides the scientific basis for software engineering that makes it a true engineering discipline. This book describes the software quality engineering metrics and models: quality planning, process improvement and quality control, inprocess quality management, product engineering (design and code complexity), reliability estimation and projection, and analysis of customer satisfaction data. Many measurement books take an encyclopedic approach, in which every possible software measurement is included, but this book confines its scope to the metrics and models of software quality. Areas such as cost estimation, productivity,

staffing, and performance measurement, for which numerous publications exist, are not covered. In this edition, seven new chapters have been added, covering in-process metrics for software testing, object-oriented metrics, availability metrics, in-process quality assessment, software project assessment, process improvement dos and don'ts, and measuring software process improvement. The chapter that described the AS/400 software quality management system has been eliminated. Updates and revisions have been made throughout the original chapters, and new sections, figures, and tables have been added.

Two of the new chapters are special contributions from two experts. This is a key feature of the new edition. The chapter on the dos and don'ts of software process improvement is contributed by Patrick O'Toole. A highly regarded process improvement expert and with over 20 years of experience, Patrick brings to this book a perspective on process improvement that I share as a practitioner. That perspective is based on practical experience, is project-centric, and is aligned with the strategic business imperative of the organization. Patrick also brings humor to this otherwise serious subject, making the reading of the chapter so enjoyable. The chapter on measuring software process improvement is a special contribution Addison Wesley: Metrics and Models in Software Quality Engineering, Second Edition Preface

(18)

by Capers Jones. A pioneer in software metrics, productivity research, software quality control, and software assessments, Capers's work is well known nationally and internationally. His data-based and fact-based approach in software assessments and benchmarking studies is unparalleled. Based on experience and data from more than 10,000 projects, he brings to the readers a practical approach to software process improvement and the major quantitative findings related to software process

improvement. The value of function point metrics is demonstrated via the analyses and findings. The chapter is a must read for software process professionals who are interested in measuring software process improvement.

Another new feature in this edition is a set of recommendations for small teams and organizations that are starting to implement a metrics program, with minimum resources. These recommendations are shown in the form of box inserts in nine of the chapters. A number of examples in the book are based on small team projects, and many methods and techniques are appropriate for large projects as well as small ones. This set of recommendations is from the perspective of small organizations or teams using a small number of metrics, with the intent to effect improvement in their software development effort.

This book is intended for use by software quality professionals; software project managers; software product managers; software development managers; software engineers; software product assurance personnel; and students in software engineering, management information systems, systems engineering, and quality engineering and management. For teachers, it is intended to provide a basis for a course at the upper-division undergraduate or graduate level. A number of software engineering, computer science, and quality engineering programs in the United States and overseas have used the first edition of this book as a text.

I l@ve RuBoard

(19)

I l@ve RuBoard

Themes of This Book

This book has several themes. First, balancing theory, techniques, and real-life examples, it provides practical guidelines in the practice of quality engineering in software development. Although equations and formulas are involved, the focus is on the understanding and applications of the metrics and models rather than mathematical derivations. Throughout the book, numerous real-life examples are used from the software development laboratory at IBM Rochester, Minnesota, home of the AS/400 and the IBM eServer iSeries computer systems, and from other companies in the software industry. IBM Rochester won the Malcolm Baldrige National Quality Award in 1990. A number of metrics described in this book were being used at that time, and many have been developed and refined since then. All metrics are substantiated by ample implementation experience. IBM Rochester develops and delivers numerous projects of different sizes and types every year, including very large and complex as well as small ones; and they range from firmware, to operating systems, to middleware, to applications.

Second, I attempt to provide a good coverage of the various types of metrics and models in the emerging field of software quality engineering. In addition to general discussions about metrics and techniques, this book categorizes and covers four types of metrics and models: (1) quality management models; (2) software reliability and projection models; (3) complexity metrics and models; and (4) customer-view metrics, measurements, and models. These metrics and models cover the entire software development process from high-level design to testing and maintenance, as well as all phases of reliability.

Furthermore, although this book is not on total quality management (TQM), it is a major consideration in the coverage of metrics. The philosophy of TQM is the linking of product quality and customer

satisfaction for the purpose of achieving long-term success. TQM is the reason for including two chapters on customer-view metrics and measurements�availability metrics and customer satisfaction�in addition to the many chapters on product and process metrics. In other discussions in the book, the customer's perspective is included where appropriate.

Third, by linking metrics and models to quality improvement strategies and improvement actions, we attempt to focus on using, not just describing, metrics. A framework for interpreting in-process metrics and assessing in-process quality status�the effort/outcome model�is presented. The direct link between a recommended quality strategy during development and the defect-removal model is shown. Examples of actions tied to specific metrics and analysis are given. Furthermore, to illustrate the metrics, many figures and graphs are used. This is a reflection of the fact that in real-life project and quality management, a clear visual presentation often improves understanding and increases the effectiveness of the metrics. Fourth, following up on quality and process improvement at a more general level than specific metric discussions, the book continues with chapters that discuss the inprocess quality assessment process, a method for conducting software project assessments, practical advice on process improvement dos and don'ts, and quantitative analysis of software process improvement. The common thread underlying these chapters, as with other chapters on metrics and models, is practical experience with industry projects. I l@ve RuBoard

(20)

I l@ve RuBoard

Organization of This Book

The following list details the focus of each chapter.

Chapter 1, What Is Software Quality?, discusses the definition of quality and software quality. The customer's role in the definition is highlighted. Quality attributes and their relationships are

discussed. The second part of the chapter covers the definition and framework of TQM and the customer's view of quality, a key focus in this book.

Chapter 2, Software Development Process Models, reviews various development process models that are used in the software industry. It briefly describes two methods of software process maturity assessment�the SEI process capability maturity model (CMM) (by the Software Engineering Institute) and the SPR assessment method (by the Software Productivity Research, Inc.). It summarizes two bodies of quality management standards�the Malcolm Baldrige National Quality Award assessment discipline and ISO 9000.

Chapter 3, Fundamentals of Measurement Theory, examines measurement theory fundamentals, which are very important for the practice of software measurement. The concept of operational definition and its importance in measurement are illustrated with an example. The level of measurement, some basic measures, and the concept of six sigma are discussed. The two key criteria of measurement quality, reliability and validity, and the related issue of measurement errors are examined and their importance is articulated. This chapter also provides a discussion on

correlation and addresses the criteria necessary to establish causality based on observational data. Chapter 4, Software Quality Metrics Overview, presents examples of quality metrics for the three categories of metrics associated with the software life cycle: end-product, in-process, and

maintenance. It describes the metrics programs of several large software companies and discusses collection of software engineering data.

Chapter 5, Applying the Seven Basic Quality Tools in Software Development, describes the

application of the basic statistical tools for quality control, known as Ishikawa's seven basic tools, in software development. The potentials and challenges of applying the control chart in software environments are discussed. In addition, a qualitative tool for brainstorming and for displaying complex cause-and-effect relationships�the relations diagram�is discussed.

Chapter 6, Defect Removal Effectiveness, is the first of five chapters about the models and metrics that describe the quality dynamics of software development. Through two types of models, quality management models and software reliability and projection models, the quality of software

development can be planned, engineered, managed, and projected. This chapter examines the central concept of defect removal effectiveness, its measurements, and its role in quality planning. Chapter 7, The Rayleigh Model, describes the model and its implementation as a reliability and projection model. The Rayleigh Model's use as a quality management model is discussed in Chapter 9.

Chapter 8, Exponential Distribution and Reliability Growth Models, discusses the exponential

distribution and the major software reliability growth models. These models, like the Rayleigh Model, are used for quality projection before the software is shipped to customers, just before development is complete. The models are also used for maintenance planning, to model the failure pattern or the defect arrival patterns in the field.

Chapter 9, Quality Management Models, describes several quality management models that cover the entire development cycle. In-process metrics and reports that support the models are shown and discussed. A framework for interpreting in-process metrics and assessing in-process quality

status�the effort/outcome model�is presented.

(21)

Chapter 10, In-Process Metrics for Software Testing, is a continuation of Chapter 9; it focuses on the metrics for software testing. The effort/outcome model, as it applies to metrics during the testing phase, is elaborated. Candidate metrics for acceptance testing to evaluate vendor-developed software, and the central question of how to know your product is good enough to ship, are also discussed.

Chapter 11, Complexity Metrics and Models, discusses the third type of metrics and models in software engineering. While quality management models and reliability and projection models are for project management and quality management, the objective of the complexity metrics and models is for software engineers to be able to improve their design and implementation of software

development.

Chapter 12, Metrics and Lessons Learned for Object-Oriented Projects, covers design and

complexity metrics, productivity metrics, quality and quality management metrics for object-oriented development, and lessons learned from the deployment and implementation of OO projects. The first section can be viewed as a continuation of the discussion on complexity metrics and models; the other sections fall within the framework of quality and project management.

Chapter 13, Availability Metrics, discusses system availability and outage metrics, and explores the relationships among availability, reliability, and the traditional defect-rate measurement. Availability metrics and customer satisfaction measurements are the fourth type of metrics and

models�customer-oriented metrics.

Chapter 14, Measuring and Analyzing Customer Satisfaction, discusses data collection and measurements of customer satisfaction, and techniques and models for the analysis of customer satisfaction data. From Chapter 3 to this chapter, the entire spectrum of metrics and models is covered.

Chapter 15, Conducting In-Process Quality Assessments, describes in-process quality assessments as an integrated element of good project quality management. Quality assessments are based on both quantitative indicators, such as those discussed in previous chapters, and qualitative

information.

Chapter 16, Conducting Software Project Assessments, takes the discussion to yet another level; this chapter proposes a software project assessment method. The focus is at the project level and the discussion is from a practitioner's perspective.

Chapter 17, Dos and Don'ts of Software Process Improvement by Patrick O'Toole, offers practical advice for software process improvement professionals. It provides a link to the process maturity discussions in Chapter 2.

Chapter 18, Using Function Point Metrics to Measure Software Process Improvement by Capers Jones, discusses the six stages of software process improvement. Based on a large body of empirical data, it examines the costs and effects of process improvement. It shows the results of quantitative analy-ses with regard to costs, time, schedule, productivity, and quality. It articulates the value of Function Point metrics. It provides a link to the process maturity discussions in Chapter 2. Chapter 19, Concluding Remarks, provides several observations with regard to software

measurement in general and software quality metrics and models in particular, and it offers a perspective on the future of software engineering measurement.

In the Appendix, a real-life example of a project assessment questionnaire is shown. Per the

methods and techniques discussed in Chapter 16, readers can customize the questionnaire for their project assessment efforts.

I l@ve RuBoard

(22)

I l@ve RuBoard

Suggested Ways to Read This Book

The chapters of this book are organized for reading from beginning to end. Later chapters refer to concepts and discussions in earlier chapters. At the same time, each chapter addresses a separate topic and chapters in some groups are more closely coupled than others. Some readers may choose to read specific topics or decide on different starting points. For example, those who are not interested in quality definitions, process models, and measurement fundamentals discussions can start with Chapter 4, Software Quality Metrics Overview. Those who intend to immediately get to the central topics of defect removals, metrics and models for quality planning, and management and projection can start with Chapter 6, Defect Removal Effectiveness. In general, I recommend that the chapters be read in groups, as follows.

Chapters 1 through 3 Chapter 4 Chapter 5 Chapters 6 through 10 Chapters 11 and 12 Chapters 13 and 14 Chapters 15 through 18 Chapter 19 I l@ve RuBoard

(23)

I l@ve RuBoard Acknowledgments

I would like to thank Terry Larson, Steve Braddish, and Mike Tomashek for their support of this project. I wish to thank the entire iSeries software development team at IBM Rochester, especially the release and project management team and the test teams, who made the many metrics and models described in this book a state of practice instead of just a theoretical discussion. A special thanks is due Diane Manlove and Jerry Parrish for their leadership in the implementation of many of the in-process metrics, and Bob Gintowt for his leadership work, knowledge, and insights on system availability and outage metrics. Appreciation is due all of my former and present colleagues in software and system quality at IBM Rochester, other IBM divisions, IBM Center for Software Engineering, and IBM Corporate Quality, for the numerous discussions, debates, and insights on the subjects of measurement, quality and process improvement. There are too many individuals and teams to name them all. Among them are: Lionel Craddock, John E. Peterson, Dave Lind, Dr. Sam Huang, Don Mitchell, Judy Wasser, Marijeanne Swift, Duane Miller, Dr. Dave Jacobson, Dick Bhend, Brad Talmo, Brock Peterson, Vern Peterson, Charlie Gilmore, Jim Vlazny, Mary Ann Donovan, Jerry Miller, Mike Tappon, Phil Einspahr, Tami Mena, Peter Bradford, Max Maurer, Roger McKnight, Jack Hickley, Marilyn Moncol, Darrell Moore, Dusan Gasich, Eileen Gottschall, Carl Chamberlin, Paul Hutchings, Gary Davidson, George Stark, Kathleen Coyle, Ram Chillarege, Peter Santhanam, Kathryn Bassin, Beng Chiu, Wanda Sarti, Brent Hodges, Bill Woodworth, and Mike Jesrani. I am grateful to Dave Amundson, Ben Borgen, Dick Sulack, Rod Morlock, Brian Truskowski, Jeff VerHeul, Judy Tenney, Mike Tomashek, and Paul Loftus, from whom I learned a great deal not only about quality and quality management, but also about prudent decision making with regard to business objectives and quality.

A special gratitude is due Capers Jones for his review of this new edition, many excellent suggestions, and his chapter on measuring process improvement. In my early career at IBM I benefitted a great deal from reading Capers' pioneer work on programming productivity, software quality control, and software assessments. Then in 1995, I had the chance to meet him in person in Salt Lake City, where he gave a keynote speech on software measurements at a software technology conference. Now many years later, I continue to learn and benefit from reading his work and from his reviews and suggestions.

My sincere thanks are due Patrick O'Toole for his special contribution of the chapter on the dos and don'ts of software process improvement, amid his very busy schedule. A special thanks is due Steve Hoisington for his review and helpful suggestions of the new materials, and as a former colleague, his passion in quality, his professionalism in quality management, and his long-time influence on the customer view of quality.

Much appreciation is due Dick Hedger, for his review of and help with both the first and second editions, his passion for and expertise in software process improvement, and his continual support over many years. I am thankful to the reviewers for the first edition, Dr. Brian Eck, Dr. Alan Yaung, Professor Wei-Tsek Tsai, and others. They contributed many constructive suggestions.

I thank Debbie Lafferty, Marilyn Rash, Diane Freed, and their Addison-Wesley colleagues for their assistance and guidance during the preparation of this new edition.

Appreciation is due the students at the MSSE program (Masters of Science in Software Engineering) at the University of Minnesota for the past several years. They are software engineering professionals in the industry, with different backgrounds and representing different types of organizations, software, projects, industry segments, and processes. The numerous discussions and exchanges with them influenced my thinking and increased my appreciation of the diversity in software engineering. I wish to thank Professor Mats Heimdahl, who founded and directs the MSSE program, for his vision and successful

implementation in bringing industry practices and experiences into the curriculum of a software engineering program.

Finally, a special word of gratitude is due my wife, Teresa, and my sons, Wilbur and Este, for their

(24)

continual encouragement, patience, and support throughout the entire process. Our discussions of the boys' lawn mowing experiences deepened my appreciation of the approaches to quality and process improvement. I could not have completed this new edition without their support.

Stephen H. Kan, Ph.D. Rochester, Minnesota

I l@ve RuBoard

(25)

I l@ve RuBoard

Chapter 1. What Is Software Quality?

Quality must be defined and measured if improvement is to be achieved. Yet, a major problem in quality engineering and management is that the term quality is ambiguous, such that it is commonly

misunderstood. The confusion may be attributed to several reasons. First, quality is not a single idea, but rather a multidimensional concept. The dimensions of quality include the entity of interest, the viewpoint on that entity, and the quality attributes of that entity. Second, for any concept there are levels of abstraction; when people talk about quality, one party could be referring to it in its broadest sense, whereas another might be referring to its specific meaning. Third, the term quality is a part of our daily language and the popular and professional uses of it may be very different.

In this chapter we discuss the popular views of quality, its formal definitions by quality experts and their implications, the meaning and specific uses of quality in software, and the approach and key elements of total quality management.

I l@ve RuBoard

(26)

I l@ve RuBoard

1.1 Quality: Popular Views

A popular view of quality is that it is an intangible trait�it can be discussed, felt, and judged, but cannot be weighed or measured. To many people, quality is similar to what a federal judge once commented about obscenity: "I know it when I see it." Terms such as good quality, bad quality, and quality of life exemplify how people talk about something vague, which they don't intend to define. This view reflects the fact that people perceive and interpret quality in different ways. The implication is that quality cannot be controlled and managed, nor can it be quantified. This view is in vivid contrast to the professional view held in the discipline of quality engineering that quality can, and should, be operationally defined,

measured, monitored, managed, and improved.

Another popular view is that quality connotes luxury, class, and taste. Expensive, elaborate, and more complex products are regarded as offering a higher level of quality than their humbler counterparts. Therefore, a Cadillac is a quality car, but a Chevrolet is not, regardless of reliability and repair records; or, a surround-sound hi-fi system is a quality system, but a single-speaker radio is not. According to this view, quality is restricted to a limited class of expensive products with sophisticated functionality and items that have a touch of class. Simple, inexpensive products can hardly be classified as quality products.

I l@ve RuBoard

(27)

I l@ve RuBoard

1.2 Quality: Professional Views

The misconceptions and vagueness of the popular views do not help the quality improvement effort in the industries. To that end, quality must be described in a workable definition. Crosby (1979) defines quality as "conformance to requirements" and Juran and Gryna (1970) define it as "fitness for use." These two definitions are related and consistent, as we will see later. These definitions of quality have been adopted and used by many quality professionals.

"Conformance to requirements" implies that requirements must be clearly stated such that they cannot be misunderstood. Then, in the development and production process, measurements are taken regularly to determine conformance to those requirements. The nonconformances are regarded as defects�the absence of quality. For example, one requirement (specification) for a certain radio may be that it must be able to receive certain frequencies more than 30 miles away from the source of broadcast. If the radio fails to do so, then it does not meet the quality requirements and should be rejected. By the same token, if a Cadillac conforms to all the requirements of a Cadillac, then it is a quality car. If a Chevrolet conforms to all the requirements of a Chevrolet, then it is also a quality car. The two cars may be very different in style, performance, and economy. But if both measure up to the standards set for them, then both are quality cars.

The "fitness for use" definition takes customers' requirements and expectations into account, which involve whether the products or services fit their uses. Since different customers may use the products in different ways, it means that products must possess multiple elements of fitness for use. According to Juran, each of these elements is a quality characteristic and all of them can be classified into categories known as parameters for fitness for use. The two most important parameters are quality of design and quality of conformance.

Quality of design in popular terminology is known as grades or models, which are related to the spectrum of purchasing power. The differences between grades are the result of intended or designed differences. Using the example of cars again, all automobiles provide to the user the service of transportation.

However, models differ in size, comfort, performance, style, economy, and status. In contrast, quality of conformance is the extent to which the product conforms to the intent of the design. In other words, quality of design can be regarded as the determination of requirements and specifications and quality of conformance is conformance to requirements.

The two definitions of quality (conformance to requirements and fitness for use), therefore, are essentially similar. The difference is that the fitness for use concept implies a more significant role for customers' requirements and expectations.

1.2.1 The Role of the Customer

The role of the customer, as it relates to quality, can never be overstated. From a customer's standpoint, quality is the customer's perceived value of the product he or she purchased, based on a variety of variables such as price, performance, reliability, and satisfaction. In Guaspari's book I Know It When I See It (1985 p.77), he discusses quality in the customers' context as follows:

Your customers are in a perfect position to tell you about quality, because that's all they're really buying. They're not buying a product. They're buying your assurances that their expectations for that product will be met.

And you haven't really got anything else to sell them but those assurances. You haven't really got anything else to sell but quality.

From a concept's high-level definition to a product's operational definition, many steps are involved, each of which may be vulnerable to shortcomings. For example, to achieve the state of conformance to

requirements, the customers' requirements must be first gathered and analyzed, specifications to meet Addison Wesley: Metrics and Models in Software Quality Engineering, Second Edition1.2 Quality: Professional Views

(28)

those requirements must be produced, and the product must be developed and manufactured accordingly. In each phase of the process, errors can occur that will affect the quality of the finished product. The requirements may be erroneous (this is especially the case for software development), the development and manufacturing process may be subject to variables that induce defects, and so forth. From the customer's perspective, satisfaction after the purchase of the product is the ultimate validation that the product conforms to requirements and is fit to use. From the producer's perspective, once

requirements are specified, developing and producing the product in accordance with the specifications is the path to achieving quality. Usually, for product quality, the lack of defects and good reliability are the most basic measures.

Because of the two perspectives on quality (customer satisfaction as the ultimate validation of quality and producer's adherence to requirements to achieve quality), the de facto definition of quality consists of two levels. The first is the intrinsic product quality, often operationally limited to the product's defect rate and reliability. This narrow definition is referred to as the "small q" (q for quality). The broader definition of quality includes product quality, process quality, and customer satisfaction, and it is referred to as the "big Q." This two-level approach to the definition of quality is being used in many industries, including the automobile industry, the computer industry (both software and hardware), and the consumer electronics industry.

The two-level concept of quality is supposed to form a closed-loop cycle: customer's wants and needs requirements and specifications products designed, developed, and manufactured in accordance with the requirements, and with continuous focus on process improvement excellent product quality, plus good distribution and service processes total customer satisfaction. However, this was not always the case in many industries, especially before the late 1980s when the modern quality era began. Product requirements were often generated without customer input, and customer satisfaction was not always a factor in business decision making. Although the final products conformed to requirements, they may not have been what the customers wanted. Therefore, the customers' role should be explicitly stated in the definition of quality: conformance to customers' requirements.

I l@ve RuBoard

(29)

I l@ve RuBoard 1.3 Software Quality

In software, the narrowest sense of product quality is commonly recognized as lack of "bugs" in the product. It is also the most basic meaning of conformance to requirements, because if the software contains too many functional defects, the basic requirement of providing the desired function is not met. This definition is usually expressed in two ways: defect rate (e.g., number of defects per million lines of source code, per function point, or other unit) and reliability (e.g., number of failures per n hours of operation, mean time to failure, or the probability of failure-free operation in a specified time). Customer satisfaction is usually measured by percent satisfied or nonsatisfied (neutral and dissatisfied) from

customer satisfaction surveys. To reduce bias, techniques such as blind surveys (the interviewer does not know the customer and the customer does not know the company the interviewer represents) are usually used. In addition to overall customer satisfaction with the software product, satisfaction toward specific attributes is also gauged. For instance, IBM monitors satisfaction with its software products in levels of CUPRIMDSO (capability [functionality], usability, performance, reliability, installability, maintainability, documentation/information, service, and overall). Hewlett-Packard focuses on FURPS (functionality, usability, reliability, performance, and serviceability). Other compa-nies use similar dimensions of software customer satisfaction. Juran calls such attributes quality parameters, or parameters for fitness for use. To increase overall customer satisfaction as well as satisfaction with various quality attributes, the quality attributes must be taken into account in the planning and design of the software. However, these quality attributes are not always congruous with each other. For example, the higher the functional complexity of the software, the harder it becomes to achieve maintainability. Depending on the type of software and customers, different weighting factors are needed for different quality attributes. For large customers with sophisticated networks and real-time processing, performance and reliability may be the most important attributes. For customers with standalone systems and simple operations, on the other hand, ease of use, installability, and documentation may be more important. Figure 1.1 shows the possible relationships of some quality attributes. Some relationships are mutually supportive, some are negative, and yet others are not clear, depending on the types of customers and applications. For software with a diverse

customer set, therefore, setting goals for various quality attributes and to meet customers' requirements is not easy.

Figure 1.1. Interrelationships of Software Attributes�A CUPRIMDA Example

In view of these discussions, the updated definition of quality (i.e., conformance to customers'

requirements) is especially relevant to the software industry. It is not surprising that requirements errors constitute one of the major problem categories in software development. According to Jones (1992), 15% or more of all software defects are requirements errors. A development process that does not address requirements quality is bound to produce poor-quality software.

(30)

Yet another view of software quality is that of process quality versus end-product quality. From customer requirements to the delivery of software products, the development process is complex and often involves a series of stages, each with feedback paths. In each stage, an intermediate deliverable is produced for an intermediate user�the next stage. Each stage also receives an intermediate deliverable from the preceding stage. Each intermediate deliverable has certain quality attributes that affect the quality of the end product. For instance, Figure 1.2 shows a simplified representation of the most common software development process, the waterfall process.

Figure 1.2. Simplified Representation of the Waterfall Development Process

Intriguingly, if we extend the concept of customer in the definition of quality to include both external and internal customers, the definition also applies to process quality. If each stage of the development

process meets the requirements of its intermediate user (the next stage), the end product thus developed and produced will meet the specified requirements. This statement, of course, is an oversimplification of reality, because in each stage numerous factors exist that will affect that stage's ability to fulfill its

requirements. To improve quality during development, we need models of the development process, and within the process we need to select and deploy specific methods and approaches, and employ proper tools and technologies. We need measures of the characteristics and quality parameters of the

development process and its stages, as well as metrics and models to ensure that the development process is under control and moving toward the product's quality objectives. Quality metrics and models are the focus of this book.

I l@ve RuBoard

(31)

I l@ve RuBoard

1.4 Total Quality Management

The term Total quality management (TQM) was originally coined in 1985 by the Naval Air Systems Command to describe its Japanese-style management approach to quality improvement. The term has taken on a number of meanings, depending on who is interpreting it and how they are applying it. In general, however, it represents a style of management aimed at achieving long-term success by linking quality and customer satisfaction. Basic to the approach is the creation of a culture in which all members of the organization participate in the improvement of processes, products, and services. Various specific methods for implementing the TQM philosophy are found in the works of Crosby (1979), Deming (1986), Feigenbaum (1961, 1991), Ishikawa (1985), and Juran and Gryna (1970).

Since the 1980s, many U.S. companies have adopted the TQM approach to quality. The Malcolm Baldrige National Quality Award (MBNQA), established by the U.S. government in 1988, highlights the embracing of such a philosophy and management style. The adoption of ISO 9000 as the quality management standard by the European Community and the acceptance of such standards by the U.S. private sector in recent years further illustrates the importance of the quality philosophy in today's business environments. In the computer and electronic industry, examples of successful TQM

implementation include Hewlett-Packard's Total Quality Control (TQC), Motorola's Six Sigma Strategy, and IBM's Market Driven Quality. In fact, Motorola won the first MBNQA award (in 1988) and IBM's AS/400 Division in Rochester, Minnesota, won it in 1990.

Hewlett-Packard's TQC focuses on key areas such as management commitment, leadership, customer focus, total participation, and systematic analysis. Each area has strategies and plans to drive the improvement of quality, efficiency, and responsiveness, with the final objective being to achieve success through customer satisfaction (Shores, 1989). In software development, the Software Quality and Productivity Analysis (SQPA) program (Zimmer, 1989) is one of the approaches to improve quality. Motorola's Six Sigma strategy focuses on achieving stringent quality levels in order to obtain total

customer satisfaction. Cycle time reduction and participative management are among the key initiatives of the strategy (Smith, 1989). Six Sigma is not just a measure of the quality level; inherent in the concept are product design improvements and reductions in process variations (Harry and Lawson, 1992). Six Sigma is applied to product quality as well as everything that can be supported by data and measurement. "Customer is the final arbiter" is the key theme of IBM's Market Driven Quality strategy. The strategy comprises four initiatives: defect elimination, cycle time reduction, customer and business partner satisfaction, and adherence to the Baldrige assessment discipline.

Despite variations in its implementation, the key elements of a TQM system can be summarized as follows:

Customer focus: The objective is to achieve total customer satisfaction. Customer focus includes studying customers' wants and needs, gathering customers' requirements, and measuring and managing customers' satisfaction.

Process: The objective is to reduce process variations and to achieve continuous process improvement. This element includes both the business process and the product development process. Through process improvement, product quality will be enhanced.

Human side of quality: The objective is to create a companywide quality culture. Focus areas include leadership, management commitment, total participation, employee empowerment, and other social, psychological, and human factors.

Measurement and analysis: The objective is to drive continuous improvement in all quality parameters by the goal-oriented measurement system.

Furthermore, an organization that practices TQM must have executive leadership, must focus on

Metrics & Models in Software Quality Engineering.pdf

Table of Contents

1

Main Page

2

Table of content

7

Copyright

9

Foreword to the Second Edition

11

Foreword to the First Edition

13

Preface

15

16

18

19

21

Chapter 1. What Is Software Quality?

22

23

25

27

30

31

33

Chapter 2. Software Development Process Models

34

39

41

43

46

50

52

55

66

67

70

Chapter 3. Fundamentals of Measurement Theory

71

74

76

82

84

88

90

92

93

94

Chapter 4. Software Quality Metrics Overview

95

106

110

114

121

127

129

131

Chapter 5. Applying the Seven Basic Quality Tools in Software Development

132

134

136

138

139

141

143

150

151

153

155

156

Chapter 6. Defect Removal Effectiveness

157

161

168

172

175

178

179