The Team Software Process SM (TSP SM ) in Practice: A Summary of Recent Results

The Team Software

Process

(TSP

)

in Practice: A Summary

of Recent Results

Pittsburgh, PA 15213-3890

The Team Software Process

(TSP

) in Practice:

A Summary of Recent Results

ESC-TR-2003-014

Noopur Davis Julia Mullaney

September 2003

Software Engineering Process Management Program

This report was prepared for the SEI Joint Program Office

HQ ESC/DIB 5 Eglin Street

Hanscom AFB, MA 01731-2116

The ideas and findings in this report should not be construed as an official DoD position. It is published in the interest of scientific and technical information exchange.

FOR THE COMMANDER

Christos Scondras Chief of Programs, XPK

This work is sponsored by the U.S. Department of Defense. The Software Engineering Institute is a federally funded research and development center sponsored by the U.S. Department of Defense. Copyright 2003 by Carnegie Mellon University.

NO WARRANTY

THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING INSTITUTE MATERIAL IS FURNISHED ON AN "AS-IS" BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTER INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE OR MERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL. CARNEGIE MELLON UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITH RESPECT TO FREEDOM FROM PATENT, TRADEMARK, OR COPYRIGHT INFRINGEMENT.

Use of any trademarks in this report is not intended in any way to infringe on the rights of the trademark holder.

Internal use. Permission to reproduce this document and to prepare derivative works from this document for internal use is granted, provided the copyright and "No Warranty" statements are included with all reproductions and derivative works. External use. Requests for permission to reproduce this document or prepare derivative works of this document for external and commercial use should be addressed to the SEI Licensing Agent.

This work was created in the performance of Federal Government Contract Number F19628-00-C-0003 with Carnegie Mel-lon University for the operation of the Software Engineering Institute, a federally funded research and development center. The Government of the United States has a royalty-free government-purpose license to use, duplicate, or disclose the work, in whole or in part and in any manner, and to have or permit others to do so, for government purposes pursuant to the copy-right license under the clause at 252.227-7013.

For information about purchasing paper copies of SEI reports, please visit the publications portion of our Web site (http://www.sei.cmu.edu/publications/pubweb.html).

Table of Contents

2.2 What Makes PSP and TSP Work...4

2.3 The PSP...5

2.3.1 PSP Measurement Framework ...7

2.4 The TSP ...8

2.4.1 The TSP Launch ...9

2.4.2 TSP Measurement Framework ...12

2.4.3 The TSP Introduction Strategy ...12

3 TSP Results...15

4 A First-Time TSP Project ...17

4.1 Data Source ...17

4.2 PSP For Engineers Training Results...17

4.2.1 Time and Size Range ...18

4.2.2 Time and Size Estimating Accuracy ...19

4.2.3 Compile...20

4.2.4 Design Time Range ...21

4.2.5 Unit Test ...22

4.2.6 Productivity ...23

4.2.7 Class Summary ...24

4.3 The Project Results ...24

4.3.1 The Launch...24

4.3.2 Plan Summary ...26

4.3.3 Executing The Plan...26

5 Summarized Project Data ... 35 5.1 Data Source ... 35 5.2 Results... 36 5.2.1 Schedule Deviation... 36 5.2.2 Quality ... 37 5.2.3 Quality is Free ... 39

5.2.4 Comparing Summary of Results ... 39

5.3 Summarized Project Data – Conclusion ... 40

6 Anecdotes ... 43

6.1 Data Source ... 43

6.2 Anecdotal Results ... 43

6.2.1 Introduction Strategy ... 44

6.2.2 Conviction to Change ... 45

6.2.3 Realistic Plans/Schedule Performance ... 47

6.2.4 Commitment ... 49

6.2.5 Minimum Schedules ... 51

6.2.6 Barriers to Success ... 54

6.2.7 Problems You Had With the TSP... 56

6.2.8 General Comments ... 57

6.3 Anecdotes – Conclusion ... 60

7 Conclusion ... 61

Appendix A: PSP Empirical Study Replication ... 63

Appendix B: Survey Questions... 79

List of Figures

Figure 1: Elements of the PSP and the TSP...5

Figure 2: The PSP Course ...6

Figure 3: The TSP Launch ...9

Figure 4: The TSP Launch Products ... 11

Figure 5: TSP Introduction Timeline ...13

Figure 6: Time and Size Range...18

Figure 7: Time and Size Estimating Error Range...19

Figure 8: Compile Defect Density and Compile Time ...20

Figure 9: Design Time Range...21

Figure 10: Unit Test Defect Density and Unit Test Time ...22

Figure 11: Productivity Range ...23

Figure 12: Earned-Value Plan ...26

Figure 13: Earned Value Through Week 46...28

Figure 14: Average Weekly Task Hours...29

Figure 15: Effort Distribution...30

Figure 16: Defects Removed By Phase...31

Figure 17: Average Defect Density of Delivered Software ...38

Figure 18: Introduction Strategy and Timeline ...44

Figure 20: Commitment to the Plan ... 49

Figure 21: Balanced Team Workload ... 51

Figure 22: Defect Removal Profile and Effort Distribution ... 52

Figure 23: Distribution of Class Size ... 64

Figure 24: Average Effort by Assignment Number ... 65

Figure 25: Average Size by Assignment Number... 65

Figure 26: Average Productivity by Assignment Number ... 66

Figure 27: Average Defect Density by Assignment Number... 66

Figure 28: Group Trends in Size Estimating Accuracy ... 68

Figure 29: Group Trend in Time Estimating Accuracy ... 70

Figure 30: Group Trends in Average Defect Density ... 71

Figure 31: Group Trends in Average Defect Density ... 72

Figure 32: Group Trends in Average Defect Yield ... 74

Figure 33: Group Trends in Productivity... 76

Figure 34: PSP Usage ... 80

Figure 35: TSP Usage ... 81

Figure 36: Organization Types... 82

Figure 37: Software Product Types... 83

List of Tables

Table 1: PSP Class Results ...24

Table 2: Plan Summary ...26

Table 3: Team Status at Week 13 ...27

Table 4: Week 46 Status ...28

Table 5: Average Weekly Task Hour Improvement...29

Table 6: Final Status ...31

Table 7: Plan Vs. Actual ...32

Table 8: Schedule Deviation ...37

Table 9: Quality...38

Table 10: Reductions In System Test Defects and System Test Duration...39

Table 11: Improvements in Productivity and Cost Of Quality...39

Table 12: Results Comparison Between 2000 and 2003...40

Table 13: Number of Engineers Reporting Totals by Assignment Number...64

Table 14: Availability of Phase-Specific Effort by Assignment...65

Table 15: Sample Size for Each Measure ...66

Table 16: Yields for Each Assignment...75

Acknowledgements

One of the principal components of the Team Software Process is building and sustaining cohesive teams. Our team at the SEI uses the TSP, and we feel we work with a great team. First, we must acknowledge our team leader, Jim Over. Jim gently reminds us of the com-mitments we made during our launches and keeps us from endlessly expanding the scope of our efforts. Jim always provides the right technical advice at the exact moment we need it. We also thank Jim for allowing us to spend three days in Knoxville to achieve a hundred per-cent earned value on this report. We thank Jim McHale, who always has a piece of candy ready just when we need it most, and who never loses his sense of humor, no matter how stressful things get. We thank Marsha Pomeroy-Huff for her editorial expertise, and no, Mar-sha, English is not our second language. We thank Dan Burton, who provided the most com-prehensive feedback on the draft version of this report and caused us endless hours of discus-sions to resolve his insightful comments. We thank Anita Carleton, our data goddess, for setting a standard we will never live up to. If there are any data not properly presented in this report, it is entirely our fault and no reflection on Anita’s comprehensive review. We thank Alan Willett for being an all around great team member and motivator, even though he didn’t provide any feedback on this report until the very last minute. We thank Kim Campbell, who is responsible for gathering much of the data that was the input to this report. She is a great interface between the TSP project and the PSP/TSP community. Jodie Spielvogle best exem-plifies being a TSP team member. She’s the first one to step up and take responsibility, and she provided invaluable support in creating this report. Last but not least, we thank Watts Humphrey. Watts is a team member and so much more. Technically, Watts keeps our team all focused in the same direction. He provides the constancy of purpose required to support a long-term technology transition effort like this. He’s a role model. He sets the ideal that we all strive to meet. However, we do not recommend using a stopwatch to measure interrupts when talking to your spouse; it doesn’t work. But what we most admire about Watts is his innate belief in the goodness of people; that given the opportunity, people will do the right thing. We know this is a long paragraph, but we don’t get many opportunities to thank our team.

Very special thanks to Jeannine Siviy, who led the development and administration of the survey on which some portions of this report are based. Jeannine also met with our commu-nity for discussion, analyzed and organized the survey results, established the SEI TSP data repository, worked with transition partners to understand their TSP data, and spent countless hours on initial versions of this report. We would also like to acknowledge Ken Smith for all his work with Jeannine.

Jim Herbsleb, from the Carnegie Mellon University School of Computer Science, and Jeff Roberts and Wai Fong Boh, from the Carnegie Mellon University Graduate School of Indus-trial Administration, developed Appendix A of this report. We thank them for their work in replicating the study originally conducted by Will Hayes and Jim Over in 1997, and for al-lowing us to use the results of the replicated study in this report.

The TSP does not work without supportive management. In this, we are lucky to have Bill Peterson, who encourages us to be a self-directed team. We also thank Bill for his review comments of this report.

Thanks to Darryl Davis for his insightful comments on the report. It is great to have expert advice available 24/7.

Pamela Curtis did an excellent job of providing us with editorial support. Thanks, Pamela.

Julia would like to acknowledge her father, Richard Gale, because during the throes of the final rewrite, she forgot to call him on his birthday.

And finally, we would like to thank the TSP community of coaches, instructors, team leaders, team members and managers. Thank you for helping us improve the method, and please keep sending us more data and more anecdotes about your experiences with the TSP.

Executive Summary

Most software organizations critically need better cost and schedule management, quality management, and cycle-time reduction. This report demonstrates that teams using the Team Software Process (TSP) meet these critical business needs by delivering essentially defect-free software on schedule and with better productivity.

The report starts with an overview of the TSP to provide the context for the results reported. These results include the benefits realized by a first-time TSP team, a summary of data from 20 TSP projects in 13 organizations, and stories from people who have used the TSP.

These TSP teams delivered their products an average of 6% later than they had planned. The schedule error for these teams ranged from 20% earlier than planned to 27% later than planned. This compares favorably with industry data that show over half of all software pro-jects were more than 100% late or were cancelled. These TSP teams also improved their productivity by an average of 78%.

The teams met their schedules while producing products that had 10 to 100 times fewer de-fects than typical software products. They delivered software products with average quality levels of 5.2 sigma, or 60 defects per million parts (lines of code). In several instances, the products delivered were defect free.

The TSP improves schedule and quality management by creating an environment where indi-viduals and teams can routinely do excellent work. This report concludes with stories and anecdotes that illustrate the personal rewards of using the TSP.

Abstract

Most software organizations are facing critical business needs for better cost and schedule management, effective quality management, and cycle-time reduction. The Team Software Process addresses these critical business needs. This report provides results and implementa-tion data from projects and individuals that have adopted the TSP. The results show that TSP teams are delivering essentially defect-free software on schedule, while improving productiv-ity. These data can be used for benchmarking and planning, motivation, lessons learned, and other guidance to those currently using the TSP or considering its use. The report also illus-trates adoption experiences of practitioners in the field, including TSP team members, their managers, and their coaches and instructors.

1 Introduction

The success of organizations that produce software-intensive systems depends on well-managed software development processes. Implementing disciplined software methods, how-ever, is often challenging. Organizations seem to know what they want their teams to be do-ing, but they struggle with how to do it. The Team Software ProcessSM (TSPSM), coupled with the Personal Software ProcessSM (PSPSM), was designed to provide both a strategy and a set of operational procedures for using disciplined software process methods at the individual and team levels. Organizations that have implemented the TSP and PSP have experienced significant improvements in the quality of their software systems and reduced schedule de-viation [Ferguson 99, McAndrews 00].

The purpose of this report is to provide updated results on the use of the PSP and the TSP. The report starts with an overview of the PSP and the TSP to provide a context for the results reported. This is followed by a detailed description of the experiences of and benefits real-ized by a first-time TSP team. Often when TSP teams report their results, those results are met by skepticism. We hope that by walking through this first-time team’s journey of using the TSP, we illustrate how the TSP creates an environment where skilled engineers can apply disciplined methods working on a cohesive and dedicated team. Next, we summarize the performance of more than 20 projects from 13 organizations that have used the PSP and the TSP. Then anecdotes from those in the PSP and TSP communities are presented to show how individuals view the PSP and the TSP. The report concludes with an appendix presenting the results of a study that reexamines the major hypothesis regarding the impact of the PSP on individual engineers first presented in a 1997 technical report [Hayes 97].

Practitioners and launch coaches will find that the data in this report provide useful guidance in implementing the TSP. For instance, the results provided can be used for benchmarking purposes. Benchmarks can be used by teams to set team goals and to identify their strengths and weaknesses. This report can also be used by people who are interested in using the TSP in their organization to show their management and teams the benefits of the TSP.

SM _{Personal Software Process, PSP, Team Software Process, and TSP are service marks of Carnegie}

2 TSP Overview

The objective of the TSP is to create a team environment that supports disciplined individual work and builds and maintains a self-directed team. The TSP guides self-directed teams in addressing critical business needs of better cost and schedule management, effective quality management, and cycle-time reduction. It defines a whole product framework of customiza-ble processes and an introduction strategy that includes building management sponsorship, training for managers and engineers, coaching, mentoring, and automated tool support.

The TSP can be used for all aspects of software development: requirements elicitation and definition, design, implementation, test, and maintenance. The TSP can support multi-disciplinary teams that range in size from two engineers to over a hundred engineers. It can be used to develop various kinds of products, ranging from real-time embedded control sys-tems to commercial desktop client-server applications.

The TSP builds on and enables the PSP. The PSP shows engineers how to measure their work and use that data to improve their performance. The PSP guides individual work. The TSP guides teamwork and creates an environment in which individuals can use the PSP to excel. Data from early pilots show that the TSP has been successful in addressing critical business needs [Ferguson 99, McAndrews 00].

2.1 History

In the 1980s, Watts Humphrey guided the development of the Capability Maturity Model® for Software (SW-CMM). An early misperception of SW-CMM by some people was that it did not apply to small organizations or projects. In order to illustrate its application to small or-ganizations, Humphrey took on the challenge to apply the SW-CMM to the smallest organi-zation possible: an organiorgani-zation of a single individual. From 1989 to 1993, Humphrey wrote more than 60 programs and more than 25,000 lines of code (LOC). In developing these 60 programs, Humphrey used all of the applicable SW-CMM practices up through Level 5. He concluded that the management principles embodied in the SW-CMM were just as applicable to individual software engineers. The resulting process was the PSP. He subsequently worked on corporate and academic methods to train others to use the PSP technology.

® _{Capability Maturity Model and CMM are registered in the U.S. Patent and Trademark Office by}

As engineers started applying their PSP skills on the job, it was soon discovered that they needed a supportive environment that recognized and rewarded sound engineering methods. In many organizations, the projects in crisis receive all the attention. Projects and individuals who meet commitments and do not have quality problems often go unnoticed. Humphrey found that if managers do not provide a supportive environment and do not ask for and con-structively use PSP data, engineers soon stop using the PSP. Humphrey then developed the Team Software Process to build and sustain effective teams.

2.2 What Makes PSP and TSP Work

Typical software projects are often late, over budget, of poor quality, and difficult to track. Engineers often have unrealistic schedules dictated to them and are kept in the dark as to the business objectives and customer needs. They are required to use imposed processes, tools, and standards, and often take shortcuts to meet schedule pressures. Very few teams can con-sistently be successful in this environment. As software systems get larger and more com-plex, these problems only get worse.

The best projects are an artful balance of conflicting forces. They must consider business needs, technical capability, and customer desires. Slighting any facet can jeopardize the suc-cess of the project. To balance these conflicting forces, teams must understand the complete context for their projects. This requires self-directed teams that

• understand business and product goals

• produce their own plans to address those goals • make their own commitments

• direct their own projects

• consistently use the methods and processes that they select • manage quality

Figure 1 illustrates how the PSP and TSP build and maintain self-directed teams. Successful self-directed teams require skilled and capable individual team members. Capable team members are critical because each instruction of a software module is handcrafted by an indi-vidual software engineer. The engineer’s skills, discipline, and commitment govern the qual-ity of that module and the schedule on which that module is produced. In turn, the modules come together to compose software products. Therefore, a software product is a team effort. The product’s modules are designed, built, integrated, tested, and maintained by a team of software engineers whose skills, discipline, and commitment govern the success of the pro-ject.

Team Member Skills Team Building Team Management Process discipline Performance measures Estimating & planning skills Quality management skills Goal setting Role assignment Tailored team process Detailed balanced plans Team communication Team coordination Project tracking Risk analysis 3 6 3 763

Figure 1: Elements of the PSP and the TSP

The objective of the PSP is to put software professionals in charge of their work and to make them feel personally responsible for the quality of the products they produce. The objectives of the TSP are to provide a team environment that supports PSP work and to build and main-tain a self-directed team. PSP and TSP are powerful tools that provide the necessary skills, discipline, and commitment required for successful software projects.

2.3 The

PSP

The PSP is based on the following planning and quality principles [Humphrey 00]:

• Every engineer is different; to be most effective, engineers must plan their work and they

must base their plans on personal data.

• To consistently improve their performance, engineers must measure their work and use

their results to improve.

• To produce quality products, engineers must feel personally responsible for the quality of

their products. Superior products are not produced by accident; engineers must strive to do quality work.

• It costs less to find and fix defects earlier in a process than later. • It is more efficient to prevent defects than to find and fix them. • The right way is always the fastest and cheapest way to do a job.

Today, most software engineers do not plan and track their work, nor do they measure and manage product quality. This is not surprising, since engineers are neither trained in these disciplines nor required to use them. The dilemma is that until they try using disciplined methods, most software engineers do not believe that these methods will work for them. They won’t try these methods without evidence, and they can’t get the evidence without try-ing the methods. The PSP addresses this dilemma by putttry-ing an engineer in a course

envi-ronment to learn the methods. The engineers use the methods in the course and can see from their personal and class data that the methods can and do work for them.

The PSP course is composed of ten programming assignments and five reports. The PSP methods are introduced in six upwardly compatible steps, PSP0 through PSP 2.1 (see Figure 2). The engineers write one or two programs at each step and gather and analyze data on their work. Then they use their data and analyses to improve their work.

PSP0 •Current process •Basic measures PSP1 •Size estimating •Test report PSP2 •Code reviews •Design reviews PSP2.1 Design templates PSP1.1 •Task planning • Schedule planning PSP0.1 •Coding standard •Process improvement proposal •Size measurement

Introduces process discipline and measurement Introduces estimating and

planning Introduces quality management and design

Figure 2: The PSP Course

PSP0 and PSP0.1. Engineers write three programming assignments using PSP0 and PSP0.1. The objective is for the engineer to learn how to follow a defined process and to gather basic size, time, and defect data.

PSP1 and PSP1.1. Once engineers have gathered some historical data, the focus moves to estimating and planning. Engineers write three programming assignments using PSP1 and PSP1.1. Engineers learn statistical methods for producing size and resource estimates, and use earned value for schedule planning and tracking.

PSP2 and PSP2.1. Once engineers have control of their plans and commitments, the focus of the course then changes to quality management. Engineers write four programming as-signments using PSP2 and PSP2.1. Engineers learn early defect detection and removal methods and improved design practices.

Mid-term and final reports. After the first six assignments have been completed, engineers write mid-term reports, and after all ten programming assignments have been completed, en-gineers write final reports. These reports document the enen-gineers’ analyses of their perform-ance. Engineers are required to analyze their data to understand their current performance, to define challenging yet realistic goals, and to identify the specific changes that they will make to achieve those goals.

By the end of the course, engineers are able to plan and control their personal work, define processes that best suit them, and consistently produce quality products on time and for planned costs.

In 1997, a study was conducted to analyze the impact of PSP training on 298 software engi-neers [Hayes 97]. This study found that engiengi-neers were able to significantly improve their estimating skills and the quality of the software products they produced. Engineers were able to achieve these notable improvements without negatively affecting their productivity. In terms of product quality and schedule variance, individuals were able to perform at a level that one would expect from a SW-CMM Level 5 organization.

The 1997 study was recently repeated on a much larger data set of over a thousand software engineers. The larger data set represents a more diverse group of instructors, engineers, pro-gramming languages, development environments, etc. The purpose of the replication was to demonstrate the statistically significant improvements in estimating and quality practices, i.e., to answer the question, can engineers learn to use their data to significantly improve their performance? The results from this replication are essentially the same as in the original study, with some minor differences. The findings are presented in Appendix A of this report.

2.3.1 PSP

Measurement

Framework

Engineers collect three basic measures: size, time, and defects. For the purposes of the PSP course, size is measured in lines of code (LOC). In practice, engineers use a size measure appropriate to the programming language and environment they are using; for example, num-ber of database objects, numnum-ber of use cases, numnum-ber of classes, etc. In order to ensure that size is measured consistently, counting and coding standards are defined and used by each engineer. Derived measures that involve size, such as productivity or defect density, use new and changed LOC (N LOC) only. “New and changed LOC” is defined as lines of code that are added or modified; existing LOC is not included in the measure. Time is measured as the direct hours spent on each task. It does not include interrupt time. A defect is anything that detracts from the program’s ability to completely and effectively meet the users’ needs. A defect is an objective measure that engineers can identify, describe, and count.

Engineers use many other measures that are derived from these three basic measures. Both planned and actual data for all measures are gathered and recorded. Actual data are used to track and predict schedule and quality status. All data are archived to provide a personal his-torical repository for improving estimation accuracy and product quality. Derived measures include:

• estimation accuracy (size/time) • prediction intervals (size/time) • time in phase distribution

• defect injection distribution • defect removal distribution • productivity

• reuse percentage • cost performance index • planned value

• earned value

• predicted earned value • defect density

• defect density by phase • defect removal rate by phase • defect removal leverage • review rates

• process yield • phase yield

• failure cost of quality (COQ) • appraisal COQ

• appraisal/failure COQ ratio

2.4 The

TSP

The TSP is based on the following principles:

• The engineers know the most about the job and can make the best plans. • When engineers plan their own work, they are committed to the plan. • Precise project tracking requires detailed plans and accurate data. • Only the people doing the work can collect precise and accurate data. • To minimize cycle time, the engineers must balance their workload. • To maximize productivity, focus first on quality.

The TSP has two primary components: a team-building component and a team-working or management component. The team-building component of the TSP is the TSP launch, which puts the team in the challenging situation of developing their plan.

“Successful team-building programs typically expose a group to a challenging situation that requires cooperative behavior of the entire group [Morgan 93]. As the group’s members learn to surmount this challenge, they generally form a close-knit and cohesive group. The TSP

follows these principles to mold development groups into self-directed teams. However, in-stead of using an artificial situation like rock climbing or white water rafting, it uses the team launch. The challenge in this case is to produce a detailed plan for a complex development job and then to negotiate the required schedule and resources with management.”1

2.4.1 The TSP Launch

The first step in developing a team is to plan the work, which is done during the TSP launch. The launch is led by a qualified team coach. In a TSP launch, the team reaches a common understanding of the work and the approach they will take, produces a detailed plan to guide the work, and obtains management support for the plan. A TSP launch is composed of nine meetings over a four-day period, as shown in Figure 3.

Day 1 1. Establish Product and Business Goals 2. Assign Roles and Define Team Goals Day 2 4. Build Top-down and Next-Phase Plans 5. Develop the Quality Plan 6. Build Bottom-up and Consolidated Plans Day 3 7. Conduct Risk Assessment 8. Prepare Management Briefing and Launch Report Launch Postmortem Day 4 9. Hold Management Review 3. Produce Development Strategy Day 1 1. Establish Product and Business Goals 2. Assign Roles and Define Team Goals Day 2 4. Build Top-down and Next-Phase Plans 5. Develop the Quality Plan 6. Build Bottom-up and Consolidated Plans Day 3 7. Conduct Risk Assessment 8. Prepare Management Briefing and Launch Report Launch Postmortem Day 4 9. Hold Management Review 3. Produce Development Strategy

Figure 3: The TSP Launch

The first step in the launch is for the team to understand what they are being asked to do. This is accomplished in meeting 1 by having marketing (or an appropriate customer represen-tative) and management meet with the team. Marketing describes the product needs. Man-agement describes the business needs and any resources and constraints under which the team will have to work. This is also a chance for management to motivate the team. The team has the opportunity to ask any questions they might have about the product or business needs. In the next seven meetings, the team develops an engineering plan to meet the business needs.

In meeting 2, the team sets its goals and organizes itself. The team reviews the business and product goals presented in meeting 1, and derives a set of measurable team goals. Next, the

team also decides which team members will take on which routine team management tasks. These tasks are designated by manager roles:

• customer interface manager • design manager • implementation manager • test manager • planning manager • process manager • support manager • quality manager

Each team member selects at least one role. For teams with more than eight members, roles are shared. With smaller teams, team members may select multiple roles.

In launch meeting 3, the team determines its overall project strategy. The team members produce a conceptual design, devise the development strategy, define the detailed process they will use, and determine the support tools and facilities they will need. They list the products to be produced.

In meeting 4, the team develops the team plan. This is done by estimating the size of the products to be produced, identifying the general tasks needed to do the work and estimating their effort, defining the tasks for the next development cycle to a detailed work-step level, and drawing up a schedule of the team’s availability week by week through the completion of the project.

In meeting 5, the team defines a plan to meet its quality goals. The team does this by estimat-ing the number of defects injected and removed in each phase and then calculatestimat-ing the defect density of the final product. The team ensures that the tasks needed to achieve its quality goal are included in the team plan. The quality plan provides a measurable basis for tracking the quality of the work as it is done.

In meeting 6, tasks on the team plan for the next cycle of work are allocated to team mem-bers, and each team member creates an individual plan. In building their plans, the engineers refine the team estimates using their own historical data, break large tasks into smaller tasks to facilitate tracking, and refine their hours available per week to work on this project. The team meets again to review the individual task plans and to ensure that the work load is bal-anced. The individual plans are consolidated into a team plan. The team uses this plan to guide and track its work during the ensuing cycle.

The team conducts a risk assessment in meeting 7. Risks are identified and their likelihood and impact are assessed. The team defines mitigation and contingency plans for high-priority risks. Risks are documented in the team plan and assigned to team members for tracking.

Meeting 8 is used to develop a presentation of the team’s plan to management. If the team’s plan does not meet management goals, the team includes alternative plans that come closer to meeting management’s goals. For instance, the team might be able to meet a schedule by adding resources to the team or by reducing the functionality delivered.

By the end of the launch, the team has formed a cohesive unit and created a plan that bal-ances the needs of the business and customer with a feasible technical solution. The team has agreed on the technical solution that they propose to build and understands how that product will satisfy business and customer needs. The team agrees on the strategy and process for developing the product. The team has a detailed plan that it can use to guide and track the work. Team members all know who is responsible for which tasks and areas. Everyone on the team understands and agrees with the quality goal, and the team can monitor progress against that goal. Finally, the team has explored all of the things that might go wrong and has done its best to mitigate those risks. In short, the TSP launch provides a team with all of the conditions necessary to become a self-directed team.

In meeting 9, the team presents the plan to management for their approval to start the work. The team explains the plan, describes how it was produced (Figure 4), and demonstrates that all team members agree with and are committed to the plan. If the team has not met man-agement’s objectives, it presents one or more alternative plans. The principal reason for showing alternative plans is to provide management with options to consider in case the team’s plan does not meet the organization’s business needs.

W hat? How ? W hen? W ho? How w ell?

W hat if?

Business needs Managem ent goals Product requirem ents

• Team goals • Conceptual design • Planned products • Size estimates • Team strategy • Team defined process • Team roles • Task plans • Earned-value plan • Task plan • Schedule plan • Earned-value plan • Quality plan • Risks • Alternative plans

W hat? How ? W hen? W ho? How w ell?

W hat if?

Business needs Managem ent goals Product requirem ents

• Team goals • Conceptual design • Planned products • Size estimates • Team strategy • Team defined process • Team roles • Task plans • Earned-value plan • Task plan • Schedule plan • Earned-value plan • Quality plan • Risks • Alternative plans

At the end of the TSP launch, the team and management agree on how the team will proceed with the project. The team has a plan it believes in, is committed to, and can track against. The launch not only creates a winning plan, it builds a cohesive team.

The TSP includes guidance for ensuring that the energy and commitment from a TSP launch are sustained as the team does its work. A TSP coach works with the team and the team leader to help the team to collect and analyze data, follow the process defined by the team, track issues and risks, maintain the plan, track progress against goals (especially the team’s quality goal), and report status to management.

2.4.2 TSP

Measurement

Framework

The TSP uses the same basic measures of the PSP—size, time, and defects—and adds task completion dates. For all measures, planned and actual data are collected at the individual level. The TSP measurement framework consolidates individual data into a team perspective. The data collected are analyzed weekly by the team to understand project status against schedule and quality goals. The TSP measurement framework also makes available other views of the data, such as by product or part, phase, task, week, day, etc. Personal and team data are archived to provide a repository of historical data for future use.

The team conducts weekly meetings to report progress against their plans and to discuss team issues. They also use their TSP data to make accurate status reports to management on a regu-lar basis. Because management can rely on the data, their job changes from continuously checking project status to ensuring that there are no obstacles impeding the team’s progress. This also allows management to make sound business decisions, since they are based on ac-curate engineering data. For example, when management is confident in the team’s estimate, management can decide how to allocate resources to obtain a schedule that best meets the business needs. When a team commitment is in jeopardy, the team solves the problem or raises the issue with management as early as possible. In all cases and at all levels, decisions are made based on data.

2.4.3 The TSP Introduction Strategy

The SEI has been transitioning TSP into organizations since 1997 and has gained significant experience with issues surrounding the introduction of this technology. Based on these ex-periences, the SEI has defined an introduction strategy (Figure 5) and has developed support-ing materials to facilitate the implementation of that strategy.

The introduction strategy starts with trial use. The TSP is first piloted on several small pro-jects to evaluate both the transition approach and the impact of TSP on the organization. The pilots also build the understanding, sponsorship, and support needed for broad acceptance of the TSP in the organization.

Task

Select participants, develop schedule X

Train managers, engineers, instructors X X X

Conduct TSP pilots X X

Train transition agents X X

Plan and initiate roll-out X

Figure 5: TSP Introduction Timeline

All team members and all of their management are trained prior to the start of the pilot effort. The senior management attends a one-and-a-half-day executive seminar and planning ses-sion; the middle and line management attend three days of training; the engineers complete the two-week PSP for Engineers course. The pilot teams are then started with a launch, and they begin to use the TSP process as they do project work. Pilot projects can rapidly demon-strate the benefits of using the TSP, and results from the pilot projects can be used to tailor and improve both the TSP and the introduction strategy.

3 TSP Results

The remainder of this technical report describes the usage and experiences that illustrate the benefits of TSP from three perspectives. We begin with a first-time project perspective. The experiences of a first-time TSP team are described in detail, from PSP training through prod-uct delivery. Next, we present some summarized project data. A summary of TSP data from thirteen organizations and at least twenty projects is presented. Finally, we switch from data to personal experiences. We present anecdotes from the PSP and TSP communities that relate experiences which cannot be described by data alone.

For each of these three perspectives, we first describe the source of the data. We then present the data. And finally we describe conclusions drawn from that data.

4 A First-Time TSP Project

4.1 Data

Source

The team described in this section was part of a multinational company that has been in-volved with CMM-based software process improvement for several years. The organization worked with the SEI to use the standard TSP introduction strategy to pilot test the TSP on two projects. The data included in this section represents one of those pilot projects. We chose this project for several reasons. We have complete data on the project, from the PSP for Engineers course data, TSP launch data, and through to project completion. Also, we feel that this project’s experiences typify what many first-time TSP teams encounter. In fact, this project was the first TSP pilot project to finish in this organization. And finally, this project encountered problems most software development projects do, such as effort underestimation and changing requirements.

4.2 PSP For Engineers Training Results

The class data presented here represents the engineers who completed most of the PSP for Engineers training. The data represents 84% of the 22 engineers who completed at least 9 of the 10 programs, and 4 of the 5 reports. The class included members of the pilot project de-scribed in this report, as well as other engineers. Throughout the PSP training, the key meas-ures defined in the PSP Measurement Frameworks section (page 7) of this report are tracked for each of the 10 programs the engineers write. For each assignment, the class average and the minimum and maximum values for that measure are plotted. Don’t assume that the maximum or minimum data point represents the same engineer. This is almost never the case. Engineers are presented with their class data throughout the course to illustrate the benefits the class is realizing from using these methods.

4.2.1 Time and Size Range

The charts in Figure 6 show the time spent developing each program in the course, as well as the size of that program.

Actual Time Range

0 2 4 6 8 10 12 14 1 2 3 4 5 6 7 8 9 10 Program Number Hour s Max Avg Min

Actual Size Range

0 100 200 300 400 500 600 1 2 3 4 5 6 7 8 9 10 Program Number N L O C Max Avg Min

Figure 6: Time and Size Range

While the maximum and minimum time to develop each program varies widely, the class average is around four hours per assignment. Similarly the size range varies greatly between the maximum and minimum size. The size variation often represents different solutions for the same requirement. In this class, size variation was not due to differences in language or counting standards. All engineers used either C or C++ and the same counting standard.

4.2.2 Time and Size Estimating Accuracy

Figure 7 shows the time and size estimating error for each of the ten programs. The time and size estimation error is calculated as

%Estimate Error = 100 * (Actual – Estimate)/Estimate

Note that, unlike the rest of the charts in this section, the class line is a composite estimate for all engineers. For time, the composite estimate is the sum of the engineers’ estimated hours. Similarly, the composite actual time is the sum of the engineers’ actual hours. The class line is the error between the composite estimated hours and the composite actual hours. The class line for size estimating error is calculated similarly.

Time Estimating Accuracy - % Error

-100 0 100 200 300 400 500 1 2 3 4 5 6 7 8 9 1 Program Number % E s tima te E rro r Max Clas s Min

Size Estimating Accuracy - % Error

-150 -100 -50 0 50 100 150 200 250 300 350 1 2 3 4 5 6 7 8 9 1 Program Number % E rr o r _Max Clas s Min

Figure 7: Time and Size Estimating Error Range

As can be seen, the composite estimation error appears to be stable, with an improving trend. This illustrates the benefits of independent, biased estimates. All engineers make non-biased estimates using statistical methods and historical data. Some engineers overestimate, while others underestimate, thus balancing the composite estimate. This helps engineers to understand the importance of breaking a problem down into parts and estimating each part independently using non-biased statistical methods.

Another point to note in the chart is the narrowing range of the estimation error. As engineers learn and use defined effort estimation procedures, the range between the maximum and minimum estimation error narrows.

As with the time estimate, the composite size estimate appears to be stable, with an improv-ing trend, again illustratimprov-ing the benefits of combinimprov-ing non-biased estimates. The range of the size estimation error narrows as the engineers learn to use a defined size estimation

proce-dure. (Note: Engineers did not estimate program size for program 1, hence the zero percent error for that data point.)

4.2.3 Compile

“Compile defect density” is the number of defects found by the compiler per thousand new and changed LOC. “Compile time” is the time it takes the engineer from when he or she starts compiling a program to when he or she gets a clean compile. The charts in Figure 8 show the defects found during compile and compile time as a percentage of total develop-ment time.

Compile Defect Density Range

0 50 100 150 200 250 300 1 2 3 4 5 6 7 8 9 10 Program Number D e fe c ts /N KL O C Max Avg Min

Compile T ime R ange

0 5 10 15 20 25 30 35 40 45 1 2 3 4 5 6 7 8 9 10 Program Number % of To ta l Ti m e Max Avg Min

Figure 8: Compile Defect Density and Compile Time

In most development environments, compile errors are the only objective measure of the quality of the code. Reviews, inspections, and test defects can all be affected by the quality of the reviewers or the quality of the tests. Compile defect density is an early indicator of final product quality.

Engineers started the course spending about 9% of total development time removing about 51 defects per thousand lines of code (defects/KLOC) in compile.2 After the introduction of per-sonal code reviews and perper-sonal design reviews in program 7, engineers are able to remove most defects before compile. By the end of the course, engineers are spending less than 2% of total development time in compile, removing an average of 9 defects/KLOC. The quality of the code the engineers were compiling improved by a factor of five, thus time spent in the compile phase was minimal.

4.2.4 Design

Time

Range

In the PSP, design time includes the time spent reviewing the requirements and producing a design to meet them. The chart in Figure 9 shows the percentage of total development time that engineers spent in the design activities.

0 10 20 30 40 50 60 1 2 3 4 5 6 7 8 9 10 Program Number % of Tot al Ti m e Max Avg Min

Figure 9: Design Time Range

Engineers use their existing methods through program 7. Starting with program 8, engineers are introduced to robust design specification methods. Engineers are surprised to see the small percentage of time spent in design activities prior to the introduction of these methods. For example, this class’s data show that at the beginning of the course, engineers were spend-ing as much time compilspend-ing their code as they were in designspend-ing their programs. Engineers claim to love design but seem most comfortable spending their time in compile and unit test (see Figure 8 and Figure 10).

4.2.5 Unit

Test

Unit test defect density is the number of defects found during unit testing per thousand new and changed LOC. Unit test time is the time it takes the engineer from when he or she starts to test a program to when all tests run successfully. The charts in Figure 10 show the defects found during unit test and unit test time as a percentage of total development time.

Defects Found in Test Range

0 20 40 60 80 100 120 140 160 180 200 1 2 3 4 5 6 7 8 9 10 Program Number De fe c ts /N KL O C Max Avg Min

Test Time Range

0 10 20 30 40 50 60 70 80 1 2 3 4 5 6 7 8 9 10 Program Number % o f T o ta l T ime Max Avg Min

Figure 10: Unit Test Defect Density and Unit Test Time

Engineers started the course spending about 22% of total development time in unit test, re-moving about 31 defects/KLOC. With the introduction of quality methods in program 7 (personal design and code reviews and robust design specifications), these numbers are re-duced. By the end of the course, engineers are spending less than 14% of total development time in unit test, removing an average of 8 defects/KLOC. Also, note the variance around the class average. The range between the maximum and minimum value narrows, but not be-cause the minimum defect density worsens, but bebe-cause the maximum defect density is re-duced. This considerably improves predictability.

The quality of the code the engineers were testing improved by a factor of four (from 31 de-fects/KLOC to 8 dede-fects/KLOC). Thus time spent in the unit test phase was minimized.

4.2.6 Productivity

Productivity is defined as development time per new and changed LOC. The chart in Figure 11 shows the change in productivity over the ten programming assignments.

0 20 40 60 80 100 120 140 160 1 2 3 4 5 6 7 8 9 10 Program Number N LO C /H our _Max Avg Min

Figure 11: Productivity Range

An argument frequently made is that disciplined methods take too much time. These data clearly show that this is not the case for this team. There is no impact to productivity, yet the quality of the product entering unit test improved by a factor of four.

4.2.7 Class

Summary

The results from PSP training were impressive, and consistent with results documented in Appendix A and in a previous SEI report [Hayes 97]. These results are summarized in Table 1. The first column describes the measure, the second column shows its value at the start of PSP training (class average for the first two programs), and the third column shows its value at the end of PSP training (class average for the last two programs).

Measure At the start of training At the end of training

Percent time spent in compile 9% 2%

Percent time spent in design 10% 17%

Percent time spent in unit test 23% 14%

Compile defect density (number of defects found during compile per thousand lines of code [KLOC])

51 defects/KLOC 9 defects/KLOC

Unit test defect density (number of defects found during unit test per KLOC)

31 defects/KLOC 8 defects/KLOC

Yield (percentage of defects found before first compile) 5% 55% Productivity (detailed design through unit test) 43 LOC/hour 53 LOC/hour

Table 1: PSP Class Results

4.3 The

Project

Results

Less than a month after completing the PSP for Engineers training, the first TSP project was launched. The team consisted of five team members, including the team leader. The team leader was open-minded about the TSP and fully supported its use. Of the other four team members, one had fully embraced the PSP, another felt comfortable with using disciplined processes, and the other two were skeptical.

4.3.1 The

Launch

Senior management was well prepared for the launch. Management discussed the importance of this product to the business, both from a functionality point of view and a sales potential point of view. Management told the team that their expectation was for the software to be delivered to the testing group in nine months.

A marketing representative followed the senior management presentation, and discussed the product needs with the team. The team members asked several questions, and launch meet-ing 1 ended on schedule.

The launch meetings proceeded in a fairly routine manner. The team members chose roles that best suited their abilities. Conceptual design took a while to complete. First, the team wanted to go into much more detail than was necessary at this point in the launch. After some attempts to get the team to step back and do the conceptual design at a very high level, the coach let the team get into more detail, knowing that the launch process was flexible enough to accommodate this. Next, the team questioned the need to define a development strategy, the products to be produced, and the development process for each product type. There was some impatience to get to what the team considered “real planning,” or who does what task when. However, they agreed to follow the process and completed each step lead-ing up to task identification, estimation, schedullead-ing, and assignment. As the launch pro-gressed, one team member told the launch coach that this was the first time he understood what commitments had been made on his behalf prior to the launch.

The plan the team came up with showed software delivery to test eleven weeks beyond man-agement’s schedule goal. Some team members were concerned about giving bad news to management. The launch coach wanted to make sure that the entire team believed in the plan, so the launch coach asked each team member if he or she thought the plan was too con-servative, and why. Every team member agreed with the schedule the team had developed.

It was now about 7:00 p.m. on day three of the launch. The team had worked very hard for three days, and had not had dinner yet. They had completed the outline, as well as most of the content of the management presentation. The team leader and the launch coach volun-teered to fill in the outline with the data captured on flip charts and artifacts during the earlier meetings of the launch, thus allowing the team members to go home. Not one single team member accepted this offer. This was their plan, and they were not going to let the team leader or the launch coach finish the management presentation. They stayed late and dis-cussed each word and sentence until they were all satisfied the presentation represented their plan.

The next day, the team leader presented the plan to management. Management asked several questions and was especially concerned with the planned task hours per team member per week. In the end, management accepted the team plan and challenged the team to improve on the planned task hours per week.

During the launch postmortem, the team remarked on how surprised they were about the amount of work they had accomplished during the week. The team leader said the launch process was very good at guiding the team step by step through the planning process, thus allowing the team to concentrate on one thing at a time. She said that if she had known on day one everything the team would have to do during the launch, she would have been over-whelmed. The team members thought they should have been given more time before the launch to work on a conceptual design. They felt they were rushed during the size estimation

phase. Finally, the team members commented that having an experienced, neutral facilitator as a launch coach helped them to be successful.

4.3.2 Plan

Summary

The plan the team developed during the launch is summarized in Table 2. The earned-value plan developed during the launch is shown in Figure 12.

Delivery to testing group Week 47

Ready to release Week 58

Effort estimate 2814 hours

New and changed LOC 14.5 KLOC

System test defect density .36 defects/KLOC Average task hours per team member per week 15 task hours/week

Table 2: Plan Summary

Cumulative Earned Value

0.0 10.0 20.0 30.0 40.0 50.0 60.0 70.0 80.0 90.0 100.0 1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 Week s E a rn ed V a lu e

Cumulative Planned Value Cumulative EV

Cumulative Predicted Earned Value

Figure 12: Earned-Value Plan

4.3.3 Executing The Plan

As with most software projects, this team encountered many obstacles as it started executing the plan it developed during the launch: some risks were realized, some tasks were underes-timated, and requirements changed and grew. In order to simplify the story of this project,

snapshots of the project are presented at week 13, after week 46, after week 49, and after week 62. Week 13 was the week prior to the first relaunch, week 46 was the week before the original date the team committed to delivering the software to test, week 49 was when the team first delivered software to test, and week 62 was when the product was ready for re-lease.

Week 13 – One Week Before Relaunch

A high-impact risk identified during the launch was that the engineers might be distracted from working on the new product because of the excessive support required for legacy prod-ucts. This risk was realized. The team found itself providing more support than planned for an existing delayed release. By week 13, as shown in Table 3, the team was about 15% be-hind in task hours (plan of 650 project hours to date versus actual of 565.1), and had under-estimated their work by 38% (plan of 306.6 for work completed versus an actual of 497.8). They had earned only half of the earned value that they had planned to earn. If the team con-tinued to earn value at the current rate, the schedule exposure was at least six months.

Week 13 Data Plan Actual

Plan/ Actual

Project hours for this week 48.0 25.0 1.92 Project hours this cycle to date 650.0 565.1 1.15

Earned value for this week 1.8 3.3 0.54

Earned value this cycle to date 46.9 23.8 1.97 To-date hours for tasks completed 306.6 497.8 0.62

Table 3: Team Status at Week 13

In order to mitigate this schedule exposure, management decided to add three new team members to the project. During the launch, management had decided against additional re-sources, but when presented with data that showed the team was not going to make the schedule, management made a rational decision to add resources to the project. Therefore, in week 15, the team relaunched with the three new team members who had partially completed PSP training. During the relaunch, the team reestimated the remaining work. Based on les-sons learned from the first 14 weeks and the additional overhead associated with the new team members, the team created a new plan. The new effort estimate was 3328 hours, an increase of 15% over the original plan. The release date was unchanged at week 58.

Week 46 – One Week Before Scheduled Delivery To Test

The team data at week 46 (Table 4) showed that the original team commitment for delivering the product to the testing group was 3% behind schedule (planned value of 96.5 versus an earned valued of 93.6). The earned value chart, shown in Figure 13, predicted completion by week 49, two weeks behind the date committed at the initial launch. The six-month delay that been predicted in week 13 had narrowed to only a two-week delay.

Week 46 Data Plan Actual Plan/Actual

Project hours for this week 103.0 102.8 1.00 Project hours this cycle to date 3319.2 3829.8 0.87

Earned value for this week 1.7 1.0 1.75

Earned value this cycle to date 96.5 93.6 1.03 To-date hours for tasks completed 3051.7 3715.9 0.82

Table 4: Week 46 Status

Cumulative Earned Value

0 10 20 30 40 50 60 70 80 90 100 15 18 21 24 27 30 33 36 39 42 45 48 Weeks E a rn e d V a lu

e Cumulative Planned Value

Cumulative EV

Cumulative Predicted Earned Value

Figure 13: Earned Value Through Week 46

The team was able to close the schedule gap because they added three new team members, continually monitored their status and took corrective action as needed, replanned and rebal-anced tasks as needed, and added a co-op student to help with non-critical tasks. The team

also improved average task hours per person, as shown in Figure 14 (week 18 was Christmas week, thus the zero task hours that week) and Table 5. The task hour improvement did not happen by accident: the team planned this during the relaunch. The task hours were not im-proved by working overtime; in fact, the team worked much less overtime on this project than on previous ones. The task hour improvement came by increasing uninterrupted time on task, by adopting quiet time, by streamlining necessary meetings and eliminating unnecessary ones, and by adopting flexible work hours. The interesting thing to notice is that just by task hour management, the team was able to increase its productivity by 28% (from 11.36 to 15.77 average task hours per team member per week).

Average Weekly Task Hours Per Team Member

0 5 10 15 20 25 1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 Week T a sk Ho u rs

Figure 14: Average Weekly Task Hours

Average Task Hours Per Engineer Per Week

Before relaunch (weeks 1-14) 11.36 After relaunch (weeks 15-49) 15.77

Table 5: Average Weekly Task Hour Improvement

Week 49 – Delivery To Test

The team was able to deliver software to the testing group in week 49, just two weeks behind schedule. The team was about a week away from delivering the software to the testing group when they were asked to implement a new requirement. The team estimated that it would take 252 task hours to develop the additional functionality. Management did not want the original release date of week 58 to slip due to the added functionality. The team was confi-dent that the software they were delivering to system testing was of high quality. Therefore, they planned to spend very little time fixing system test defects. Using their historical

aver-age task hours, their plan showed that they could implement the additional functionality without delaying the final release date. They achieved this goal by planning to develop the additional functionality while the testing group was testing the initial functionality.

Week 62 – Project Completion

The team spent more than a third of their total development effort in quality activities such as design, reviews, and inspections (Figure 15). The software delivered to the testing group was thus of high quality. Before the software was delivered to system testing, 945 defects were removed, leaving less than 0.44 defects/KLOC to be found in system testing. Note that the team has an improvement opportunity for personal design and code reviews, but their inspec-tions worked well (Figure 16).

As with typical software projects, testing proved to be an unpredictable activity. Even though very few defects were found during system testing, one defect was not discovered until a last-minute regression test. The product was released in week 62, four weeks beyond the date to which the team had committed over a year earlier, and with more functionality than they had originally planned. The final project status is shown in Table 6 and Table 7. Compared to a previous release of a similar product, these results represent

• 10x reduction in the number of problems logged by the testing group • 8x reduction in system test duration

The team leader said the engineers were happy to have worked on a project that was deliv-ered on time. She said it was a positive experience for everyone involved. The manager of the testing group said this was one of the most stable releases that his group had ever tested. The team members all said they enjoyed working on this project.

0.00% 5.00% 10.00% 15.00% 20.00% 25.00% Pla nning an d P ost Mo rtem Req uirem ents Des ign Code Com pile Insp ection s and Revi ew s Test Set up a nd Unit Tes t Syst em T est

0.00 10.00 20.00 30.00 40.00 50.00 60.00 70.00 DLD Rev iew DLD Insp ectio n Code Rev iew Comp ile Cod e In spec tion Uni t Tes t Build and Integ ratio n Te st Syste m Te st Phase D e fe ct s/K L O C Plan Actual

Figure 16: Defects Removed By Phase

Project length 62 weeks

Schedule variance 6.9%

Percent effort in system test 4.3%3

Defects removed before delivery to testing group 945

Table 6: Final Status

3 _{Percent effort in system test is the effort the developers spent supporting the testing group (fixing}

Plan Actual

Delivery to testing group Week 47 Week 49

Ready to release Week 58 Week 62

Effort estimate4 2814 hours 3670 hours

New and changed LOC 14.5 KLOC 28.9 KLOC

System test defect density .36 defects/KLOC .44 defects/KLOC Average task hours per team member

per week

15 15.77

Table 7: Plan Vs. Actual

4.4 First-Time TSP Team – Conclusion

This project illustrated many of the problems that most software development projects face. For instance, the initial estimates were wrong, the requirements grew, and the team was con-stantly interrupted with work not directly related to the project. It’s impossible to point to any one thing and say “this is why the team succeeded.” First, the team has to know as early as possible when there is a problem. TSP teams collect data daily and review schedule and quality status weekly. This allowed this team to recognize in the early weeks of the project that they were falling behind schedule. Second, the team must understand what is causing the problem. In this case, the team had underestimated the work and overestimated available task hours. Then, the team must understand possible ways to address the problem. This team was able to take several steps to address the schedule slip. The team was able to provide management with the engineering data needed for management to make a business decision: should more resources be allocated to this project to bring in the schedule or should those resources be allocated elsewhere, letting the schedule slip? Management decided to allocate more resources to the project to maintain the original schedule. The team members analyzed their data and discovered that interruptions were preventing them from achieving their planned task hours. The team took active steps to correct that problem as well.

Another contributor to the team’s ability to meet its schedule commitment was its focus on quality. The team made a quality plan and did not abandon that plan under schedule pressure. Because the team actively managed quality, they were able to produce a high-quality product and did not get caught up in endless test and fix cycles.

Many teams don’t realize how much development time is “lost” while fixing defects that are found during system test and customer use. This time could instead be spent developing a new product or implementing new functionality. While typical software teams would be

4 _{The plan and actual effort data does not reflect the additional functionality added in week 48 of}