Current Defect Density Statistics

(1)

You may not reprint or copy any part of this presentation withou

You may not reprint or copy any part of this presentation without express and written t express and written consent from Ann Marie Neufelder

Current Defect Density

Statistics

Ann Marie

Neufelder

Copyright SoftRel, LLC 2007

(2)

Copyright SoftRel, LLC 2007

Actual fielded defect density from 90+ projects

spanning nearly every industry

Defect density clusters

0 0.5 1 1.5 2 2.5 3 0 0.5 1 Percentile group A v e ra g e d e li v e re d n o rm a li z e d d e fe c t d e n s it y World class Very good Good Average Fair Poor Ugly

This data is in terms of fielded (escaped) defects per 1000 lines of effective code normalized to assembler.

Seven “clusters” are visible. A method to predict which cluster your project will fall into was developed from this data.

(3)

How to determine normalized effective size

Predict/count new and modified lines of code

Predict/count deleted lines

Multiply existing but unchanged code by 10%

Entire functions deleted reduce existing size

Effective size = Modified + New + Lines subtracted + (10% of exi

sting code)

Multiply effective size by conversion ratio to

assembler using industry tables as summarized

below

Second generation (C, Fortran)

–

3

3 Object oriented (Java, C++,

Object oriented (Java, C++,

Ada

9x)

–

6

6 Visual Basic

(4)

Use the SoftRel Survey to predict the best

cluster

Appropriate

“

Cluster

”

determined by these things

Inherent stability of existing design and code

Methods and techniques use to prevent defects and develop

software

Application type

Existence of major obstacles (new technology, new

environments, etc.)

Existence of major opportunities (end user domain experts

available to project, etc.)

Inherent stability of development process

Process alone will not guarantee a world class cluster!

An SEI CMM level 1 organization can be world class

An SEI CMM level 4 or 5 does not guarantee world class

(5)

How the survey score maps to the clusters

SoftRel survey score versus defect density

clusters

0

20

40

60

0

0.5

1 Percentile group

S

o

ft

R

e

l

S

u

rv

e

y

s

c

o

re

World class

Very good

Good

Average

Fair

Poor

Ugly

The most variation exists in the world class cluster, however, this cluster is

easily predictable because of void of major obstacles and presence of major

opportunities as shown next.

(6)

The World Class Cluster had no major

obstacles

Major project obstacles versus defect

density clusters

0

1

2

3

4

5

6

0

0.5

1 Percentile group

N

u

m

b

e

r

o

f

m

a

jo

r

p

ro

je

c

t

o

b

s

ta

c

le

s

World class

Very good

Good

Average

Fair

Poor

Ugly

Obstacles are defined specifically as – new technology, new operating system,

new development environment, new compiler, new target hardware

(7)

The

“

Ugly

”

group had no opportunities

Major project opportunities versus defect

density clusters

0

2

4

6

8

0

0.5

1 Percentile group

N

u

m

b

e

r

o

f

m

a

jo

r

p

ro

je

c

t

o

p

o

rt

u

n

it

ie

s

World class

Very good

Good

Average

Fair

Poor

Ugly

Opportunity – Explicitly defined as the degree to which end user domain

experts are available to the software engineers for this project

(8)

Average defect density by system type

5.1 2.104 0.414 Total/average 3.6 0.448 0.123 No special target hardware

n/a 1.0925 Power systems n/a 0.134 GPS 23.7 4.787 0.202 Small devices 3.8 2.495 0.649 Large stationery capitol

equipment 4.1 0.358 0.087 Satellite n/a 0.106

Military ground vehicle

33.9 0.366

0.011 Command, control and

communications

1.7 0.180

0.106 Command and control

Ratio of

test to

field

Average

testing

defect

density

Average

fielded

defect

density

System application type

(9)

Average defect density by software type

0.7 0.278

0.378 Application process evolving

4.0 2.588

0.642 Target HW is new or evolving

11.1 0.091 0.008 Web based 3.5 1.513 0.430 Mathematically intensive 3.2 1.459 0.456 DB interfaces 4.7 2.104 0.449 Multi-tasking 4.6 2.172 0.476 Real time 4.0 0.434 0.108 Client server n/a n/a 0.068 Domain knowledge can be acquired via public

domain in short period of time

3.2 1.290 0.400 Biometrics 18.7 3.092 0.165 Wireless capabilities Ratio of test to field Average testing defect density Average fielded defect density

Software application type

(10)

Average defect density by risk level

22.5 3.165 0.141 Government regulated 13.7 3.158 0.230 Recall risk 3.2 1.535 0.476 Monetary risks (loss of product with monetary

value)

26.9 4.539

0.169 Legal risks (banking, etc)

5.1 2.608

0.509 Safety risk (occupational, regional, national or

global) Ratio of test to field Average testing defect density Average fielded defect density

Risk level

(11)

You can also predict the risk of a late delivery

Normalized Fielded Defect Density

Percentile

Ratio of testing to fielded defect

density Ave Min Max

Stddev Probability of a late delivery (%) Margin of error when delivery is late (%) World Class 8.5 .011 0.0055 0.0180 .006 10 17.5 Very Good 12.4 .060 0.0396 0.0756 .0172 20 25 Good 10.7 .112 0.0888 0.135 .0169 25 25 Average 10.6 .250 0.180 0.366 .0590 36 41 Fair 2.1 .618 0.400 0.835 .177 85 125 Poor 16.1 1.111 1.0357 1.224 .081 100 100 Ugly .5 2.069 1.743 2.674 .524 83 75

Probability of late delivery – If your organization makes 10 releases and the

probability of being late is 10% then 1 out of 10 will be late

(12)

How to predict your cluster

Answer a survey based on

Risks

Product characteristics

Application type

Resources

Practices, techniques and methods

Process stability

Determine a baseline cluster for your

“

typical

”

project

Each project specific additional obstacle lowers

the cluster while adding domain expertise raises

the cluster

(13)

Lessons Learned

Risks cannot be overcome by any of the following

New expensive automated tools to theoretically speed up

development (this will actually increase the risk level for the

first

time project)

Wishful thinking

Risks can be minimized by

More granular milestones

Addressing high risk items before everything else in the

schedule

software engineers tend to work on the low risk tasks first

Design prototyping when design is a risk

Requirements prototyping when end user requirements are

volatile

Defect prevention techniques such as formal unit testing

Increasing the end user domain knowledge of the team

This does not mean software experience

–

this means application

experience