Evaluating the Quality of Software Engineering Performance Data

(1)

Evaluating the Quality of

Software Engineering

Performance Data

James Over

Software Engineering Institute

Carnegie Mellon University

(2)

2

Team Software Process

This material is based upon work funded and supported by Industry cost recovery under Contract No. FA8721-05-C-0003 with Carnegie Mellon University for the operation of the Software Engineering Institute, a federally funded research and

development center sponsored by the United States Department of Defense.

Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of Industry cost recovery or the United States Department of Defense.

NO WARRANTY. THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING INSTITUTE MATERIAL IS FURNISHED ON AN “AS-IS” BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTER INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE OR MERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL. CARNEGIE MELLON UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITH RESPECT TO FREEDOM FROM

PATENT, TRADEMARK, OR COPYRIGHT INFRINGEMENT.

This material has been approved for public release and unlimited distribution.

This material may be reproduced in its entirety, without modification, and freely distributed in written or electronic form without requesting formal permission. Permission is required for any other use. Requests for permission should be directed to the Software Engineering Institute at [email protected].

Carnegie Mellon®_{is registered in the U.S. Patent and Trademark Office by Carnegie Mellon University.}

Team Software ProcessSM_{and TSP}SM_{are service marks of Carnegie Mellon University.}

(3)

3

Managing Software Development

Managing software engineering is

challenging due to the nature of

the work.

Intuition

The ability to understand something

immediately without the need for

conscious reasoning

Counterintuitive

Contrary to intuition or

common-sense, but often true

(4)

4

Management Visibility

Data can provide visibility into a

software project’s past and future

performance.

•

Did the project meet key

milestones?

•

Is the project on schedule?

•

Are costs under control?

•

How does project performance

compare to other projects?

•

Will the software work when

(5)

5

Can We Trust the Data?

“In God we trust, all others

bring data” - Deming

Manufacturing data

•

standard operational definitions

•

machines measure, collect, and

report

•

precise and accurate

Software engineering project data

•

lack of standards and operational

definitions

•

people often measure, collect,

and report

(6)

6

TSP Measures

Work Product

Size

Time on Task

Defects

Resource

Availability

Schedule

TSP Measurement Framework

Five direct measures

Team and team member data

Estimated during planning

Measured while working

Evaluated weekly and when

•

task completed

•

phase completed

•

component completed

•

cycle completed

(7)

7

Measurement Context Supports Evaluation

The five direct measures include a reference to

•

Who – team and team member

•

What – project, product or component, size measure, process, process

phase, and task

•

When – project cycle, calendar, time of day

The five direct measures have dependencies that can be evaluated to test

for internal consistency, for example:

•

Work or tasks are ordered and time on task cannot overlap

•

Defects are found while performing a task and the timestamp and time to fix

cannot exceed the start and stop time for the task

•

Product size, time to perform a task, and defects injected and removed are

(8)

8

Performance Evaluation Process

Import Data

• Assess

submission

• Store data

• Add to catalog

Data

Quality

Check

• Consistency

Analysis

• Statistical

Analysis

• Falsified values

Baseline

• Project Facts

• Product Facts

• Process Facts

Benchmark

• Projects

• Organizations

Produce Certificate

(9)

9

TSP Data

Database 1

Projects

114 Time Log Entries

100,466

Defect Log Entries

10,860

Tasks

73,000

Components

8,412

Database 2

Projects

109 Time Log Entries

103,023

Defect Log Entries

18,408

Tasks

11,499

(10)

10

Example Data Quality Tests

Check for missing or incorrect value

•

time log entry without a start or stop date

•

defect log entry without a reference to the phase where the defect was

discovered

Check for internal inconsistency

•

Time log entry outside of the project start and end date, or overlaps with

another entry, or violates process sequencing

•

Defect log entry timestamp inconsistent with outside the start and end date

for the associated task

Check for statistical anomalies and expected distributions

•

Test the distribution of each direct measures to expected values

•

Outlier evaluation

(11)

11

Time Log Leading Digit Analysis

0.00% 5.00% 10.00% 15.00% 20.00% 25.00% 30.00% 35.00% 1 2 3 4 5 6 7 8 9 Fre qu en cy Leading Digit

Time Log Leading Digit Analysis

Expected Actual

The TSP Time Log tracks time spent

on tasks in the plan.

Accurate data are important.

•

How many task hours this week?

•

How many task hours to complete a

module or component?

Tracking in real time improves

accuracy.

The leading digit analysis compares

data to the ideal distribution.

(12)

12

Defect Log and Size Log Analysis

0.00% 5.00% 10.00% 15.00% 20.00% 25.00% 30.00% 35.00% 40.00% 45.00% 1 2 3 4 5 6 7 8 9 Fre qu en cy

Leading Digit (fix time)

Defect Log Leading Digit

Analysis

Expected Actual 0.00% 5.00% 10.00% 15.00% 20.00% 25.00% 30.00% 35.00% 40.00% 1 2 3 4 5 6 7 8 9 Fre qu en cy Leading Digit

Size Log Leading Digit Analysis

Expected Actual

(13)

13

(14)

14

Team Size

18 15 12 9 6 3 18 16 14 12 10 8 6 4 2 0

Team Size

Frequency

Mean

= 8.2

Median = 8.0

(15)

15

Mean Team Member Weekly Task Hours

23.6 20.0 16.4 12.8 9.2 5.6 2.0 18 16 14 12 10 8 6 4 2 0

Mean Team Member Weekly Task Hours

Frequency

Mean

= 10.3

Median = 9.0

(16)

16

Productivity (LOC/Hr)

Frequency

Productivity

Mean

= 10.3

Median = 7.1

(17)

17

Plan Vs. Actual Hours for Completed Parts

R² = 0.952

0 1,000

2,000

3,000

4,000

5,000

6,000

7,000

8,000

9,000

10,000

0 2,000

4,000

6,000

8,000

10,000

Actual Hours for

Completed Parts

(18)

18

Plan Task Hours Vs. Actual Task Hours

R² = 0.8038

0 1000

2000

3000

4000

5000

6000

7000

8000

9000

0 2000

4000

6000

8000

10000

Actual

Task Hours

(19)

19

Defect Density – Median of Defects Per KLOC

0.15 0.7 3.8 3.3 5.2 2.2 0 1 2 3 4 5 6 System Test Build/Integration Test Unit Test Code Inspection Code Review DLD Review

(20)

20

Summary

Management should be part instinct and part fact-based, but the lack of

data and facts in software engineering slows learning.

Deming urged us to trust data more than our intuition, but can the data

be trusted?

This presentation has shown a measurement system that

•

can be evaluated to determine the accuracy of project data.

•

has built-in checks and balances to guard against intentional and

unintentional errors.

•

produces data that supports estimating, planning, tracking, baselines,

(21)

21

(22)

22