Measuring the Cost of Software Quality of a Large Software Project at Bombardier Transportation

44  Download (0)

Full text

(1)

Measuring the Cost of Software Quality 

Ingeniería para la Industria

g

Q

y

of a Large Software Project 

at Bombardier Transportation

p

Presented by Claude Y. Laporte, Eng., Ph.D.

Professor

Department of Software and IT Engineering

p

g

g

École de technologie supérieure, Québec, Canada

Project Editor of ISO/IEC 29110 for Very Small Entities

Professor Normand Séguin, Ph.D.

Universitad de Lima, Peru

September 13

th

2012

September, 13

th

2012

http://www2.ulima.edu.pe/

(2)

Ingeniería para la Industria

Content

Introduction

Introduction

Business Rationale for Software Quality Assurance 

and the Measurement of the Cost of Quality

and the Measurement of the Cost of Quality

Undergraduate/Graduate Software Quality 

Assurance Courses and the Measurement of the 

Cost of Quality

Measurement of the Cost of Software Quality – A 

Case Study

Conclusion

2

(3)

École de technologie supérieure

(4)

École de technologie supérieure

Over 6,3000 students

Each year 2,200 paid industrial internships in over 900 companies.

Students are paid about 36,500$ for 3 internships of 4 months

Undergraduate Programs

• Software Engineering

• 600 students (400 in Software Eng.)

20 P

f

i th d

t

t h

g

g

• IT Engineering

• Construction Engineering

• Production Engineering

• 20 Professors in the department have a

mean industrial experience of 10 years.

• Electrical Engineering

• Mechanical Engineering

• Logistics and Operations Engineering

• Graduate Programs

• Software Engineering

• Information Technology

175 students

• Information Technology

• Other programs

www.etsmtl.ca

(5)

Overview of my Background

National Defense

Defence Nationale

Oerlikon

Oerlikon

Aerospace

Aerospace

Defence Nationale

(1973‐1991)

(1991‐1999)

(1999‐2000)

f

f

d

i

i

5

Department of Software and IT Engineering

(2000 ‐ )

(6)

Project Editor 

of ISO/IEC 29110 Standards and Guides 

f

Joint Committee for IT

Sub committee (SC) 7

Standardization of 

processes, supporting 

tools and supporting 

technologies for the 

Sub committee (SC) 7

technologies for the

engineering of software 

products and systems.

Mandated to develop

Working Group (WG) 24

Mandated to develop 

Standards and Guides for 

Very Small Entities 

*

6

(7)

ISO/IEC 29110 Standards and Guides

for Very Small Entities (VSEs)

for Very Small Entities (VSEs) 

Entry - Targets VSEs typically

d

l i

6

th

developing

6 person-month

projects and

start-up VSEs

;

Basic - Targets VSEs developing

only

one project at a time

;

Intermediate – Targets VSEs

developing

multiple projects

Advanced

Intermediate

p g

p p j

within the organizational context;

Advanced – Targets VSEs which

want to

sustain and grow

as an

Entry

Basic

want to

sustain and grow

as an

independent competitive software

development business.

(ISO/IEC 29110)

Very Small Entities 

are enterprises, projects or departments having up to 25 people.

(8)

Ingeniería para la Industria

Content

Introduction

Introduction

Business Rationale for Software Quality Assurance 

and the Measurement of the Cost of Quality

and the Measurement of the Cost of Quality

Undergraduate/Graduate Software Quality 

Assurance Courses and the Measurement of the 

Cost of Quality

Measurement of the Cost of Software Quality – A 

y

Case Study

Conclusion

8

(9)

Software Development – What the customer wants

9

(10)

Why Software Fails ?

Why Software Fails ?

‘Studies have shown that software specialists spend about 40%

to 50% of their time on avoidable rework rather than on what

to 50% of their time on avoidable rework rather than on what

they call value-added work, which is basically work that’s done

right the first time.

Once a piece of software makes it into the field, the cost of

fixing an error can be 100 times as high as it would have been

g

g

during the development stage.’

Software managers and software engineers ‘must’ select and use

appropriate practices to reduce rework (i.e. waste or ‘scrap’)

(11)

Components of Project Cost

Project Cost

Cost of Quality

Cost of Performance

Cost of Quality

Cost of Performance

• Generation of plans

• Software Development

Cost of Non

Conformance

Cost of Conformance

Fixing  defects

Re testing 

Re‐reviews

Appraisal Costs

• Reviews

• Inspections

Prevention Costs

• Training

• Methodologies

Updating source code

Modifying documents

Inspections

• Testing

• IV&V

• Audits

• Methodologies

• Tools

• Data gathering

11

• Audits

(12)

Cost of Quality

• Data from Professional Software Engineers

Site A American Site A American Site B European Site C European Site D European Course A 2008 Course B 2008 Course C 2009 Course D 2010 Course E 2011 Course F 2012 American Engineers (19) American Managers (5) European Engineers (13) European Engineers (14) European Engineers (9) 2008 (8) 2008 (14) 2009 (11) 2010 (8) 2011 (15) 2012 (10) Cost of performance

41%

44%

34%

31%

34%

29%

43%

45%

45%

34%

40%

Cost of rework

30%

26%

23%

41%

34%

28%

29%

30%

25%

32%

31%

Cost of appraisal pp

18%

14%

32%

21%

26%

24%

18%

14%

20%

27%

20%

Cost of prevention

11%

16%

11%

8%

7%

14%

10%

11%

10%

8%

9%

Quality

71

8

23

35

17

403

19

48

35

60

55

Quality = Number of Defects/ 1,000 Lines of Code

(13)

Defect ‘Injection’ During Development

D f

t (%)

Defects (%)

W

id

if d f

ibl

We must identify defects as soon as possible

to reduce cost and schedule slippage.

Development Phases

(14)

Defect Detection During Development

Defects Detected/

Defects Detected/

Defects Injected

We must implement software engineering practices

We must implement software engineering practices

to detect and correct 90% of defects as early as

possible to reduce cost and schedule slippage

Adapted from (Selby,2007)

Development Phases

14

(15)

Start of Initiative

% f T

l

Cost of Quality

40 45

Cost of Non Conformance (Rework)

Start of Initiative

41%

% of Total

Project Cost

30 35

Appraisal & Prevention Costs

15 20 25

18%

5 10 15

11%

0

M t it L

l

1

2

3

4

1988

1990

6%

5%

Maturity Level

1990

1992

1996

1994

15

(16)

Ingeniería para la Industria

Content

Introduction

Introduction

Business Rationale for Software Quality Assurance 

and the Measurement of the Cost of Quality

and the Measurement of the Cost of Quality

Undergraduate/Graduate Software Quality 

Assurance Course and the Measurement of the 

Cost of Quality

Measurement of the Cost of Software Quality – A 

y

Case Study

Conclusion

16

(17)

Software Quality Assurance Course at ETS

• Fourteen Topics

p

Chapter 1: Basic Software Quality Assurance Knowledge

Chapter 2: Quality Culture

and Cost of Quality (COQ)

Ch t 3 Q lit R

i

t

French Textbooks

Chapter 3: Quality Requirements

Chapter 4: Software Engineering Standards and Models

Chapter 5: Reviews

and COQ

Vol. 1

Chapter 6: Software Audit

Chapter 7: Verification and Validation

and COQ

Chapter 8: Tests and Software Quality Assurance

English Textbook

p

Q

y

Chapter 9: Software Configuration Management

and COQ

Chapter 10: Policies, Processes and Procedures

and COQ

Chapter 11: Measurement

and COQ

Vol. 2

Chapter 11: Measurement

and COQ

Chapter 12: Management of Suppliers and Contracts

Chapter 13: Risk Management

and COQ

Chapter 14: Software Quality Assurance Plan

17

(18)

Software Quality Assurance Course

Laboratory

• Software Development Project with ISO/IEC 29110

Put in practice the software quality practices learned in class,

y

Each team manages and executes a project

using

ISO/IEC 29110

A process is a component of the COQ (i.e. Prevention)

Statement

S ft

Customer

Implementation

Process

Initiation

Statement 

of Work

Software

Configuration 

Planning

Control

Project Management Process

Analysis

Design

Measure the

Cost of

Software

Organizational Management

Planning

Control

Execution

Closure

Construction

Tests

Delivery

Quality

18

Organizational Management

Delivery

(19)

Graduate 

Software Engineering Program

Twelve 3-credit courses and a 9-credit Project in Software

engineering

Software Engineering Program

engineering

Students are typically from industry with a minimum of 2 years

of experience

of experience

Students typically have an undergraduate degree in either computer

science or management information systems and work for small and

di

i d

i ti

medium-sized organizations.

• Software Quality Assurance Course

Students in teams of 3 have to do a Project in an

Students, in teams of 3, have to do a Project in an

Organization

Measurement of the Cost of Quality

Q

y

(20)

C

t

t

Ingeniería para la Industria

Content

Introduction

Business Rationale for Software Quality Assurance 

and the Measurement of the Cost of Qualityy

Undergraduate/Graduate Software Quality 

Assurance Courses and the Measurement of the 

Cost of Quality

Measurement of the Cost of Software Quality – A 

Case Study

Conclusion

20

(21)

M

i

th C t f S ft

Q

lit

Measuring the Cost of Software Quality 

of a Large Software Project 

at Bombardier Transportation in Canada

Laporte C Y Berrhouma N

Laporte, C. Y., Berrhouma, N., 

Doucet, M., Palza‐Vargas, E. 

(Laporte et al. 2012)

21

(22)

Challenges Facing Railway Manufacturers

Criticality of software

ƒ Better, Faster, Cheaper

y

Financially, environmentally or for

human safety.

Multi-disciplinary system development,

Integrator-Suppliers Relationships,

Multi-country development,

Multi-cultural teams

Multi cultural teams,

Downsizing/Merger/Turnover,

Off shoring.

ERTMS / ETCS (European Rail Traffic 

/

(

p

Management System / European Train 

Control System) 

22

(23)

Overview of Bombardier

Workforce of some 80,000 people in over 24 countries.

Bombardier Aerospace

W ld l d i h

f

f b i

j

d

i

l

World leader in the manufacture of business jets and regional transport,

Bombardier Transportation

Leader in the manufacture of rail transport equipment.,

p

q p

Manufactures locomotives, freight cars, and propulsion and control

systems,

Provides systems and signalling equipment

Provides systems and signalling equipment.

Modern trains and subways are increasingly complex, and more

and more subsystems are computer-controlled, such as

propulsion and braking systems.

At the time of this case study, there were over 30 software

development centers within Bombardier Transportation for a

development centers within Bombardier Transportation, for a

total of about 950 software engineers.

(24)

The Software Development Group 

f B

b di

T

t ti

i Q éb

(C

d )

of Bombardier Transportation in Québec (Canada)

A group of 30 software engineers whose role is to design, develop,

and maintain embedded software for trains and subways

and maintain embedded software for trains and subways

Software monitoring system used for collecting software

maintenance information,

Software for controlling car inclination.

SD Media Converter LINKPWRLINK 5VDC. 1A _ __ __ + UP LINK RX TX

(Laporte et al. 2012)

24

(25)

Software Engineering Process

of the Software Engineering Group *

g

g

p

25

(26)

ISO/IEC 15939 ‐ Systems and Software 

Engineering Measurement Process Standard

Engineering ‐ Measurement Process Standard

Technical &

Management

Measurement Requirements

User Feedback

g

Processes

Information

Products

Information

Needs

Core Measurement Process

Establish

Plan

Perform

Evaluate

Commitment

Measurement

E

i

B

Plan

Performance

Information

Measurement

Measurement

Experience Base

Improvement Actions

Evaluation

Results

(ISO/IEC 15939)

Scope of ISO/IEC 15939

26

(27)

Objectives of the Measurement Project

f h S f

D

l

G

of the Software Development Group

1. Identify a project to measure the Cost of Software Quality

y p j

Q

y

(CoSQ),

2. Collect data on costs,

3 Categorize costs related to software quality

3. Categorize costs related to software quality,

4. Develop a data model for the measurement of CoSQ,

5. Analyze data collected, on the selected project, using the

y

,

p j

,

g

data model,

6. Present the CoSQ report to senior management,

7 Expand the measurement of the CoSQ to other software

7. Expand the measurement of the CoSQ to other software

projects in the Software Development Group.

(28)

Overview of the Project Selected

Overview of the Project Selected

Development of the software to control the subway of a

large American city,

Team of 15 software engineers,

L

b

f

ft

t k

Large number of software tasks

1,121 tasks

Total effort of the completed project

Total effort of the completed project

88,598 hours

(29)

The Five Stages of the Measurement Project

/

using the ISO/IEC 15939 Standard

1.

Identification of the project tasks related to the CoSQ

Identify measurement requirements and resources

2.

Development of a list of typical tasks related to the CoSQ

Define data measures

Define data measures

3.

Categorization of the tasks related to the CoSQ

Select and plan measures

4.

Development and application of weighting rules

Define measurement criteria

5

D t

i ti

f th

fid

f th

i hti

l

5.

Determination of the confidence of the weighting rules

Define data collection and analysis

29

(30)

Stage 1. Identification of Project Tasks 

R l

d

h C SQ

Related to the CoSQ

Software engineering process is composed of 4 ‘types’

Lif C l

Life Cycle,

Process,

Type

Process Element

- Sub-process,

- Activity.

yp

Life Cycle

Primary Life Cycle Process

Process

Development

Sub-process

Code and Debug

Activity

Unit Tests

Representation of BSEP elements

WBS Element

Task Name

Effort [hours]

Code document

Monitoring-Unit Testing

91.1

(Laporte et al. 2012)

Sample registration of a task in the Project accounting system

(31)

Stage 2. Development of a list of tasks 

l

d

h C

f S f

Q

li

related to the Cost of Software Quality

A list of tasks related to the CoSQ is developed for this project

A list of tasks related to the CoSQ is developed for this project.

Primary Life Cycle Processes

Supply

Organizational Life Cycle Processes

Management

pp y

Development

System Requirements Analysis

System Architectural Design

S ft

R

i

t A l i

g

Estimate the Project

Plan the Project

Manage Risks

T

k th P j t

Software Requirements Analysis

Software Architectural Design

Software Detailed Design

Software Coding and Testing

Track the Project

Infrastructure

Plan Infrastructure Activities

Establish the Project Infrastructure

Subset of CoSQ-related Tasks

Software Integration

Software Validation Testing

Establish the Global Infrastructure

Maintain the Infrastructure

(Laporte et al. 2012)

Q

(32)

Stage 3. Categorization of tasks 

l

d

h C SQ

related to the CoSQ

All 1,121 tasks were sorted as follows:

Implementation (I),

Evaluation (E),

Prevention (P)

Prevention (P),

Rework (R) (internal and external anomalies).

Task

Task

Identification

WBS Element

Task Name

I

E

P

R

2410

Code

Trace Requirements -

-

x

-Example of classification of the Requirements Traceability Task

(33)

Stage 4. Development of weighting rules

Many tasks belong to more than one CoSQ category

e.g. The ‘Test and coding’ task overlaps the Evaluation and

Implementation CoSQ categories.

p

Q

g

Twenty-seven (27) weighting rules have been defined.

Rule

Number

Name of Rule

Typical Tasks

Implementation Evaluation

Prevention Rework

Number

3

Process Audit

Software Project &

Process Audit

100%

7

Review

Design review

100%

7

ev ew

es g ev ew

00%

12

Problem

Correction

and Coding

Corrections,

Debugging and Final

Coding

30%

70%

17

Training

Training of new

resources

100%

26

Follow-up &

Follow-up &

Validation (all

85%

15%

26

Validation

Validation (all

releases)

85%

15%

(34)

Stage 5. Determining the Confidence 

f W i h i

R l

of Weighting Rules

What is the level of confidence of the CoSQ measures ?

Another component was added to the rules

"H" for high precision,

"M" for medium precision and

M for medium precision, and

"L" for low precision.

C

fid

ith th W i hti

R l

P

t

(%)

Confidence with the Weighting Rules

Percentage (%)

H

High

88%

M

Medium

11%

L

Low

0,2%

Total number of activities

1,121

(35)

Cost of Software Quality Measurement Results 

Perform

Evaluation Prevention

Rework

CoSQ

Q

Hours

59,231

18,551

2,025

8,791

28,824

P

t

67 %

17 %

4 %

11 %

32 %

Percentage

67 %

17 %

4 %

11 %

32 %

Total effort (i e cost) of the project: 88 598 ho rs

Total effort (i.e. cost) of the project: 88,598 hours

Number of software tasks: 1,121 tasks

(36)

Cost of Software Quality at Raytheon 

compared to Bombardier Transportation

p

p

70

CMM level 3

Start of intiative

CMM level 1

50

60

c

t cost

TCoSQ

Rework

Total Cost of Software Quality (40%)

32 %

40

of

tota

l pro

je

c

Cost of

Conformance

Percentage of Rework ( ≈13%)

Percentage of Evaluation ( ≈20%)

17 %

11 %

20

30

P

ercentage

o

Conformance

Percentage of Rework ( ≈13%)

Percentage of 

P

ti

( 7%)

4 %

11 %

0

10

P

Prevention

Rework

Appraisal

Prevention ( ≈7%)

4 %

0

Year

87

88

89

90

91

92

93

94

95

96

36

(37)

Cost of Quality and Maturity Levels

A study of the Software Engineering Institute (SEI) showed

that rework varies between 15% and 25% of the cost of

CMM M t

it L

l

P

t

f R

k

that rework varies between 15% and 25% of the cost of

developing a CMM Maturity Level of 3.

CMM Maturity Level

Percentage of Rework

1

≥ 50 %

2

25 % to 50 %

3

15 % to 25 %

4

5 % to 15 %

5

≤ 5 %

11 %

Relationship between CMM Maturity

Levels and percentage of rework

(38)

Recommandations

1. Continue data collection and CoSQ Measurement

Continue the data collection using procedures and methods (for

example use of tools database structures) within the SDG

example use of tools, database structures) within the SDG.

2. Control activities related to the correction of problems

Most defects are found mainly during the execution of software

i

f

d i

d

di

k

requirements, software design and coding tasks

Reduce the defects by increasing prevention effort

e.g. Execute more peer reviews (walk-through, inspection)

3. Present the CoSQ measurement results to Management

To enable them to develop better budgets for business process

improvement

improvement.

Establish a scoreboard to display the CoSQ and the results of

process improvement activities to illustrate the relationships

between them

between them.

38

(39)

Recommandations

4. Measure the CoSQ at other Bombardier Transportation sites

To identify opportunities for reducing costs,

T

id

b i

b d

d

l

d

li

i i i

To provide a basis to budget development and quality activities,

To use the results of CoSQ measures to improve processes.

O

I

Tasks

Outputs

Inputs

Project WBS CoSQ Report 1. Identify CoSQ tasks, using  Project Plan and WBS  2. Record CoSQ effort by  category 3. Apply weighting rules  CoSQ Measurement Process Typical CoSQ list of tasks Weighting rules Findings Recommendations pp y g g 4. Compute the CoSQ 5. Develop findings and  recommendations 6. Review findings and  recommendations with  management

Entry Criteria

Exit Criteria

management 7. Produce final CoSQ Project  Report 8. Archive Report

39

(Laporte et al. 2012)

Entry Criteria

Measures

Exit Criteria

Project Plan Approved

CoSQ Project Report Approved

Effort (staff‐hours) 

(40)

C

t

t

Ingeniería para la Industria

Content

Introduction

Introduction

Business Rationale for Software Quality Assurance 

and the Measurement of the Cost of Quality

and the Measurement of the Cost of Quality

Undergraduate/Graduate Software Quality 

Assurance Courses and the Measurement of the 

Cost of Quality

Measurement of the Cost of Software Quality – A 

y

Case Study

Conclusion

40

(41)

Conclusion

Conclusion

The concept of Cost of Software Quality is a powerful

tool to help quantify the rework indentify weaknesses

tool to help quantify the rework, indentify weaknesses

and prevention activities,

The Software Quality Assurance courses we teach help

Q

y

p

software engineers in understanding the value of the

quality assurance practices learned in class and put into

ti

i th l b

t i

practice in the laboratories,

The Case study at Bombardier Transportation

demonstrated the business value of the measurement of

demonstrated the business value of the measurement of

the Cost of Software Quality.

Best Approach Better

Faster

Cheaper

41

(42)

Ingeniería para la Industria

Th

k

f

i

Thank you for your attention

(43)

Contact Information

Claude Y Laporte

Voice: + 1 514 396 8956

Voice: + 1 514 396 8956

E-Mail:

Claude.Y.Laporte@etsmtl.ca

Web:

http://profs.etsmtl.ca/claporte/English/index.html

Public site of WG 24

l

k

i

i l

d

Free access to Deployment Packages, presentation material and

articles:

http://profs.logti.etsmtl.ca/claporte/English/VSE/index.html

p p

g

p

g

(44)

References

• Fanmuy, G., L’Ingénierie et Management des Systèmes pour les PME/TPE et petits projets, Association Française d'Ingénierie Système (AFIS)/International Council on Systems Engineering (INCOSE) May 24th 2011 Paris France

(AFIS)/International Council on Systems Engineering (INCOSE), May 24th, 2011, Paris, France.

• Gibson, D.L., D.R. Goldenson, and K. Kost, Performance Results of CMMI®-Based Process Improvement. Technical Report, CMU/SEI-2006-TR-004, ESC-TR-2006-004, 2006

• Haley, T., Ireland, B., Wojtaszek, E., Nash, D., Dion, R., Raytheon Electronic Systems Experience in Software Process Improvement, Technical Report CMU/SEI-95-TR-017, Software Engineering Institute, November 1995.

• Iberle K., « But Will It Work For Me », Proceedings of the Pacific Northwest Software Quality Conference, p. 377-398, Portland, United States , , g Q y , p , , 2002,

• Iberle K., « They Don’t Care About Quality », Proceedings of STAR East, Orlando, United States, 2003.

• ISO/IEC JTC1/SC7 N3288, New Work Item Proposal – Software Life Cycles for Very Small Enterprises, Mai 2005.

• ISO/IEC 12207:2008, Information technology – Software life cycle processes, International Organization for Standardization/ International Electrotechnical Commission: Geneva, Switzerland.

• ISO/IEC 29110 - Lifecycle Profiles for Very Small Entities (VSEs) – Part 1: Overview. International Organization for Standardization/International Electrotechnical Commission: Geneva, Switzerland.

• ISO/IEC 15289:2006 - Systems and software engineering - Content of systems and software life cycle process information products (Documentation)

• Kabli, S., Conception, réalisation et mise a l’essai de trousses de déploiement pour faciliter et accélérer l’implémentation de la norme ISO/CEI 20000 l è i ÉTS 2009

20000 par les très petites structures, ÉTS, 2009.

• Konrad, M., Overview of CMMI Model, Presentation to the Montréal SPIN, November 21, 2000, Montréal, Canada.

• Kroll, P.; Kruchten, P.; The Rational Unified Process Made Easy – A Practionner’s Guide to the RUP.; Addison-Wesley, 2003

• Laporte, Claude Y., Berrhouma, Nabil, Doucet, Mikel, Palza-Vargas, Edgardo, "Measuring the Cost of Software Quality of a Large Software Project at Bombardier Transportation", Software Quality Professional Journal, ASQ, Vol. 14, number 3, June 2012, p 14-31

L t Cl d Y F G thi Pt k K Th D l t f S t E i i I t ti l St d d d S t T l f V • Laporte, Claude, Y, Fanmuy, Gauthier, Ptack, Ken, The Development of Systems Engineering International Standards and Support Tools for Very

Small Enterprises, 22nd Annual International Symposium of the International Council on Systems Engineering, Rome, July 9-12, 2012. • Laporte, Claude Y., Berrhouma, Nabil, Doucet, Mikel, Palza-Vargas, Edgardo, "Measuring the Cost of Software Quality of a Large Software

Project at Bombardier Transportation", Software Quality Professional Journal, ASQ, Vol. 14, number 3, June 2012, p 14-31.

• Laporte, C.Y., Alexandre, S., O’Connor, R., A Software Engineering Lifecycle Standard for Very Small Enterprises, in R.V. O’Connor et al. (Eds.): EuroSPI 2008, CCIS 16, pp. 129–141.

(Eds.): EuroSPI 2008, CCIS 16, pp. 129 141.

• Long, L., The Critical Need for Software Engineering Education, Crosstalk - The Journal of Defense Software Engineering, January 2008, pp 6-10.

• Reifer D., « Let the Numbers Do the Talking », Crosstalk – The Journal of Defense Software Engineering, march 2002.

Figure

Updating...

References