Measures to get the best performance from your software suppliers

‘Measures to get the best performance

from your software suppliers’

Charles Symons

Founder & Past President, COSMIC

2

COSMIC

COSMIC is a not-for-profit organization, founded in 1998 by

an international group of software metrics experts. We

developed an ISO standard method of measuring software

size based on fundamental software engineering principles.

The method is now used world-wide for measuring

business, real-time and infrastructure software for use in

software project performance measurement, benchmarking

and estimating.

Aims of this talk

• What is publicly known about software

industry performance?

• Why do we get this performance?

• How might customers use software

contracts and metrics to get better

performance?

First, what is the

‘Software industry’?

• Very difficult to define – a few % of world GDP

• I have no data from organizations that do not

usually publish their performance, e.g.

– Major hardware and packaged software suppliers

(MS, Oracle, Google, SAP, IBM, Apple, etc)

– Much real-time software (defence & transportation

systems, appliances, etc.) and for games

• I include data from publicly-available sources:

– Business & Government application software

– Some embedded software (telecoms, etc.)

4

Some other constraints

• No distinction of in-house versus

outsourced software suppliers

• No exemplars – our focus is

industry-average performance

‘Performance’ is a matter of

balance

6

Delivery to budget & time:

• Actual vs. Estimated Cost

• Actual vs. Estimated Duration

Project productivity

• Size / Effort

Project speed

• Size / Duration

Product quality

• Post-delivery defects /

Size

Delivery to time & budget is

notoriously poor

The cost of these over-runs

is a scandal

8

Annual cost of failures and over-runs:

• US market (Standish) ~100 Billion US$

• European market

~100 Billion €

The ‘world-class’ software suppliers’

profit margins on the UK contracts

: 10 – 20+ %

© Charles Symons 2012

Study

No. of

Cost Over-runs/

Country Projects

Write-offs

UK Public Sect.

105

£ 29B £ 9B (31%)

Measuring industry work product

(size) is critical, but primitive

NFR

FPA

‘

Traditional FPA’:

IFPUG

,

NESMA

,

(

MkII,

FiSMA

)

SLOC

FSM

OP

Productivity shows no sign of

improving

10

‘US software industry ‘productivity’ has declined over the

last 15 years’ (Business Week, 2004

)

Comments on this finding:

• ‘Productivity’ = industry value-added output / resources used

• ‘The complexity of software projects has increased’

Study of ISBSG

average

productivity

1995 – 2005

(3433 projects)

More recent ISBSG data

suggests productivity is falling

Looking more closely, the

project mix has changed

significantly

12

When roughly corrected for

the change in mix and

economy of scale, productivity

has not changed much over

the 20 years

But these projects are being

completed faster

New development projects

Enhancement projects

When roughly

corrected for

the lower

software sizes

delivered

Software quality is often

amazing!

• The world’s critical transportation, financial

and communications systems are largely

free of defects

• Less critical systems are less impressive

14

Maybe quality

is improving?

-

or maybe this is

also an effect of an

increasing %age of

enhancements?

Summary: industry performance

is uneven (and poorly measured)

Delivery to budget & time

• Actual vs. Estimated Cost

• Actual vs. Estimated Duration

Project productivity

• Size / Effort

Project speed

• Size / Duration

Product quality

• Post-delivery defects /

Size

Poor, especially

for mega-projects

No real change?

Maybe improving

(smaller, more

heavily-staffed projects)

Meanwhile: most software

suppliers make nice profits

• It’s very rare for a major software

supplier to go bankrupt

• But in-house suppliers who are believed

to under-perform tend to get outsourced

16

Agenda

• What is publicly known about software

industry performance?

• Why do we get this performance?

• How might customers use software

contracts and metrics to get better

performance?

Some of the main reasons are

obvious

(High) Quality

• No choice (for mission-critical &

customer-facing systems)

(Poor) Delivery to time & budget

• Changing requirements

• Poor estimating & project management

... in spite of >20 years of process

improvement !

18

Traditional software contracting

is ‘flaky’, especially early in a

project

Customer

produces outline

requirements &

asks for estimates

Supplier measures

requirements

using a simple

sizing method

Supplier estimates

cost & time from

past experience and

... and goes from bad to worse

20 20

Customer commits

and then has to

add & change

requirements

Supplier now has

excuse to try to

re-negotiate cost &

time estimates

Supplier focuses on

meeting finally-

agreed requirements

to ensure payment

(or) De-scope

... so the final result?

Delivery to budget & time:

• Actual vs. Estimated Cost

• Actual vs. Estimated Duration

Project productivity

• Size / Effort

Project speed

• Size / Duration

Product quality

• Post-delivery defects / Size

Over-run, as usual,

for which the

customer pays

Customer has no idea

(and learns nothing)

Customer has no idea,

(and learns nothing)

OK. Any defects are

corrected during

Conclusion: the traditional

contracting process is

seriously flawed

• Customers reasonably want some idea of

total cost up-front, but an accurate

estimate is impossible on an outline SOR

• Commonly-used sizing methods are poor

• Estimating tools are only as good as their

input

• ‘Agile’ approaches help, but do not meet

all the customer’s needs

22

Agenda

• What is publicly known about software

industry performance?

• Why do we get this performance?

• How might customers use software

contracts and metrics to get better

performance?

Imagine buying vegetables in

this market

24

$5

$

8

$2

Customer: ‘I want some onions for a soup’

Vegetable seller 1: ‘Onions are $5 today’

Vegetable seller 2: ‘Onions are $7 today’

Vegetable seller 3: Onions are $4 today’ ...

Key elements for software

customers to re-balance risk

Contract on a price-per-unit size basis

(e.g. the ‘

Southern Scope

’ process),

using:

– appropriate software sizing method(s)

– ‘open’ project estimating methods

The key difference is to contract

with an agreed unit price

26 26

Customer

produces outline

requirements &

asks for estimates

Supplier measures

requirements

using

appropriate

sizing method

Supplier:

estimates

total size

(including

contingency)

x fixed unit price

= Total price,

and estimated duration

Supplier is selected based on unit price and other qualities

As requirements change,

the unit price stays fixed*

Customer commits

to the

unit price

and

then adds/changes

requirements*

Total cost varies

with size**.

Duration may need

to be re-estimated

Final total price =

Final size**

x unit price

* The unit price may increase for very late changes

** Sizes may be measured by an independent ‘quantity surveyor’

Agile

activities

can start

here

The process helps balance the

customer -supplier relationship

28

Customer

Takes risk on total

size of requirements

& changes

Supplier

Takes risk on unit

price, quality &

delivery

Shared

• Understanding of

sizes & estimates

• Knowledge of

productivity & speed

Better software sizing may also

lead to better performance!

COSMIC

A major global bank invested heavily in process

improvement:

Produc

tiv

ity

Time

Traditional FPA

A major European pension fund studied why a project,

estimated using traditional FPA, had seriously over-run its

budget. The software was re-measured using COSMIC.

80% of the over-run was accounted for by elementary

processes ≥ 40 CFP. (Traditional FPA max = 7 UFP)

COSMIC sizing & unit pricing fit

perfectly with Agile processes

• Replace ‘velocity’ in USP/work-hour by

COSMIC FP/work-hour $/CFP

• Ensure consistent sizing and unit pricing

at the User Story level and at all levels of

aggregation (iteration, release, etc) up to

the total software system size

For more details see Ref. 8

30

And the benefits of the

‘Southern Scope’ process?

Project Management $/FP

Av. Cost

Approach

Over-run

Traditional

1500

84%

Southern Scope

500

<10%

Australian experience

9

Similar results now reported by Ministry of

Justice in Finland

Renault: Southern Scope +

COSMIC = major benefits

Software embedded in automotive

Electronic Control Units

• Requirements stored in modelling tools

• COSMIC sizes measured automatically

• Past data used to predict software cost

and memory requirement

• Renault invites bids from software

suppliers knowing what to expect

32

Software metrics are widely used

by Central Governments

• Known use in Australia, Brazil, Finland, Italy, Korea,

Netherlands, UK, USA, and probably others

• In the UK, some of the largest departments use some

metrics:

• Work & Pensions (Social benefits)

• Revenue & Customs (Taxation)

• Environment, food and agriculture

• Defence ... and many smaller departments

Note: a UK Parliamentary Committee report in 2011 was titled:

“Government & IT – ‘a recipe for rip-offs’

– time for a new approach”

Food for thought:

Who are the most common types

of known COSMIC method

users?

1. Software Houses

2. Real-time software

developers

3. (etc)

Conclusion: Customer/Supplier

relationships must be

re-balanced

Suppliers deliver amazing software products

and enormous benefits, but do not deliver

reliably, or demonstrate good value for money

Customers must use advanced processes, &

measurement methods to get a better deal

The approach is powerful but not

easy, and needs patience

Requirements for success

• A good understanding of software size

measurement, its uses, and of estimating

• Good quality measurements of size, effort, and

project characteristics

• A lot of data needed to understand what drives

performance in your environment

• Strong commitment from senior management

There are no ‘silver bullets’, but the potential ROI

is huge

36

Thank you for your attention

Questions?

References

Background to this presentation: “Software industry performance: what you measure is what you get”, IEEE Software, November/December 2010

Information on the COSMIC method and its uses (e.g. by Renault) is available for free download from www.cosmicon.com

‘Southern SCOPE: avoiding software budget blowouts’, the Government of the State of Victoria, e-Government Resource Centre, www.egov.vic.gov.au.

1. Standish CHAOS Report, 2009, www.standishgroup.com/newsroom/chaos_2009.php 2. McManus, J. and Wood-Harper, T., “A Study in Project Failure”, www.bcs.org, June 2008 3. Whitfield, D., ‘Cost over-runs, delays and terminations: 105 outsourced public sector ICT

projects’, European Services Strategy Unit, Research Report No. 3, December 2007 4. “The SIP (Software Industry Performance) Report” and ISBSG benchmark data,

www.isbsg.org

5. ‘Why your IT project may be riskier than you think’ Bent Flyvbjerg, Alexander Budzier, Harvard Business Review, September 2011

6. ‘Industry outlook 2004’, Business Week, January 12, 2004

7. Jiang, Z., Naudé, P. and Comstock, C., ‘An investigation on the variation of software development: productivity’, International Journal of Computer & Information Science & Engineering, Vol. 1, No. 2, 2007

8. “Predictable pricing for agile development’, Grant Rule, UKSMA/COSMIC International Conference on Software Metrics & Estimating’, October 2011, www.uksma.co.uk

9. ‘Software development projects in government; performance, practices and predictions’,

ISBSG, 2005. www.isbsg.org ₃₈