Benchmarking Experiences and Guidelines for Improvement

Christof Ebert, Vector Consulting Services

Benchmarking



… supports clients worldwide in

improving their product

development and IT and with

interim management



… with clients such as

Accenture, Audi, BMW, Bosch,

Daimler, Ford, Huawei, Hyundai,

IBM, Lufthansa, Munich RE,

Porsche, Siemens, Thales,

Toyota and ZF



… offers with the Vector Group a

portfolio of tools, software

components and services



… is as Vector Group globally

present with 1500 employees

and well over 300 Mio. € sales



www.vector.com/consulting

Vector Consulting Services

Welcome

Railway

IT & Finance

Automotive

Aerospace

Industry

Medical

Welcome



Performance Improvement

Benchmarking

Case Study

Summary

Agenda

Product Development and the Business Impact

Performance Improvement

Department

Ping Pong

Added value

30%

Overheads

40%

Pure waste

30%

Sources:

Vector

Benchmarking

Suite,

Optimizing Value – Balancing “Good Enough”

Performance Improvement

Value

Effort

Target

Realistic

Good enough

Insufficient

quality

Technical

debt

Over-engineering

Welcome

Performance Improvement



Benchmarking

Case Study

Summary

Agenda

Benchmarking is the

structured and systematic learning

from the best in class.

Benchmarking of IT and Software

applies to different dimensions



Products (e.g., quality

requirements, product complexity,

variance, platforms, technical

debt, architecture)



Processes (e.g., roles, methods,

tools, value stream, productivity)



Projects (e.g., estimation, supplier

management, interfaces)



People (e.g., competences,

distributed teams, organization)

What is Benchmarking?

Benchmarking

Productivity

Time

Improvement

needs

Sustainable

Improvements

Learning,

Transitition



Step 1: Understand needs and set goals



Which strengths and gaps need to be addressed?



Which products, services and practices need to be improved?



What results are expected from a benchmark?



Step 2: Measure against the best in class



Which performance indicators should be measured?



What reference data is available?



Step 3: Implement sustainable improvements



Which concrete actions will translate benchmarking results to benefits?



How to avoid that benchmarking is a short-term hype?

Benchmarking Consists of Three Steps

Benchmarking

Benchmarking means measurement and improvement.

If Benchmarking is misunderstood as mere data collection it will not

yield any value.

Business Performance and Improvement

Benchmarking

Business

Performance

Business

Pain Points

Marketing, Ops,

Product Mgmt

(Customer

Satis-faction, Prizing,

Service, Revenues,

Organization,

Corporate Culture)

Sales, Marketing,

Finance …

IT/Software Impact

Pain Points

Performance

Change Program

Continuous Improvement

Vector Benchmarking Suite



Measurements

(Gap analysis, Engineering and IT performance,

contribution to business, improvement progress)



Industry benchmarks

(Product, Defects, Rework, Complexity,

Locations, Competences, People management,

Processes)



Productivity levers: Value orientation, Waste

reduction, People focus

Vector Benchmarking Suite

Benchmarking

Organization rating and

improvement suggestions

•

…

•

…

Strengths and weaknesses

from benchmarks

•

…

•

…

Improvement

Potentials

Performance

Rating

Benchmarks

Vector

DB

Client

Data

Quantitative

Qualitative

Practices

Data

Data and

M

eta-Data

The role for measurement



80% of all life-cycle costs are (pre-)

defined in development, e.g., rework

or variation



75% of companies don’t reliably

measure their cost drivers



An average project accumulates

43% of unplanned costs before

finished or aborted

Benchmarking needs Measurement

Benchmarking

Our recommendations:



Set up mandatory internal standard measurements



Do not just collect data, but analyze and use the data, such as trends



Compare apples and apples



Provide ways to learn and to improve from benchmarks



Evaluate contribution and pinpoint to external effects, e.g. changes, complexity

Dvmt cost

Cost of

sales

Others

Waste:

defects,

rework

etc.

Value:

Regular

project

cost

Focus

Product cost

Dvmt cost

Business

impact

Welcome

Performance Improvement

Benchmarking



Case Study

Summary

Agenda

Vector Case Study: Qualitative Analysis

Case Study

Focus on

value

Continuously

improve

Empower

people

Eliminate

waste

Optimize

value streams



Agile development



Requirements

valuation (e.g.,

Kano)



Earned value

management



Variant reduction



Review culture



Test-oriented

requirements

engineering



Defect root cause

analysis



Reduce and

manage

require-ment changes



Early defect

detection



Criticality analysis



Train people on key

competences, e.g.

project managers,

product managers



Core teams with

business ownership



Empowered feature

teams



Less site

distribution of

projects



Vector 3-S

approach for each

improvement:

Set-up, Success,

Sustain



Client practices and

data are

continuously

benchmarked



Reduce interface frictions



Combined design / test teams



Higher process maturity

Case Study

Focus

Always

7%

Often

13%

Sometimes

16%

Rarely

19%

Never

45%

Vector Case Study: Set a Clear Focus

Case Study

Feature Usage

Sources:

Vector Benchmarking Suite,

Microsoft, Daimler, BITKOM 2012,

Ebert 2016

Our recommendations:



Set a clear goal: Reduce waste



Analyze waste in work flows

such as requirements value,

requirements changes,

duration of open defects,

defect detection effectiveness.



Learn from industry best

practices and extract concrete

prioritized actions



Project: Use quality gates as

defined thresholds and to

ensure transparent

governance.



Product: Reduce technical debt

such as insufficient

architecture with concrete

refactoring actions.

Case Study

Focus

Vector Case Study: Quantitative Analysis

Case Study

Producti

vi

ty

# Tasks allocated per person

Optimum:

2-3 tasks or

projects

Producti

vi

ty

# Sites per project

Optimum:

2-3 sites

Producti

vi

ty

Project complexity

Optimum:

< 1year duration

< 10 PY effort

< 1000 FP size

Producti

vi

ty

Effort for requirements

and architecture

Optimum:

10-15%

Producti

vi

ty

# Requirements changes

Optimum:

<40%

Producti

vi

ty

# Project delays

at handover

Optimum:

5-10%

Sources:

Vector

Benchmarking

Suite,

Ebert 2016

Focus

Vector Case Study: Implement the Change Project

Case Study

From Classic Development

Client Benefits



25% reduction of lead time and

effort before project start



Reduction of hand-offs and

ping-pong between project functions



_{20% reduction of project lead time}

by concurrent engineering

To Lean Agile

Vector Contribution



Streamlined development

processes and interfaces



Value-orientation end-to-end



Product line engineering from

requirements onwards



Efficient yet effective safety

Welcome

Performance Improvement

Benchmarking

Case Study



Summary

Agenda

Two thirds of productivity

improvement projects fail



Productivity improvement is a

substantial culture change



Change is often underestimated

and handled ad-hoc

Reasons



No management leadership



Insufficient communication



Lack of internal know-how

Performance Improvement Is Not Easy

Summary

Sources: Vector Consulting Services 2016

48%

Internal

resistance

Insufficient know-how

Pressure

without

ownership

Other

reasons

22%

17%

13%



Consider business impacts



Start with strengths and weaknesses



Set improvement targets

in line with your business needs



Start benchmarking top down

to achieve sustainable results



Learn from best in class



Consider value, waste, people

– and your ability to improve



Don’t just collect and compare measurements



Use benchmarking to grow



Continuously challenge your performance

Recommendations

Summary

Benchmarking must be supported by experts

Concrete Benchmarks and Measurements

Summary

Software Measurement

Establish, Extract, Evaluate,

Execute

Christof Ebert and Reiner Dumke

Third edition, Springer, 2007

www.vector.com/books

“Few organizations have really

institutionalized measurement of

their products and processes.

This book is bang up-to-date in

both fields and packed with

practical advice. For every

software engineer."

Charles R. Symons, Inventor of

Function Points

Optimizing Global Projects and Teams

Summary

Global Software and IT

Christof Ebert

2. extended Edition, Wiley, 2011

Discount:

http://bit.ly/cSjZgD

"This book stands out as the best

source of information on

distributed software

development. Seldom do we see

a book with the concepts

completely backed by industry

experiences and views. Software

developers and managers benefit

from the broad case studies.“

S M Balasubramanian, Vice

President, Wipro Technologies

Vector Consulting Services

For more information about Vector Consulting

Services please visit

www.vector.com/consulting

Your Partner in Achieving Engineering Excellence

Phone +49 711 80670-0

www.vector.com/consulting

Fax +49 711 80670-444

[email protected]