Process Improvement. Objectives

cmsc435 - 1

Process Improvement

● To explain the principles of software process

improvement

● To explain how software process factors influence

software quality and productivity

● To introduce the SEI Capability Maturity Model

and to explain why it is influential. To discuss

the applicability of that model

● To explain why CMM-based improvement is not

universally applicable

cmsc435 - 3

● Understanding existing processes

● Introducing process changes to achieve organizational

objectives which are usually focused on quality

improvement, cost reduction and schedule acceleration

● Most process improvement work so far has

focused on defect reduction. This reflects the

increasing attention paid by industry to quality

● However, other process attributes can be the focus

of improvement

Process improvement

Why do process improvement?

● We want to build better products (cheaper,

more dependable, quicker, …)

● We really don’t know how to measure the

product characteristics

● There, measure the process under the

assumption – a better process will produce a

better product.

cmsc435 - 5

Process attributes

Process characteristic Description

Understandability To what extent is the process explicitly defined and ho w easy is it to understand the p rocess definition ?

Visibility Do the process activities culmin ate in clear results so that the p rogress of th e process is externally vis ible?

Supportability To what extent can the process activiti es be support ed by CASE tools? Acceptability Is the defined process acceptable to and u sable by the engineers

responsibl e for produ cing th e software produ ct?

Reliability Is the process designed in such a way th at process errors are avoided o r trapped before they result in produ ct errors?

Robustn ess Can the process continu e in spite of un expected prob lems?

Maintainability Can the process evolv e to reflect changing org anisational requirements or id entified process improv ements?

Rapidity How fast can the process of delivering a system from a given specification b e compl eted?

● Process analysis

Model and analyze (quantitatively if possible) existing processes

● Improvement identification

Identify quality, cost or schedule bottlenecks

● Process change introduction

Modify the process to remove identified bottlenecks

● Measure improvement

What happened? May need to tailor process.

● Process change training

Train staff involved in new process proposals

● Change tuning

cmsc435 - 7

● Process quality and product quality are closely

related

(Are they?)

● A good process is usually required to produce a

good product

● For manufactured goods, process is the

principal quality determinant

● For design-based activity, other factors are also

involved especially the capabilities of the designers

Process and product quality

Principal product quality factors

Product

quality

Development

technology

Cost, time and

schedule

Process

quality

People

quality

cmsc435 - 9

Quality factors

● For large projects with ‘average’ capabilities,

the development process determines product

quality

● For small projects, the capabilities of the

developers is the main determinant

● The development technology is particularly

significant for small projects

● In all cases, if an unrealistic schedule is

imposed then product quality will suffer

● Study an existing process to understand its

activities

● Produce an abstract model of the process. You

should normally represent this graphically.

Several different views (e.g. activities,

deliverables, etc.) may be required

● Analyze the model to discover process

problems. Involves discussing activities with

stakeholders

cmsc435 - 11

● Published process models and process

standards

It is always best to start process analysis with an

existing model. People then may extend and

change this.

● Questionnaires and interviews

Must be carefully designed. Participants may tell

you what they think you want to hear

● Ethnographic analysis

Involves assimilating process knowledge by

observation

Taxonomy of validation methods - Review

“experimentation” notes from several weeks ago

Process analysis techniques

The module testing activity

Test module Signed-off test record Module test data Module specification Module compiles without syntax errors

All defined tests run on module Test engineer Pre-condition Input Process Rôle Post-condition Outputs Responsible for

cmsc435 - 13

Activities in module testing

Prepare test data according to specification Read module

specification

Submit test data

for review Review test data

TEST DATA PREPARATION

Read and understand module interface Checkout module

from configuration management system

Prepare test harness for module

Compile test harness

MODULE TEST HARNESS PREPARATION

Incorporate module with test harness

Run approved tests on module

Record test results for regression tests

TEST EXECUTION

Write report on module testing including details of discovered problems Submit report for approval Submit test results to CM TEST REPORTING

©Ian Sommerville 1995 Software Engineering, 5th edition. Chapter 31. Slide ##

● Wherever possible, quantitative process data

should be collected

However, where organizations do not have clearly defined process standards this is very difficult as you don’t know what to measure. A process may have to be defined before any measurement is possible

● Process measurements should be used to

assess process improvements

But this does not mean that measurements should drive the improvements. The improvement driver should be

organizational objectives

cmsc435 - 15

● Time taken for process activities to be

completed

E.g. Calendar time or effort to complete an

activity or process

● Resources required for processes or activities

E.g. Total effort in person-days

● Number of occurrences of a particular event

E.g. Number of defects discovered

Classes of process measurement

● Goals

What is the organization trying to achieve? The

objective of process improvement is to satisfy

these goals

● Questions

Questions about areas of uncertainty related to

the goals. You need process knowledge to derive

these

● Metrics

Measurements to be collected to answer the

questions

cmsc435 - 17

THE MEASUREMENT INFRASTRUCTURE

Goal Based Measurement

Meas. Goal Meas. Goal Meas. Goal Meas. Goal Meas. Goal Meas. Goal Meas. Goal Meas. Goal

Question Question Question Question Question

Metric Metric Metric

• Each metric supports multiple goals

• Questions focus metric selection and in-process analysis

GOAL/QUESTION/METRIC PARADIGM

Goal and Model Based Measurement

● A Goal links two models: a model of the object of interest and a model of the focusand develops an integrated GQM model

● Goal: Analyze thefinal productto characterizeit with respect to the

various defect classesfrom the point of view of theorganization

● Question: What is the error distribution by phase of entry

● Metric: Number of Requirements Errors, Number of Design Errors, ...

Goal Goal Goal

Question

Question Question

cmsc435 - 19

DEFINING MEASUREMENT GOALS

A GOAL/QUESTION/METRIC EXAMPLE

Sources of Software Errors

Req. Hi Level Detailed Other Design Design Type of Error • Business Goal

- Understand problem areas in the software business

• A Measurement Goal

- Analyze the final product to characterize it with respect to the various defect classes from the point of view of the organization

• Question

- What is the error distribution by type of error?

• Metrics

- Number of Requirements Errors, Number of Design Errors, ...

Importance of baseline studies

*Data from 11 Flight Dynamics

COMPUTA-TIONAL 15% DATA 27% INTERFACE 22% LOGIC/ CONTROL 20% INITIALIZA-TION 16% TEST 30% DESIGN 23% CODE 21% OTHER 26% 85% code writing 15% code reading

NASA/SEL PROCESS BASELINE EXAMPLE

Effort Distribution* Classes of Errors*

Source Code Growth Rate*

Percent source code growth (LOC)

0 20 40 60 80 100

DESIGN CODE SYSTEM TEST

ACCEPTANCE TEST

cmsc435 - 21

Quality Improvement Paradigm

Developed by Basili as part of NASA/GSFC Software Engineering Laboratory (SEL) research

QIP – 6 step process:

● Characterize– Analyze and understand the environment. Need to understand current processes before improvement. Compare to CMM. Measurement crucial at beginning (not level 4) and each environment is unique.

● Set goals– Quantifiable goals must be based on discovered characteristics.

● Choose process– Based on characterization and goals, processes, methods, and tools have to be determined.

● Execute the process– Perform the processes constructing the required products.

● Analyze– At the end of each project, analyze the data gathered, determine problems, and make recommentations for improvement.

● Package– Structure knowledge and store it an experience base for future reuse.

Forms of Packaged Experience

Equationsdefining the relationship between variables, e.g. Effort = 1.48*KSLOC.98_,

Number of Runs = 108 + 150*KSLOC

Histograms or pie chartsof raw or analyzed data, e.g., Classes of Faults: 30% data, 24% interface, 16% control, 15% initialization, 15% computation

Effort Distribution: 23% design, 21% code, 30%test, 26% other

Graphsdefining ranges of “normal” e.g.,

Fault Slippage Rate: halve faults after each test phase (4, 2, 1, .5)

Specific lessons learned, e.g., an Ada design should use library units rather than a deeply nested structure;

minimize the use of tasking as its payoff is minimal in this environment;

size varies inversely with defect rate up to about 1KLOC per module

cmsc435 - 23

Quality Improvement Paradigm - 2

Choose processes, methods, techniques, and tools Set goals Characterize & understand Package & store experience Analyze Results Process Execution Analyze Results Provide Process with Feedback Corporate learning Project learning

● US Defense Dept. funded institute associated

with Carnegie Mellon

● Mission is to promote software technology

transfer particularly to defense contractors

● Maturity model proposed in 1987, refined in early

1990s.

Software Capability Evaluation (SCE) – a self assessment of an organization

Capability Maturity Model (CMM) – A mechanism to evaluate the “maturity” of a development group

Major developer Watts Humphrey
Soon merged into one process

● Work has been very influential in process

improvement

cmsc435 - 25

The SEI process maturity model

Level 3 Defined Level 2 Repeatable Level 1 Initial Level 4 Managed Level 5 Optimizing

CMM Overview

An evaluation is based upon a 5-level rating. Each level represents a more mature organization:

1. Initial– No predefined process. Development is ad hoc. No established rules. (Note: This does not mean bad practices, but usually means it can be greatly improved. There are good development practices followed by some level 1 organizations.) 2. Repeatable– Policies are in place to manage software development.

Each project follows some process that is generally similar to others. 3. Defined– Processes for software development fit into a corporate

standard for software development. Development processes are well documented.

4. Managed– Quantitative goals for products and processes are set. Software process attributes are quantifiable.

cmsc435 - 27

CMM Key Process Areas (KPAs)

CMM evaluations are based upon KPAs. 18 KPAs govern the CMM. In order to be at level N, all the KPAs up to that level must be satisfied.

Initial – None – This is the initial state.

Repeatable– Requirements management; Project planning; Project tracking; Subcontract management; Quality assurance; Software configuration management

Defined– Organization process focus; Organization process definition; Training program; Integrated software management; Software product engineering; Intergroup coordination; Peer reviews

Managed– Software quality management; Quantitative process management [Is it really level 4?]

Optimizing– Defect prevention; Technology change management; Process change management

Key process areas

Software quality management Quantitative process management

Peer reviews Intergroup coordination Software product engineering Integrated software management Training programme

Organization process definition Organization process focus

Software configuration management Software quality assurance Software subcontract management Software project tracking and oversight Software project planning

Requirements management Initial Repeatable Defined Managed Optimizing

• 18 KPAs

• 52 goals

• 316 key practices

• 500 pages of

documentation

cmsc435 - 29

For each KPA …

For each KPA, need to address commitment:

1. What is commitment to perform this KPA? What

policies and management leadership are behind this

KPA?

2. What is the ability to perform this KPA? What

mechanisms are in place?

3. What activities are performed? What are the

procedures in place to perform that KPA?

4. What is the measurement of the KPA and the analysis

of its adherence?

5. What verifies the implementation? How to ensure

compliance?

Required questions for level 2 of CMM

1.1.3 Does the Software Quality Assurance function have a management reporting channel separate from the software development project management?

1.1.6 Is there a software configuration control function for each project that involves software development?

2.1.3 Is a formal process used in the management review of each software development prior to making contractual commitments? 2.1.14 Is a formal procedure used to make estimates of software size? 2.1.15 Is a formal procedure used to produce software development

schedules?

2.1.16 Are formal procedures applied to estimating software development cost?

2.2.2 Are profiles of software size maintained for each software configuration item over time?

2.2.4 Are statistics on software code and test errors gathered? 2.4.1 Does senior management have a mechanism for the regular

review of the status of software development projects? 2.4.7 Do software development first-line managers sign off on their

cmsc435 - 31

SEI Process Improvement Cycle

Initialize

● Establish Sponsorship ● Create vision and strategy ● Establish improvement structure

For Each Maturity Level

● Characterize current practice in terms of key process areas ● Assessment recommendations

● Revise strategy (generate action plans and prioritize key process areas)

For Each key Process Area

● Establish process action teams

● Implement tactical plan, define processes, plan and execute pilot(s), plan and execute institutionalization

● Document and analyze lessons ● Revise organizational approach

Aggregate results from SEI benefits study

Category Range Median

Total yearly cost of software process improvement activities $49,000 to $1,202,000

$245,000

Years engaged in software process improvement 1 to 9 3.5

Cost of software process improvement per engineer $490 to $2004 $1375

Productivity gain per year 9% to 67% 35%

Early detection gain per year (faults discovered pre-test) 6% to 25% 22%

Yearly reduction in time to market 15% to 23% 19%

Yearly reduction in post-release fault reports 10% to 94% 39%

Business value of investment in software process improvement (value returned on each dollar invested)

cmsc435 - 33

What is a level 5 organization?

● It is an organization that can manipulate process to achieve

various product characteristics.

● This requires that we have a process and an organizational

structure to help us:

Understand our processes and products

Measure and model the project and the organization

Define and tailor process and product qualities explicitly

Understand the relationship between process and product

qualities

Feedback information for project control

Experiment with methods and techniques

Evaluate our successes and failures

Learn from our experiences

Package successful experiences

Reuse successful experiences

KPA issues

Each KPA provides needed features for good

development, but there are issues with CMM:

•

Companies often look only one level ahead and ignore

ultimate goal of level 5.

•

Management only. Role of technology mostly ignored

•

Expensive, especially for small companies

•

Process is “bottom up.” “One size fits all.” (Contrast

with QIP next.)

•

“Level 3” is often a “requirement” for a DoD contract

even though not fully demonstrated higher level is an

accurate predictor of better quality products.

cmsc435 - 35

● It focuses on project management rather than

product development.

● Management often more interested in number than in

improvement.

● It ignores the use of technologies such as rapid

prototyping.

● It ignores measurement until level 4.

● It does not incorporate risk analysis as a key

process area

● It does not define its domain of applicability.

● Is expensive to use. Too much overhead and reporting

for a small company.

● Not very “agile.” Hard to use if rapid turnaround

needed.

SEI model problems

CMM model value

●

_But

_{it does impose a structure}

Often any structure better than no structure.

● Some indications that levels 2 and 3 do help

● Not clear if levels 4 and 5 add much value

Some companies satisfied with a level 3 ranking.

Although level 5 is supposed to indicate agility in

making changes, some companies are afraid to

make a change in fear of losing their ranking.

cmsc435 - 37

ISO 9000 Certification

ISO standard 9001 (software component) for standard

quality of a product. (Note that “good” quality is not

required, only that quality can be accurately determined.

Concrete life preservers can be “ISO certified,” but not

be very useful.)

● Companies want ISO certification as a requirement to sell

in Europe.

● Based upon 20 clauses. “ISO certification” is comparable

to CMM level 2 or level 3, but processes are different.

ISO 9000 clauses

Management Design control Quality team Purchasing Contract review Process control

Training Service

Control of customer supplied

product

Product identification and traceability

Document and data control Inspection and test cases Control of nonconforming

products

Control of quality record Integration and testing Control of inspection,

measuring, test Corrective and preventative

actions

cmsc435 - 39

ISO 9000 tasks

● writing a quality manual, describing the

organization's quality system at a high level

● writing procedure documents describing the

work is carried out in the organization

● creating a system to control distribution and

re-issue of documents

● identifying training needs for most positions in

the organization

● training people in the organization on

operation of the quality system

● planning and conducting internal audits

ISO 9000 standards

ISO 9000, "Quality management and quality assurance standards - Guidelines

for selection and use," clarifies the distinctions and interrelationships between quality concepts and provides guidelines for the selection and use of a series of international standards on quality systems that can be used for internal quality management purposes ( ISO 9004 ) and for external quality assurance purposes (ISO 9001, 9002 and 9003 ).

ISO 9001, "Quality systems - Model for quality assurance in

design/development, production, installation, and servicing." Of the ISO 9000 series, it is the standard that is most pertinent to software development and maintenance.

ISO 9002, "Quality systems - Model for quality assurance in production and

installation."

ISO 9003, "Quality systems - Model for quality assurance in final inspection

and test"

ISO 9004, "Quality management and quality system elements – Guidelines."

ISO 9000-3, provides "Guidelines for the application of ISO 9001 to the

cmsc435 - 41

Goals of ISO 9000

● An organization should achieve and sustain the quality of

the product or service produced so as to meet continually

the purchaser's stated or implied needs.

● An organization should provide confidence to its own

management that the intended quality is being achieved

and sustained.

● An organization should provide confidence to the

purchaser that the intended quality is being, or will be,

achieved in the delivered product or service provided.

The CMM and ISO 9000

● There is a clear correlation between the key

processes in the CMM and the quality

management processes in ISO 9000

● The CMM is more detailed and prescriptive

and includes a framework for improvement

● Organizations rated as level 2 in the CMM are

cmsc435 - 43

People capability maturity model

(Curtis, Hefley and Miller 1995)

Level Focus Key practices

5: optimizing

Continuous knowledge and skills improvement Continuous workforce innovation Coaching

Personal competency development 4: managed Effectiveness measured and

managed, high performance teams developed

Organizational performance alignment

Organizational competency management Team-based practices

Team-building Mentoring 3: defined Competency-based workforce practices Participatory culture

Competency-based practices Career development Competency development Workforce planning Knowledge and skills analysis 2: repeatable Management takes responsibility

for managing its people

Compensation Training Performance management Staffing Communication Work environment 1: initial

CMMI -1

CMM have become a growth industry …

●

_SW-CMM

_{- Capability Maturity Model for Software}

●

_P-CMM

_{- People Capability Maturity Model}

●

_SA-CMM

_{- Software Acquisition Capability Maturity}

Model

●

_SE-CMM

_{- Systems Engineering Capability Maturity}

Model

●

_IPD-CMM

_{- Integrated Product Development}

Capability Maturity Model

cmsc435 - 45

CMMI -2

The CMM Integration project was formed to address the

problem of having to use multiple Capability Maturity

Models. The initial mission of the project was to

combine three source models:

1. Capability Maturity Model for Software (SW-CMM) v2.0

draft C

2. Electronic Industries Alliance Interim Standard (EIA/IS) 731, Systems Engineering Capability Model (SECM)

3. Integrated Product Development Capability Maturity Model (IPD-CMM) v0.98

into a single model for use by organizations pursuing

enterprise-wide process improvement.

CMMI -3

● 24 process areas

● Each process area has a set of goals

● Up to seven practices are associated with each goal as a

way to satisfy that goal

● 6 point measurement scale for each process area:

Performed (goals satisfied)

Managed (Organizational policies define process)
Defined (Organizational standardization)

Quantitatively managed (Measurement involved)
Optimizing (Adaptive processes)

cmsc435 - 47

Capability assessment

● An important role of the SEI is to use the

CMM to assess the capabilities of contractors

bidding for US government defence contracts

● The model is intended to represent

organizational capability not the practices

used in particular projects

● Within the same organization, there are often

wide variations in processes used

● Capability assessment is questionnaire-based

The capability assessment process

Select projects for assessment Distribute questionnaires Analyse responses Clarify responses Identify issues for discussion Interview project managers Interview engineers Interview managers Brief managers and engineers Present

cmsc435 - 49

● _{Process improvement involves process analysis, standardization,}

measurement and change

● _{Process models include descriptions of tasks, activities, roles,} exceptions, communications, deliverables and other processes

● _{Measurement should be used to answer specific}

questions about the software process used

● _{The three types of process metrics which can be collected are}

time metrics, resource utilization metrics and event metrics

● _{The SEI model classifies software processes as initial,}

repeatable, defined, managed and optimising. It identifies key processes which should be used at each of these levels

● The SEI model is appropriate for large systems developed by

large teams of engineers. It cannot be applied without modification in other situations