OPM Example- Improving Software Code Quality by reducing Code Complexity using Klocwork

(1)

SEPG NORTH AMERICA│The CMMI CONFERENCE6-7 May 2014Washington DC. ASSARAF-COHEN|OPM- SW Improvement using Klocwork

OPM Example-

Improving Software

Code Quality by reducing Code

Complexity using Klocwork

©

Sarit Assaraf

Yossi Cohen

• _{Largest industrial company in Israel}

• _{Missiles, Satellites, UAVs, Avionics, Upgrades, Radars, etc.}

• _{Activities encompassing: Development, Production,}

Maintenance and Service of Aerospace Systems

• _{IAI divisions are}

_ISO9000

_and

_AS9100-C

_certified

• _{CMMI ML5}

_-

Systems Missiles

&

Space Group(5 Div.), ELTA Systems

Group Ltd. (4 Div.), MALAT Div.

• _{CMMI ML3}

_-

LAHAV Div., ENG Div.

(3)

Agenda

‏

3 OPTSAT 3000 OFEQ EROS C TECSAR Synthetic Aperture Radar Venus Ground control stations “SHAVIT” launch vehicles Observation satellites AMOS 3 AMOS 1,2 Communication satellites AMOS 4

Advanced comm. Sat.

Introduction and background Motivation and benchmarking Expected improvement and validation Deployment _Summary

(4)



Integration and testing of large scale systems can be a long and

tedious phase in the systems’ lifecycle.



Typical characteristics of this phase include many defects and

failures, not enough resources and significantly tight schedule.



In high maturity organizations we expect to find relatively few

defects in these late stages and expect even to prevent their

injection in the early lifecycle stages.



Software code complexity

is an important indicator of the amount

of defects expected at integration, by lowering the complexity, the

defect rate will drop and so will costs and schedule deviations.



Controlling complexity means controlling injection, and allows

planning realistic schedules, thus supporting the achievement of

quality and operational goals

4

(5)



Studies and publications show relationships between external SW

characteristics (quality) and internal product (static code)

attributes:

• Code complexity can predict defective SW components

• Static code complexity measures can be useful indicators of software quality, i.e. the amount of defects



Since defects cost increases through lifecycle, project costs may

increase dramatically



The "ideal" design process implies lower defect injection and early

detection at early project stages, thus preventing bugs and lowering

design costs and meeting projects' schedule.



This leads to a change in the SW design process and provides a new

tool for project’s risk management

5

Software Code Quality

(6)



A positive correlation between code complexity and the amount of

defects (faults) was suggested



Code complexity therefore, would be an important SW attribute

we will need to measure and control



The software engineers evaluate complexity, and by simplifying the

software code, adjust it until the defined goal.

6

Cyclomatic Complexity vs. Defects

(7)

SEPG NORTH AMERICA│The CMMI CONFERENCE6-7 May 2014Washington DC. ASSARAF-COHEN|OPM- SW Improvement using Klocwork Research results

Injected: 85% of the defects during Software coding Discovered: 30% during Unit Testing

50% during later testing stages

7

Conclusions:

Combining inspections, static analysis, and testing

is cheaper than testing by itself and leads to much better defect removal efficiency levels.

In concert, these

approaches also shorten development schedules by more than 45% because, when testing starts after inspections, almost 85% of the defects already will have been addressed.

Defect detection- Caper Jones' research

(1)

(8)

SEPG NORTH AMERICA│The CMMI CONFERENCE6-7 May 2014Washington DC. ASSARAF-COHEN|OPM- SW Improvement using Klocwork 8



The benchmarking research directed us to control code complexity



By analyzing code complexity immediately after compilation using

the static tool metrics, and prior to Unit Testing and Build release,

the Software designer can proactively control the actual amount of

bugs expected during testing stage



A benchmark research and analysis to evaluate and compare static

code tools was performed, collecting knowledge and experience

from other divisions and groups



The tools evaluation process roadmap:

• Mapping of current SW Dev. environments in our projects

• Identification of the alternatives (COTS available tools)

• Definition of the criteria to be used for the alternatives evaluation

• Alternatives evaluation according to criteria

• Summary and decision

(9)

Tools identification and mapping (projects)

DC – Double Check: X – Currently, Xp – In the past: Project: Currently in process, Was in process, Currently not in process. Multi 5 Parasoft DC PcLint Project Name Environment Target Aircraft name X ‏ …..,…… C & VRTX MCPA 486 S X ‏ …… C & VRTX MCPA 486 R X ‏ …. C & GHS MCPA PPc H X ‏ …., , …… C & VRTX MCPA 486 M1 X ‏ …. C & GHS MCPA PPc X ‏ X ‏ ….. X ‏ ….. C & GHS UMAM PPc X ‏ Xp ‏ …… C++, Rhapsody & GHS AMAC PPc

M2 AMAC PPc C++, Rhapsody & GHS ….. ‏X ‏

Xp ‏ X ‏ ….. C++, Rhapsody & GHS EIOC PPc X ‏ …. C Borland PC DSim

X(Lint for Unix)

‏

…… C Solaris Workshop, Motif

UNIX A Xp ‏ …… C# UNIX + PC I

(10)

LDRA

Klocwork

Parasoft

Multi+DC

Compatibility with dev. languages

V V V V V V -V(29) V(DC) V V X X

Compatibility with Operation systems VRTX (make file) ? V X X GHS -Multi initial V V V V GHS -Multi adjusted X+ V -V V Borland V V ? X Solaris V X X Motif V X X PC V V V V UNIX X V X X Check rules Misra2004 V V V V Misra2008 V X X

(11)

SEPG NORTH AMERICA│The CMMI CONFERENCE6-7 May 2014Washington DC. ASSARAF-COHEN|OPM- SW Improvement using Klocwork 11 LDRA Klocwork Parasoft Multi+DC Subject 4 10 NR NR Response to RFI 4 9 NR NR

Responsiveness (to questions)

2 9 NR NR Response time 4 9 NR NR Appointments 1 10 NR NR

Detailed explanations upon request

3 NR 10 NR Installation easiness 2 NR 10 NR Installation support 9 8 5 5

Compatibility to existing environment

9 9 9 9 Work environment 4.22 9.14 8.50 7.00 Sub Total

1. Contact Initiation & general issues

Alternatives evaluation according to criteria

(1)

(12)

Criteria Definition: 1 to 10 (10=full compliance) NR no response –

LDRA Klocwork Parasoft Multi+DC Subject 7 9 10 10 Project selection 10 9 5 2

Environment and compiler

10

9

3

Files addition and removal

8

10

4

10

Tailoring to project needs

8 10 10 8 Specific tests 7 9 10 10

Specific testing rules

9 10 10 10 Tests run 2 9 9 9

Tests run lead time for project M1

7.63

9.50

8.38

7.75

Sub Total

(13)

SEPG NORTH AMERICA│The CMMI CONFERENCE6-7 May 2014Washington DC. ASSARAF-COHEN|OPM- SW Improvement using Klocwork 13 LDRA Klocwork Parasoft Multi+DC 4.22 9.14 8.50 7.00

1. Contact Initiation & general issues

7.63

9.50

8.38

7.75

2. Tests Run process

5.75 7.88 7.88 2.25 3. Report generation 5.80 8.83 8.20 5.22 Total 0 1 2 3 4 5 6 7 8 9 10 םדק יללכו תצרה תוקידב תקפה תוחוד םוכיס יללכ LDRA Klocwork Parasoft Multi+DC Total Report generation General issues Tests run

Alternatives evaluation summary

At the end of the decision

process that involved senior

SW managers,

Klocwork

was

selected

(14)

Improvement Analysis and Validation

 One of the requirements of the OPM process is to analyze and validate the impact of the suggested improvement

 The validation can be performed either by conducting a pilot, or by simulation

 The benchmarking research implied that complexity influences the amount of defects, we needed to validate it on our processes

 We would improve the SW code development process, and particularly the sub-process of “code writing” to meet the quality goal of the project

 By selecting projects at a late stage of their lifecycle, that did not used Klocwork nor measured and controlled complexity, we could measure:

• The Quality Goal during later testing stages (i.e. No. of defects)

• The Process Performance prior to the improvement (i.e. Complexity)

 The data was analyzed and the complexity baseline and the defects amounts baselines were established

(15)

 The complexity of routines/functions from 9 projects was analyzed using Klocwork: 70% of the items had less than “Complexity=60”

15

“Software Code Complexity” Baseline

Scree plot of “Complexity” vs. Routine# Pareto dist. of “Complexity”

60 →

↑ #45

(16)

 The number of defects per 1000 work-hours from 9 projects was analyzed

 Analysis showed that project's lifecycle, “Reuse” or “New” (i.e. full scale development) was also a factor, and resulted in establishing two different baselines:

16

“Number of Defects” Baseline

Descriptive Statistics: VEN LR-SKR A4 FS A4 GS Tir A3 GS H-super LR-WCS H 30 20 10 0 -10 proj In d iv id u a l V a lu e _ X=3.20 UCL=12.44 LCL=-6.04 NEW REUSE

Total Defects vs Actual K work hours

Two project's Types: New & Reuse UCL=27.17

LCL=12.24 19.71 USL=21.74

USL=9.95

Variable Type Mean StDev Minimum Maximum defects vs. K work hours NEW 19.71 2.83 15.40 22.72 REUSE 3.203 1.981 0.963 5.000

(17)

Process improvement goal

 We set the target of lowering the amount of defects found during integration and testing stage, by 20% relative to the current baseline.

 By analyzing code complexity immediately after compilation and prior to Unit Testing and Build Release, the Software designer can predict and proactively control the actual amount of bugs expected during the later testing stages.

Goal baseline Current Baseline

Project type

9.8<y_New<21.74 (y_avg.=16)

12.24 < y_New < 27.17 (y_avg.=20)

“NEW”

y_Reuse<9.95 (y_avg.=2.5)

y_Reuse < 12.44 (y_avg.=3)

“REUSE”

(18)

Deployment Plan

 Validation on two projects of each type was performed, and the deployment plan was established, including:

• Success criteria of the deployment stage, and evaluation milestones

• List of the projects that will use Klocwork

• Resources – budget, number of licenses

• Installation, training and coaching activities

• Monitoring and meetings milestones and activities

• Measurements management activities

• Data analysis (from Klocwork)

• Complexity baseline update (every 6 months)

• Number of defects prediction and confidence interval analysis

 Klocwork will be used during compilation, and according to the complexity results the code will be corrected (<60), thus allowing meeting the “number of defects” goal.

(19)

The IAI conclusions

 Meeting project milestones strongly depends on a system readiness to be integrated, tested, delivered, and most importantly – whether it will adequately perform at the deployment stage.

 Software code complexity is an important indicator of the amount of defects

expected at integration, by lowering the complexity, the defect rate will drop and so will costs and schedule deviations.

 Klocwork tool provides complexity measurement immediately after compilation

and prior to Unit Testing and Build release, by simplifying the software code the Software designer adjusts it until the defined target goal.

 We distinguished two types of projects (“Reuse” and “New”), thus established two improvement goals values, and planned the deployment accordingly.

 The OPM implementation triggered changes in SW design processes (specifically

of SW “code writing”), leading to a different approach of project’s risk mitigation

based on the amount of defects prediction

 Deployment continues and final conclusions will be analyzed during this year