Does SRE practice have meaningful interpretation in Agile software development? How to apply common SRE practices in Agile software development?

(1)

SENG 521

Software Reliability &

Software Quality

Software Qualityyy

Chapter 15: Advances Topics

D t t f El t i l & C t E i i U i it f C l Department of Electrical & Computer Engineering, University of Calgary

B.H. Far （[email protected]）

http://www.enel.ucalgary.ca/People/far/Lectures/SENG521

[email protected] 1

Section 1

Software Reliability Engineering for

Agile Software Development

SENG521 (Winter 2008) [email protected] 2

SRE: Process

Requirement & Requirement & Architecture

Architecture ImplementationImplementationDesign &Design & TestTest 1 Define Necessary 1 Define Necessary 1. Define Necessary 1. Define Necessary Reliability Reliability 2. Develop Operational 2. Develop Operational Profile Profile

SRE ProfileProfile 3. Prepare for Test 3. Prepare for Test

5 A l 5 A l SRE Proc 4. Execute 4. Execute Test Test 5. Apply 5. Apply Failure Failure Data Data People involved:

Senior management; Test coordinator; Data coordinator; Customer or user

3

Customer or user

SRE: Critics …

Practical implementation of an effective SRE



Practical implementation of an effective SRE

program is a non-trivial task.



Occurrence probabilities of operational profile are



Occurrence probabilities of operational profile are

usually developers’ best guess.



Mechanisms for collection and analysis of data on

y

software product and process must be in place.



Fault identification and elimination techniques must

b i

l

be in place.



Other organizational abilities such as the use of

reviews and inspections reliability based testing

reviews and inspections, reliability based testing

and software process improvement are also

necessary for effective SRE.

4

y

Research Questions



Does SRE practice have meaningful

interpretation in Agile software

p

g

development?



How to apply common SRE practices in



How to apply common SRE practices in

Agile software development?

5

1. Necessary Reliability

i

i i

i

?

i

i i

i

?



How much reliability is good enough?

1) Define failure with “failure severity classes (FSC)” for

h d

the product.

2) Set a “failure intensity objective (FIO)” for each system t b t t d

to be tested.

3) Find the developed software failure intensity objective.

4) E i t t i t t th ft f il i t it

4) Engineer strategies to meet the software failure intensity objective. (Balance among fault prevention, fault-removal and fault tolerance strategies )

removal, and fault tolerance strategies.)

Steps 1

Steps 1--3 are valid for Agile process, too.3 are valid for Agile process, too. 6

(2)

Fault Prevention & Removal

U i

A il

i

th

diff

t

f



Using Agile process requires rather different way of

thinking about fault prevention and fault removal.

T t D i D l t (TDD)

T t D i D l t (TDD) i d



 Test Driven Development (TDD):Test Driven Development (TDD):unit tests and

acceptance tests are written prior to functioning code. Continuous Integration:

Continuous Integration:regression tests are performed



 Continuous Integration:Continuous Integration:regression tests are performed

frequently.



 Pair programming:Pair programming:shifts the activity of code reviews 

 Pair programming:Pair programming:shifts the activity of code reviews

into prevention.

Most of fault

Most of fault removal activities (in traditional removal activities (in traditional SRE definition) are

SRE definition) are moved into fault preventionmoved into fault prevention 7

2. Operational Profile

O

ti

l P

fil

(OP) i

l t

t f



Operational Profile

(OP) is a complete set of

operations with their probabilities of occurrence



The purpose of OP is to provide the developers with

the information about how users will use the

product being built

Needed for better management of Needed for better management of

d l d

development and test resources development and test resources

8

Operational Profile vs. User Story

Si i

i i

Si i

i i



Similarities:

 Both contain a complete piece of functionality  Both are directly related to requirements

 Both share an emphasis on developing more critical functionality first in order to achieve faster time to market



Differences:

 SRE process has an additional requirement on frequency

of use of each operation

 Collecting usage data by user story (operation in SRE

terms) is often missed by teams following agile practices

9

How to Find Operations?

Use-Case Realization (identified) Supplementary Specifications Glossary (identified) Use-Case Model Software Architecture Document Operations Use-Case Realization (developed) Design p Boundary Entity Control Analysis Classes Elements Classes Subsystems Interfaces 10 Interfaces Signals & events

Case of Agile Development /1



Main source: User stories



Main source: User stories



Example:

11

Case of Agile Development /2

S t t i

f

h

t



Sort stories for each user type



Example:

p

BSC clerk operation list:

p

Using blind TDD may lead to tests that do not reflect usage! 12

(3)

3. Prepare & Execute Test



Prepare test cases and test procedures.



User stories

already contain a list of



User stories

already contain a list of

acceptance tests equivalent to feature (or

it) t t

t t f

if i

unit) tests or tests for verifying

non-functional requirements.

E

t t t

d

ll t f il

d t



Execute tests and collect failure data.

13

Prepare & Execute Test



SRE:

regression tests run after every build

involving significant change.

g



Agile process:

continuous integration.

Checking code and tests into the common

repository several times per day. Each time a

h k i i

d

ll

i

check-in is made, all regression tests are run,

therefore regression tests are run very

frequently.

14

Preparing Test



Differs significantly from common SRE

practices:

in Agile, tests are not allocated

p

g

based on usage and/or criticality



All testing should be done in conjunction



All testing should be done in conjunction

with development, with tests written prior to

f

i

li

i

l

i

functionality implementation.



Test definitions are established based on

‘need’ rather than usage.

whose ‘need’?

15

Number of Test Cases



The number of test cases is another point of

deviation from SRE which is calculated

based on available budget and time.



In Agile development a common practice is



In Agile development a common practice is

that the development team will enforce a unit

i (

h

95% f

ll

test coverage metric (such as 95% for all

developer written code).



Every functional user story will receive at

least one test case

16

least one test case.

Test Case Selection & Run

f

i



For performance testing, acceptance tests are

repeatedly run by selecting each based on its

i

d

i

d i

h

i

l

proportionate usage depicted in the operational

profile.



Occurrence probabilities

indicated in the

operational profile can be used as a bias for test

selection.

17

Test Preparation Summary



Deviates significantly from the standard SRE

approach.

pp



Rather than planning for a certain number of

tests and allocating tests and time for tests

among defined systems, test driven approach

ill

l i

h

i

f

will result in a comprehensive set of tests.



Since functionality is not written without

y

tests, every part of the system will be well

tested

18

tested.

(4)

Reporting Tools

NU i



NUnit

for running unit tests



CruiseControl.NET

for continuous

integration

g



CCTray

for instant notification



CCTray

for instant notification



LoadRunner

for performance testing



LoadRunner

for performance testing



CASRE

for failure analysis

19 

CASRE

for failure analysis

Tools:

NUnit

 NUnitgives meaningful real-time feedback to developers  NUnitgives meaningful, real time feedback to developers

for both new tests and for regression test.

 Users can choose to run only a single test, or the entire suiteUsers can choose to run only a single test, or the entire suite of unit tests for a given project or a group of projects.

20

Tools:

CruiseControl.NET



CruiseControl NET

examines the common code



CruiseControl.NET

examines the common code

repository for changes. Finding a change, it

compiles the entire solution and runs all tests

compiles the entire solution and runs all tests.

21

Tools:

CCTray

CCT



CCTray

runs on clients and can instantly

notify each developer about the current state

y

p

of the build status (pass or fail), and upon

failure the web interface can be used to

determine the exact cause of the failure.

22

Tools:

LoadRunner

L dR



LoadRunner

can be used for performance

testing.

g



It allows for simulation of multiple users, and

can execute code or html web queries

can execute code or html web queries.



It also allows for the spawning of agents to

multiple client machines, if necessary, to

simulate a real web application.

pp

23

Tools:

LoadRunner

(5)

Tools:

CASRE

Ti B t F il ith F il Time Between Failure vs. ith Failure

900 1000 700 800 400 500 600 H our s 200 300 0 100 1 11 21 31 41 51 61 71 81 91 ith F il F il Ti 25

ith Failure Failure Time

5. Guided Decision



Assessing

rather than

Measuring

reliability



Reliability verification procedure is via

Certification Test

process

I

tifi ti

t ti

t

l

l t



In certification testing we cannot calculate

the exact amount of reliability of the

developed product but we can be sure that it

has exceeded a certain minimum level of

reliability defined by the FIO set during the

defining necessary reliability phase

26

defining necessary reliability phase.

Reliability Demo Chart /1

 An efficient way of  An efficient way of

checking whether the FIO (F) is met or not.

i b d  It is based on

collecting failure data at time points.p  Vertical axis:failure

number (n)  Horizontal axis:  Horizontal axis: normalized failure data (Tn), i.e., f il i  failure time  F or failure time / MTTF 27

Lessons Learnt /1

i

i i

f S

i



Reliability growth models

of SRE may not suit a

test-driven process



Test-driven development (TDD)

approach and

Certification Testing

together can be very useful



Allocation of tests (number of test and time for

system testing) can be better managed by an

y

g)

g

y

operational profile



Certification Testing will assure minimum level of



Certification Testing will assure minimum level of

reliability has been achieved

28

Lessons Learnt /2



Regarding test allocation, there seems to be a

more natural fit in defining tests on a user

g

scenario basis rather than allocating a number

of tests per operation.



However we lose the benefit of knowing that

ll

i i l

i

i d d d

l

all critical operations are indeed adequately

covered, and we must rely on developer skill

rather than the established metrics and

statistics of SRE.

Operational profile will resolve this

29

stat st cs o S

.

p p

Lessons Learnt /3



More research:



Establishing relationship between test coverage

g

p

g

and mean time to failure or failure intensity.



Incorporating performance testing into the



Incorporating performance testing into the

continuous integration and automated build

practices that agile promotes.



Classifying failures and ensuring that potential

failures are in fact tested

failures are in fact tested.

(6)

Enhancing Agile



Define failure severity classes and cover the

potential failures in the acceptance tests

p



Defining operational profile and using it to

define proper number of tests and system test

duration



Add mechanisms for tracking performance



Add certification testing to ensure minimum



Add certification testing to ensure minimum

required quality is delivered

[email protected] 31

Conclusions

A il

f

d

l



Agile software development

treats quality as ‘cats for the

bli d’!

blind’!



While SRE related concepts

and techniques may initially

seem incompatible with the

test-driven emphasis of Agile

methods, we showed that SRE

does in fact have value within

the Agile process context.

32

Section 2

Cleanroom

Cleanroom Software

Software

Development

D t t f El t i l & C t E i i U i it f C l Department of Electrical & Computer Engineering, University of Calgary

B.H. Far （[email protected]） http://www.enel.ucalgary.ca/People/far/Lectures/SENG521/12/ [email protected] 33

Background

Ch R [St di h 1995]

 Chaos Report [Standish 1995]

 Based on data representing 8,380 SE projects, only 16.2% of projects met the delivery date, the budget and with all of the specified features met the delivery date, the budget and with all of the specified features and functions. 31% of projects were cancelled before they were completed, 52.7% were delivered with over-budget, over-schedule or with fewer features and functions than specified

with fewer features and functions than specified.  Software Productivity Research [Chapman 2000]

 %60 of the United State’s software work force is dedicated to fixing software errors that could have been avoided. In addition, there are only 47 days in a calendar year dedicated to doing development or y y y g p enhancement of software applications. The rest is spent mainly on fixing the bugs.

The First Computer Bug!

O S t b 9th₁₉₄₅

 On September 9th _1945, Grace Murray Hopper was working on the Harvard g University Mark II Aiken Relay Calculator when the machine was experiencing machine was experiencing problems.

 An investigation showed g that there was a moth trapped between the points of Relay #70 in Panel F of Relay #70, in Panel F.

Courtesy of the Naval Surface Warfare Center, Dahlgren, VA., 1988.

Courtesy of the Naval Surface Warfare Center, Dahlgren, VA., 1988. http://www.history.navy.mil/photos/pers-us/uspers-h/g-hoppr.htm

Research Question



Is it possible to build software without any

bug in it?

g

A

M

b

B

i

l



Answer: May be. By using

cleanroom

software development

(7)

Causes for Bugs in Programs

Th

i

f

b

i



The main causes for bugs in programs:



Design flaws



Coding error



Other (including human related error)

(

g

)



The first two can be eliminated by

formal



The first two can be eliminated by

formal

(e.g. box structure) design verification

and

automated code generators Certification

automated code generators

.

Certification

testing

will take care of the last.

Cleanroom SE

Cl ft i i (CSE) i i i

 Cleanroom software engineering (CSE) is an engineering process for the development of high quality software with certified reliability with the emphasis on designy p g with no defectsand test based on software reliability engineering

concepts.

CSE focuses ondefect preventioninstead of defect  CSE focuses on defect preventioninstead of defect

correction, and certificationof reliability for the intended environment of use.

 CSE yields software that is correct by mathematically sound design, and software that is certified by statistically valid testing.g

 CSE represents a paradigm shift from traditional, craft-based SE practices to rigorous, engineering-based practices.

CSE: Characteristics



Objective:

Achieve zero defects with

certified reliability

y



Focus:

Defect prevention rather than defect

correction



Process:

Incremental (short) development

cycles; long product life

CSE: History

 1983:Original idea of Cleanroom came from one of Dr Harlan Mills’

 1983:Original idea of Cleanroom came from one of Dr. Harlan Mills published papers

 1987:Proposed by Dr. Mills as a SE methodology. The name “Cleanroom” was borrowed from the electronics industryCleanroom was borrowed from the electronics industry

 1988:Defense Advanced Research Projects Agency (DARPA) Software Technology for Adaptable Reliable Systems (STARS) focus on Cleanroom

 1991-1992:Prototyping of Cleanroom Process Guide

 1992:A book of CSE published, foundation of CSE

 1992-1993:1992 1993:Army and Air Force Demonstration ofArmy and Air Force Demonstration of Cleanroom Technology

 1993-1994:Prototyping of Cleanroom tools

 1995:Commercialization of a Cleanroom Certification

 1995:Commercialization of a Cleanroom Certification Tool

 1995:Cleanroom and CMM Consistency Review

[email protected] 40  …

Comparison

Craft Craft--Based SEBased SE

Sequential or chaos development

Cleanroom SE

Incremental development

Informal design Disciplined engineering specification and design

Unknown reliability Measured reliability

Individual development

Individual unit testing

Peer reviewed engineering

Team correctness verification Individual unit testing

Informal load or coverage testing

Team correctness verification

Statistical usage testing

Cleanroom SE: Technologies

D l i b d h i l f i

 Development practicesare based on mathematical function theory

Test practicesare based on applied statistics  Test practicesare based on applied statistics

 Analysis and design models are based on incremental y g software model and created using box structure representation

 A boxencapsulates the system (or some aspect of the system) at a specific level of abstraction

 Correctness verification is applied once the box structure design is complete

(8)

Cleanroom SE: Technologies

 Software is tested by defining a set of usage scenarios(i.e.,

operationsor operational modes), determining the

probability of use for each scenario (i.e., operational profile), and then defining random tests that conform to the probabilities.

 Error records are checked. No corrective actions are taken. Only certification test is conducted to check whether errors (i.e., current failure intensity) meet the projected reliability

( y) p j y

(i.e., failure intensity objective) for the software component.

CSE: Processes /1

Cl

Cleanroom processes:

1.

Management process

2.

Specification process

3

Development process

3.

Development process

4.

Certification process

CSE: Processes /2

1

Cleanroom Management Process

1.

Cleanroom Management Process

 Project planning  Project management  Project management  Performance improvement  Engineering change

2.

Cleanroom Specification Process

 Requirements analysis  Function specification  Usage specification  Architecture specification  Architecture specification  Increment planning [email protected] 45

CSE: Processes /3

3.

Cleanroom Development Process



Increment design

g



Correctness verification



Software reengineering (reuse)



Software reengineering (reuse)

4.

Cleanroom Certification Process



Usage modeling and test planning



Statistical testing and certification



Statistical testing and certification

CSE: Management Process

P

j t Pl

i



Project Planning



Cleanroom engineering guide



Software development plan (incremental)



Project Management

oject

a age e t



Project record



Performance Improvement



Performance Improvement



Performance improvement plan



Engineering Change



Engineering change log

CSE: Specification Process /1



 Requirements AnalysisRequirements Analysis



 Requirements AnalysisRequirements Analysis

 Elicitation and analyzes of requirements  Define requirements for the software productq p

 Obtain agreement with the customer on the requirements  Requirements are reconfirmed or clarified throughout the

i t l d l t d tifi ti

incremental development and certification process. 

 Functional Functional SpecificationSpecification

 Based on the result of Requirements Analysis  Based on the result of Requirements Analysis

 Specify the complete functional behavior of the software

in all possible modes of use

Ob i i h h h ifi d

 Obtain agreement with the customer on the specified

function as the basis for software development and certification

(9)

CSE: Specification Process /2

U S ifi ti U S ifi ti



 Usage SpecificationUsage Specification

 Identify and classify software users, usage scenarios, and

environments of use (operational modes) environments of use (operational modes)

 Establish and analyze the probability distribution for

software usage models

 Obtain agreement with the customer on the specified

usage as the basis for software certification

A hit t S ifi ti A hit t S ifi ti



 Architecture SpecificationArchitecture Specification

 Define the conceptual model, the structural organization,

and the execution characteristics of the software and the execution characteristics of the software

 Architecture definition is a multi-level activity that spans

the life cycle

CSE: Specification Process /3



Increment Planning



Allocate customer requirements defined in the

q

Function Specification to a series of software

increments that satisfy the Software Architecture,

y

,



Define schedule and resource allocations for

increment development and certification



Obtain agreement with the customer on the

increment plan

CSE: Development Process

Increment 1 Increment 1 RG RG BSSBSS FDFD CVCV CGCG CICI SUTSUT CC Increment 1 Increment 1 TP TP Increment 2 Increment 2 SE SE RGRG BSSBSS FDFD CVCV CGCG CICI TP TP SUTSUT CC TP TP

SE: System Engineering

RG R i t G th i CG: Code GenerationCI: Code Inspection RG: Requirement Gathering

BSS: Box structure specification

FD: Formal Design

CV: Correctness Verification

CI: Code Inspection

SUT: Statistical Use Testing C: Certification Test TP: Test Planning [email protected] 51

Cleanroom Strategy /1

i

( G)

i

( G)



Requirement gathering (RG)

 A detailed description of customer level requirements for

h i

each increment.



Box structure specification (BSS)

 Functional specification using box structure to separate

behavior, data and procedures.



Formal design (FD)

 Specifications (black boxes) are refined to become analogous to architectural (state boxes) and procedural (clear boxes) design.

C

ifi

i

(C )

C

ifi

i

(C )



Correctness verification (CV)

 A set of correctness verification activities on the design

d l d Fi l l ifi i i i

and moves later to code. First level verification is via application of a set of “correctness questions”.

C d

ti

i

ti

&

ifi

ti

(CG

C d

ti

i

ti

&

ifi

ti

(CG



Code generation, inspection & verification (CG

& CI)

 The box structure transformed to a programming

language. Walkthrough and code inspection techniques are used to ensure semantic conformance with the box are used to ensure semantic conformance with the box structure. [email protected] 53

Cleanroom Strategy /3

S

i i

i

(

)

S

i i

i

(

)



Statistical test planning (TP)

 Planning the test based on operational modes, operational fil d li bili

profiles and reliability.



Statistical use testing (SUT)

 Creating test case, execute them and collecting error data.



Certification (C)

( )

 Conducting certification test rather than reliability growth

to accept/reject developed software components (using reliability demonstration chart, etc).

(10)

Box Structure /1

 Box structures are used to move from an abstract  Box structures are used to move from an abstract

specification to a detailed design providing implementation details [email protected] 55

Box Structure /2

Bl

k b

Bl

k b



Black box

 Specifies the behavior of a system or a part of a system. The system responds to specific stimuli (events) by The system responds to specific stimuli (events) by applying a set of transition rules that map the stimuli to response.



State box

 Encapsulates state data and services (operations). Input to

the state box and outputs are represented.



Clear box

 Transition function that are implied by the state box. It

contains the procedural design of the state box.

Box Structure /3

f: S* →R S R _State T Black box _S R g11 g12 ccg1 State T g13 g clear box f: S* →R S R clear box State box

Black boxes (specifications) State boxes (architectural designs)

Cl b ( t d i )

State box Clear boxes (component designs)

Box Structure /4

CB CB1.1.11.1.1 CB CB1 1 21 1 2 BB BB1.1.11.1.1 SBSB1.1.11.1.1 CB CB1.1.21.1.2 CB CB1.1.31.1.3 BB BB BB BB1.11.1 BB BB1 21 2 BB BB1.1.21.1.2 BB BB11 BBBB1.21.2 BB BB1.1.31.1.3 Clear box BB BB1.n1.n State box Black box [email protected] 58

Correctness Verification

 Mathematical based techniques are used to verify the  Mathematical based techniques are used to verify the

correctness of a software increment  Examples

 If a function fis expanded into a sequence gand h, the correctness rule for all input to fis:

Does gfollowed by hdo f?

 If a function fis expanded into a condition if-then-else, the correctness rule for all input to fis:

Whenever condition <c> is true does gdo fand whenever <c> is f l d hd f?

false does hdo f?

 When function fis refined as a loop, the correctness rule for all input

to fis:

I i i d?

Is termination guaranteed?

Whenever <c> is true, does gfollowed by fdo f? and whenever <c> is false, does skipping the loop still do f?

Black Box Structure /4



If having more than one black box or nested

black boxes verify the mapping

y

pp g

f f f f g h Sequential split g c Sequential split h P ll l lit c [email protected] 60 Parallel split

(11)



black boxes verify the mapping

y

pp g

f f f f h g c g c g Loop split c Loop split [email protected] 61

Advantages of Verification



Design verification has the following

advantages:

g



Verification is reduced to a finite process



Every step of design and every line of code can



Every step of design and every line of code can

be verified

N

d f t l

l i

hi

d



Near zero defect level is achieved



Scalability is possible



Better code (than unit testing) can be generated

CSE: Certification Process /1

U d li d t t l i



 Usage modeling and test planningUsage modeling and test planning

 A usage model represents a possible usage scenario of the software

 Usage model is based on usage specification and is used for testing  Usage model is based on usage specification and is used for testing

Similar to the way we defined operations [email protected] 63

CSE: Certification Process /2

S

i i

i

C

ifi

i

S

i i

i

C

ifi

i



Statistical Testing and Certification

 Testing is conducted in a formal statistical design under

i l l

experimental control.

 The software is demonstrated to perform correctly with

t t it ifi ti respect to its specification.

 Statistically valid estimates of the properties addressed by

the certification goals are derived for the software the certification goals are derived for the software.

 Management decisions on continuation of testing and

certification of the software are based on statistical certification of the software are based on statistical estimates of software quality.

Cleanroom Testing

i

i i

l

f

i



Using

statistical usage

concept for testing.



Determine a

usage probability distribution

via the

following steps:

1) Analyze the specification to identify a set of stimuli (direct and indirect input variables).

2) Create usage scenarios (operational modes).

3) Assign probability to use of each stimuli (operational profile).

4) Generate test cases for each stimuli according to the usage probability distribution.

Certification Test

Cl

h

O S

O

h i



Cleanroom approach DOES NOT emphasize on

 Unit or integration testing.

 Bug fixing as a result of test and regression.



Certification procedure involves the followings:

 Create usage scenarios.  Specify a usage profile.p y g p

 Generate test cases from the profile.  Execute test cases and record failure data.

 Compute reliability and certify the component or system

using reliability demo chart, etc.

(12)

Reliability Demo Chart

A ffi i t f

 An efficient way of checking whether the failure intensity

bj ti () i t objective (F) is met or

not based on collecting failure data at time

i points.

 Vertical axis:failure numberu be

 Horizontal axis: normalized failure data, i e i.e., failure time  F [email protected] 67

Example



Automated Teller Machine (ATM)



Requirements:



The customer has a PIN number and access-card

to use the ATM



The customer can deposit, withdraw money from

th

t

the account



Transaction involves no bank employee

Example: Usage Model

Customer

<<extends>> <<extends>>

Withdraw Deposit

Usage percentage: 50% Usage percentage: 50%

Card Processor Card Processor Cash Dispenser Cash Dispenser Transaction Manager Transaction Manager [email protected] 69

Example: Black Boxes



 Black boxesBlack boxes 

 Black boxesBlack boxes  Card Processor  In: ValidCard(cardNum)  Out: showMessage(message) Card Processor Card Processor g ( g ) Boolean  Cash Dispenser  In: enoughCashInMachine(amount) Cash Cash dispenseCash(amount)  Out: showMessage(message) dispense(amount) Boolean Cash Dispenser Cash Dispenser Boolean  Transaction Manager

 In: ValidCustomer(cardNum, pin)

AmountLimit(amount) Transaction Manager Transaction Manager EnoughCashInAccount(amount)  Out: showMessage(message) Boolean Manager Manager [email protected] 70

State Box: Card Processor

Menu

/insert card /idle

Menu

/get pin [false] /send message Check /notify user [true] /call notify To cash dispenser Notify /call notify /send card verified [email protected] 71

State Box: Cash Dispenser

Menu /get card verified Menu

/get amount [false] [true] Check Check Machine cash [false] [false] [true]

/call check account [true]

[ ]

Check Account /call check account

(13)

Example: Clear Box Spec /1

M

/ insert card

// Get customer PIN no ValidCustomer(cardNum, pin) Menu / get pin [false] [true] Check CheckMachine Cash [false] [false] [true] [true] [false] CheckAccount [true] /get amount [email protected] 73

M

/ insert card

// Bank returns false // Show message showMessage(mesg); Menu / get pin [false] [true] Check CheckMachine Cash [false] [false] [true] [true] [false] CheckAccount [true] /get amount [email protected] 74

Example: Clear Box Spec /3

M

/ insert card

// Bank returns true // get amount getAmount(amount); Menu / get pin [false] [true] Check CheckMachine Cash [false] [false] [true] [true] [false] CheckAccount [true] /get amount [email protected] 75

M

/ insert card

// Bank returns false for daily limit // and/or balance // Show message showMessage(mesg); Menu / get pin showMessage(mesg); [false] [true] Check CheckMachine Cash [false] [false] [true] [true] [false] CheckAccount [true] /get amount [email protected] 76

Example: Clear Box Spec /5

M

/ insert card

// Bank returns true for daily limit // and balance Dispenser.enoughCashInAccount(amount) Menu / get pin [false] [true] Check CheckMachine Cash [false] [false] [true] [true] [false] CheckAccount [true] /get amount [email protected] 77

M

/ insert card

// Dispenser returns false for // cash level // Show message showMessage(mesg); Menu / get pin showMessage(mesg); [false] [true] Check CheckMachine Cash [false] [false] [true] [true] [false] CheckAccount [true] /get amount [email protected] 78

(14)

Example: Clear Box Spec /7

M

/ insert card

// Dispenser returns true for // cash amount Dispenser.dispense(amount); Menu / get pin [false] [true] Check CheckMachine Cash [false] [false] [true] [true] [false] CheckAccount [true] /get amount [email protected] 79

Example: Testing



50% of tests go to test Withdraw; 50% to test

Deposit

p



Test cases are created to test each execution

path (or state transition path)



A subset of tests are selected for validation

CSE: Team

S ifi ti t

 Specification team:

 Responsible for developing and maintaining the system specification

 Development team:

 Responsible for developing and verifying the software

 The software is not executed during this process

 Certification team:

 Responsible for developing a set of statistical tests to exercise the

ft ft d l t

software after development

 Use reliability growth models to assess reliability

CSE: Evaluation



Basic features of Cleanroom development

that distinguishes it from other SE

g

methodologies are:

1

Formal specification (Box structure) &

1.

Formal specification (Box structure) &

Correctness verification

2

Statistical certification test

2.

Statistical certification test

Evaluation: Formal Spec

Ad

t

Ad

t



Advantages:

 Mathematical and logical foundation for defining requirements accurately with precise notation requirements accurately with precise notation.

 Proactive versus reactive approach with regards to

requirements validation. q

 Ambiguous, inconsistent and conflicting requirements are caught before the system test.

 Box structure uses black, state, and clear box and it is a

stepwise approach to refine requirements.

 Usage models define how the software is to be used by

 Usage models define how the software is to be used by the customer.

Evaluation: Formal Spec



Disadvantages:

 Requires extra skills and knowledge (e.g. mathematics)

 Requires substantial effort to fully express the system in

formal specification

 On average Cleanroom projects require 60-80% of the time used in analysis and design

 Ideal for safety or mission critical systems and not for ordinary  Ideal for safety or mission critical systems and not for ordinary

commercial development

 Lacks good enough CASE tools supporting  Project specific

 If time-to-market & conditions are issues, then might not be used

(15)

Evaluation: Incremental Devel

Ad t



 AdvantagesAdvantages

 Quick and clean development in Cleanroom Engineering

 Continuous validation  Provides measurable progress

 Manage higher risk requirements (i.e. prototype).

T ki f i t

 Tracking of requirements

 Stepwise building functionalities that satisfies stakeholders’

requirements

 Allows for fast delivery on important parts

 Focus on planning and discipline at management level and technical level

 Statistical testing make the project quality control in proper level

 Verifiable specifications

Evaluation: Incremental Devel

ii



Disadvantages:

 Incomplete or conflicting requirements cannot be

l d h b i i d i i

resolved at the beginning to determine increments

 Risk analysis has not been incorporated explicitly

 Need more care about configuration management  Requires extra planning at both the management and

t h i l l l technical levels

 Stable requirements for each increment is needed, i.e.,

cannot adapt quickly to “rapidly changing” requirements cannot adapt quickly to rapidly changing requirements

Evaluation: Certification Test



Advantages



Determines a level of confidence that a software

system conforms to a specification



Able to statistically evaluate and infer the quality



Able to statistically evaluate and infer the quality

of the software system to meet all requirements



Quantitative approach that is verifiable



Quantitative approach that is verifiable



Quantitative data could be recorded and used

l t

f

b

h

ki

t

later for benchmarking, etc.

Evaluation: Certification Test

ii



Disadvantages

 Testing is derived from a usage modelthat must be

h i i d l b f i

exhaustive in order to select a subset for testing

 Statistical testing and verification will be more reliable if

it i b d th hi t d t

it is based on the some history data

 It would be effective if it could be integrated with other

testing methods testing methods

 Testing is not suitable for bug-hunting

Human residual coding errors may not be addressed

 Human residual coding errors may not be addressed

Section 3

Case Study

Cleanroom SE: Case Study

Cl

ft

d

l

t

li



Cleanroom software development relies on a

mathematically sound model of design to

h

d f

i

d

d i

h

ensure that no defects are introduced into the

software.



Cleanroom Software Specification and

Design begins with an external view (black

g

(

box), and is transformed into a state machine

view (state box), and is fully developed into a

(

),

y

p

procedure (clear box).

(16)

Box Structure



Box structures map system inputs and the

stimulus histories (previous inputs) into

outputs.



Is Black-Box construct sufficient to represent

this?



e g Jackson model

this?



e.g. Jackson model



No

Box structure inputs outputs history [email protected] 91 history

Cleanroom SE: Process /1

1) D fi

th

t

i

t

1) Define the system requirements

2) Specify and validate the black box

 Define the system boundary and specify all stimuli and

responses

 Specify the black box mapping rules  Specify the black box mapping rules

 Validate the black box with owners and users

3) Specify and verify the state box

 Specify the state data and initial state values

 Specify the state box transition function  Specify the state box transition function

 Derive the black box behavior of the state box and compare the derived black box for equivalence

Cleanroom SE: Process /2

4)

Design and verify the clear box

 Design the clear box control structures and operations  Embed uses of new and reused black boxes as necessary  Derive the state box behavior of the clear box and

compare the derived state box to the original state box for equivalence

5) Repeat the process for new black boxes

6) Convert to code

)

7) Certification test the code

Requirements Analysis /1

Th

b

i

h

f th f ll

i

t



The process begins when one of the following entry

criteria is satisfied.

E t

1



Entry 1

 The Statement of Work or other initial artifact, such as a statement of allocated system requirements is available statement of allocated system requirements, is available.



Entry 2

 Changes including additions and corrections to the  Changes, including additions and corrections, to the

Software Requirements are proposed.



Entry 3



Entry 3

 A completed increment is ready for customer execution

and evaluation.



Task 1



Define the software requirements.

q



Task 2

U

l ti

f

h i

t

fi



Upon completion of each increment, reconfirm or

clarify requirements through customer evaluation

f th

t bl

t

of the executable system.



Verification 1



Review the evolving Software Requirements

g

q

work product.



Verification 2



Verification 2



Validate the Software Requirements work

d

t

ith th

t

d

product with the customer and peer

organizations.

(17)

Requirements Analysis



Example:



Build a simple calculator

p

D

il d d fi i i

f h

l

f

i



Detailed definition of the calculator function

and what it does must be given and verified

with the customer



Various formal methods can be used: graph



Various formal methods can be used: graph

theory, automaton model, etc.

Formal Specification

Example:



Use logic constructs such as “is-a”

Use logic constructs such as is a

(inheritance hierarchy) , “has-a” (association)

and “such as” (examples) to formulate the

and such-as (examples) to formulate the

specification.



Verify that the total set of sentences form a

tree (directed graph with no loop)

(

g p

p)



Entry #1: first operand (xxx digits)



Entry #2: calculation symbol (add, subtract)



Entry #3: second operand



Entry #4: equal symbol



Exit #1: calculation result



Exit #1: calculation result

Box structure

inputs outputs

history

Black Box Structure /2

calculator

1st_operand _null

null

calculator

Calc symbol null

1st_operand

1 operand

queue PushPush--down automaton modeldown automaton model

queue

Black Box Structure /3

l l t 2nd d ll calculator 2ndoperand null 1stoperand Calc sym calculator Calc sym queue

Equal symq y Calc results

1st_operand Calc sym 2nd_operand [email protected] 101 2nd_operand queue

Black Box Structure /4



f f g h g g Sequential split g h c h Parallel split [email protected] 102

(18)



If having more than one black box or nested



f f f f h g c g c g Loop split c Loop split [email protected] 103

State Box Structure /1



State transition diagram



happy path



State transition diagram



happy path

non-numeric key Error

y pressed

State Box Structure /2

S

l t t b

b

t d



Several state boxes can be generated

depending on the combination of acceptable

(

bl ) i

d hi

i

(unacceptable) inputs and histories



Example:

p



If 1

st

operand is non-numeric and calc symbol are

typed the next state is error state

yp



If 1

st

operand is numeric and any other key other

than calc symbol is typed the next state is error

y

yp

state



etc.



State boxes should be generated for all

possible combinations of input(s) and history

p

p ( )

y

states.



The set of state boxes can easily grow



The set of state boxes can easily grow

beyond control!

Clear Box Structure /1

Code Generation



Coding will be based on the clear boxes



Use of automatic code generation tools is

encouraged to reduce the probability of

human error

(19)

Cleanroom Testing /1



Cleanroom testing teams must determine a

usage probability distribution for the software

g p

y



The operational profile can be used

S

th t th i

t t th

l

l t



Suppose that the inputs to the calculator program

are

Input percentage number

A1 1st_{operand (correct)} _{% 22} _{0 – 21} A2 1st _{d (i} _t) _%3 ₂₂ ₂₄ A2 1st_{operand (incorrect)} _%3 _{22 – 24} B1 2nd_{operand (correct)} _{% 22} _{25 – 46} B2 2nd_{operand (incorrect)} _%3 ₄₇ ₄₉ B2 2nd_{operand (incorrect)} _%3 _{47 – 49}

C1 Calculation symbol (correct) % 22 50 – 71 C2 Calculation symbol (incorrect) %3 72 – 74 C2 Calculation symbol (incorrect) %3 72 74 D1 Equal symbol (correct) % 22 75 – 96 D2 Equal symbol (incorrect) %3 97 – 99

Cleanroom Testing /3

W t t f t t th t

 We must generate a sequence of usage test cases that conform to the usage probability distribution.

 A series of random numbers are generated between 0 and 99  A series of random numbers are generated between 0 and 99

that corresponds to the probability of stimuli occurrence.  For example, the following random number sequences are

generated:

 14 –95 –26 –44 : A1; D1; B1; B1

 81 19 31 69 Test case

 81 –19 –31 –69

 38 –21 –52 –84

 The testing team executes the test cases noted above (and

Test case

others) and verifies software behavior against the specification for the system.

Cleanroom Testing /4



For example for the test case

T1:

A1; D1; B1; B1

T1:

A1; D1; B1; B1

The input sequence is:

p

q

1. 1st_operand

2. Equal symbolq y 3. 2nd_operand 4. 2nd_operand 4. 2 operand

And the output should be:

Error

Cleanroom Certification

 The verification and testing techniques lead to certification  The verification and testing techniques lead to certification

of software components

 Certification implies that the reliability can be specified for h

each component.

 Each component would have a certified reliability under the usage scenario and testing regime. This information is usage scenario and testing regime. This information is needed for future use of the components.

 The certification approach involves five steps:

1 U i i t d

 1. Usage scenario is created.

 2. A usage profile is specified.

 3. Test cases are generated from the profile.

 4. Tests are executed and failure data are recorded and analyzed.

 5. Reliability is computed and certified.

CSE: Overall Advantages

S it bl f

it

ti

d i

t l



Suitable for iterative and incremental

software development.



Uses formal specification that defines more

accurate, less conflict and complete

,

p

requirements.



Continuous verification of software quality is



Continuous verification of software quality is

possible.

Software quality can be certified using



Software quality can be certified using

software reliability engineering method.

(20)

CSE: Overall Disadvantages

Cl

d

t

th

f

b

d



Cleanroom advocates the use of sequence-based

specifications. These are better suited to problems

with a high degree of logical interactions Not

with a high degree of logical interactions. Not

suitable for black boxes used in numerical or highly

computational applications.



Non-functional requirements (real-time, security

constraints) and a significant portion of algorithmic

requirements are hard to be represented by Box

structure.



After requirements changes, rework of the

box-structure is a time-consuming process.



Statistical test data may be hard to be collected.

Conclusions

Cl

h i

i

h



Cleanroom approach is a rigorous approach

to software engineering that has emphasis on:



Formal specification



Mathematical verification of correctness of

design



Certification of software reliability



Cleanroom approach is yet to become a

common practice in software development

industry because of emphasis on the above

three points

References

Li R d T l C (1996) Cl S ft

 Linger, R. and Trammel, C. (1996). Cleanroom Software Engineering Reference Model Version 1.0.

http://www sei cmu edu/pub/documents/96 reports/pdf/tr022 96 pdf http://www.sei.cmu.edu/pub/documents/96.reports/pdf/tr022.96.pdf  Wolack, C. (2001). Taking The Art Out of Software

Development – An In-Depth Review of Cleanroom Software Engineering.

http://www.scisstudyguides.addr.com/papers/cwdiss725paper1.htm  Pressmen and Associates (2000) Cleanroom Engineering  Pressmen and Associates (2000). Cleanroom Engineering

Resources.

http://www.rspa.com/spi/cleanroom.html

One Last Advice



Want to impress your customers:

use failure

intensity + reliability growth methodology!

y

y g

gy



Want to impress your boss (development):

p

y

(

p

)

use failure density + zero time failure

methodology!



Want to impress yourself:

use target failure



Want to impress yourself:

use target failure

intensity + reliability demonstration chart!

That is all folks!