Software Reliability Estimation Based on Static Error Detection

(1)

7 th

_{Central and Eastern European}

Software Engineering Conference

in Russia - CEE-SECR 2011

October 31 – November 3, Moscow

M. Moiseev, M. Glukhikh, A. Karpenko,

H. Richter

Software Reliability Estimation

Based on Static Error Detection

(2)

Importance of

Software Reliability Analysis

Modern software

contains errors

Errors can lead

to disasters

Software

Reliability Analysis

Error detection

should be

organized

(3)

Known Approaches

Heuristics

approaches

Dynamic

approach

Architecture-based

approach

Program

metrics

Development

process

(4)

Known Approaches –

Program Metrics

Based on simple code properties, such as

number of statements

number of conditions

number of loops

number of functions

...

(5)

Known Approaches – Development

Process Metrics

Based on development process properties, such as

duration of development

number & qualification of developers

number & qualification of testers

methodology used

(6)

Known Approaches –

Others

Runtime

Based on failures observed at run-time

Architecture-based

(7)

Our Approach

Based on source code static analysis

Delivers

Ranking of errors (based on failure probability)

Reliability characteristics

Limitations

Single-threaded C programs

Error types

uninitialized variable use

(8)

Features of Our Approach

Analysis of a program model

Analysis of all possible execution paths

Advantages

Reliability estimations is based on real errors

Results are applicable for any exploitation conditions

Makes debugging more effective

Drawbacks

Does not consider quantitative time

Does not consider normal program exploitation

Execution path probability estimation

(9)

Program Classes

Programs

(10)

Reliability characteristics used

Computational programs

Probability of whole program

successful execution P(

_∞

)

Server programs

Probability of n statements

successful execution P(n)

Mean executed statement

(11)

Algorithms

Model building

State determination

Error detection

Error ranking

Reliability

estimation

(12)

Program Model Features

Control flow graph

Three-operand assignment form A = B op C

If and Phi statements

If

Phi

(13)

State Determination Algorithms

State representation

Control flow analysis

Statement analysis

Sequential

If statement analysis

Phi statement analysis

Loop analysis

(14)

Program State Representation

Based on objects, values, and probabilities

set of triples

state probability

Object values

intervals

pointers

resource descriptors

(

)

{

o

j

v

k

p

jk

}

Q

=

,

( )

Q

P

(15)

Probability normalization

Control flow normalization

State normalization

( )

∑

( )

∑

∈

∀

=

s

Output

Q

out

j

s

Input

Q

in

j

out j in j

Q

P

Q

P

(

)

(

p

)

P

( )

Q

p

v

o

Q

p

v

o

jk

k

j

jk k j

=

⇒

∈

∃

∀

∑

∈

∀

,

:

(16)

Sequential Statement Analysis

a = b + c

( )

(

)

( )

(

)













=

,...

1 ,

6 ..

3 ,

,

1 ,

2 ..

1 ,

c

b

Q

in

( )

(

)

{

a

,

4 ..

8 ,

1 ,...

}

Q

out

=





























































































=

,...

4

1 ,

6 ,

,

4

1 ,

5 ,

,

4

1 ,

4 ,

,

4

1 ,

3 ,

2

1 ,

2 ,

,

2

1 ,

c

b

















































































=

,...

8

1 ,

8 ,

,

4

1 ,

7 ,

,

4

1 ,

6 ,

,

4

1 ,

5 ,

,

8

1 ,

4 ,

a

(17)

If Statement Analysis

True and false combination consideration

Normalization of state probabilities

Normalization of non-affected triples probabilities

( )

(

)

( )

(

)

.

,

, , , ,

∑

∏

∑

∏

∈ ∈ ∈ ∈

=

false jk k j true jk k j C c o v p c jk false C c o v p c jk true

p

Q

P

p

Q

P

(18)

If Statement Analysis Example

172 combinations where a < b

28 combinations where a >= b

Normalization: 0.86 for true, 0.14 for false

(

)

(

)

(

)

(

)













=

,...

1 ,

23 ..

4 ,

,

1 ,

10 ..

1 ,

b

a

Q

in true

Q

false

Q

(

)

(

)

(

)

(

)













86 .

0 ,

23 ..

4 ,

86 .

0 ,

10 ..

1 ,

b

a

(

)

(

)

(

)













14 .

0 ,

10 ..

4 ,

14 .

0 ,

10 ..

4 ,

b

a

(19)

Phi Statement Analysis

Identical triples are added together

Control flow normalization

(

)

(

)

(

)

out

jk

k

j

in

jk

k

j

in

jk

k

j

k

j

Q

r

p

v

o

Q

r

v

o

Q

p

v

o

v

o

∈

+

⇒

∈

∀

,

:

,

₁

₂

In

₁

In

₂

Phi

( ) ( ) ( )

out in in

Q

P

Q

P

Q

P

=

₁

+

₂

(20)

Error Detection

Based on incorrect values in state

uninitialized variable use

pointer dereference

out of bounds

correct if

otherwise error is detected

(

o

j

,

v

noninit

,

p

k

)

(

o

j

,

v

noninit

,

p

k

)

(

o

j

,

v

invalid

,

p

k

)

(

o

j

,

v

null

,

p

k

)

(

)

(

o

i

,

o

j

,

offset

j

,

p

k

)

( )

j

sizeof

o

offset

<

≤

0

(21)

Error Inhibition

obj use

(obj, valid, p1)

(obj, invalid, p2)

P(Q)=p1+p2

(obj, valid, p1)

P(Q)=p1

(22)

Error Ranking

Errors are sorted according to probability of

occurrence

Most dangerous errors can be corrected first

Probabilities are summarized

(23)

Overall reliability estimation

probability of successful execution

probability of n statements successful execution

mean executed statements number before failure

(

)

∑

−

+

⋅

=

max

(

)

(

1 )

n

P

n

P

n

( )

=

∑

executed

statements

n

Q

P

n

P

(

)

( )

=

∑

statements

end

Q

P

n

P

(

)

(24)

Implementation

AEGIS static analyzer

analysis of C/C++ source code

interval, points to, resource analysis

loop & interprocedural analysis

spread range of program errors detected

Results

error ranking table

P(n) table

P(

_∞

)

(25)

Experiments made

Purpose

Testing of our approach

Debugging example

Test

programs

Students'

projects

Real-world projects

(embedded software)

(26)

Sample of reliability analysis

while (!(feof(f))) // 0.5

{

i = t = 0;

// Failure in one of three cases

prov(&t, strlen(st), st);

}

Probability of successful execution is

0.75 = 0.5 + 0.5 * 0.33 + 0.5 * 0.33

2 _{+ ...}

(27)

Amount of errors in real-world projects

0 20 40 60 80 100 Er ro r n u m b e r

More than 500 errors, 2/3 of considered types

Density about 0.8/1KLOC

(28)

Distribution of error number

0

25

50

75

100

125

150

175 1.E-06 1.E-05 1.E-04 1.E-03 1.E-02 1.E-01

0,25

0,5

1,0

Error probability

Er

ro

r

n

u

m

b

e

r

(29)

Debugging results

0.00 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90 1.00 32 33 34 35 36 37 38 39 40 41 Original Corrected

(30)

Directions for Future Work

Reliability estimation

Annotations for path probability estimations

Run-time analysis for path probability estimation

Execution time estimation

Static analysis itself

Soundness & precision

Parallel program analysis

(31)

Conclusion

Approach for software reliability estimation

based on error detection using static analysis

Implementation in AEGIS tool (prototype)

ranking of errors by the probability of occurrence

probability of successful execution

probability of N statement successful execution

(32)

Contacts

Saint Petersburg State Polytechnical University

Digitek Labs

http://digiteklabs.ru

Mikhail Glukhikh, Mikhail Moiseev,

Anatoly Karpenko

E-mail:

[email protected]

E-mail:

[email protected]

E-mail:

[email protected]

Clausthal University of Technology

Harald Richter