How To Understand The Rules Of The Game Of Chess

tugraz

Quality Assurance in Software Development

Qualitätssicherung in der Softwareentwicklung

A.o.Univ.-Prof. Dipl.-Ing. Dr. Bernhard Aichernig

Institute for Software Technology

Graz University of Technology

Austria

tugraz

Agenda

1

Conformance Testing

Properties

2

Labelled Transition Systems

Equivalence

Preorder

ioco

Examples

tugraz

Input/Output Conformance Relation

I

Jan Tretmans - 1996

tugraz

Input/Output Conformance Relation

I

Jan Tretmans - 1996

tugraz

tugraz

Conformance Testing - Soundness

I

Test suite is sound: conformance

⇒

all tests pass

tugraz

Conformance Testing - Soundness

I

Test suite is sound: conformance

tugraz

Conformance Testing - Exhaustiveness

I

Test suite is exhaustive:

conformance

⇐

all tests pass

tugraz

Conformance Testing - Exhaustiveness

I

Test suite is exhaustive:

tugraz

Conformance Testing - Completeness

I

Test suite is complete:

conformance

≡

all tests pass

tugraz

Conformance Testing - Completeness

I

Test suite is complete:

tugraz

Conformance Testing with ioco

I

System (Implementation) is modeled as IOTS

I

weakly input enabled

I

Specification is an IOLTS

possibly incomplete

possible non-deterministic

!g1

τ

τ

!g1

?g2

?g2

!g3

!g3

!g1

tugraz

Conformance Testing with ioco

I

System (Implementation) is modeled as IOTS

I

weakly input enabled

I

Specification is an IOLTS

possibly incomplete

possible non-deterministic

!g1

τ

τ

!g1

?g2

?g2

!g3

!g3

!g1

tugraz

Input Output Labeled Transition Systems

Input Output Labeled Transition System

An IOLTS is an LTS

M

= (

Q

M

,

A

M

,

→

M

,

q

0

M

)

with

I

Q

M

a finite set of states

I

A

M

=

A

M

I

∪

A

M

O

∪ {

τ

}

where

I

A

and

A

O

are input and output alphabets

I

τ

6∈

A

I

∪

A

is an unobservable, internal action

I

→

M

⊆

Q

M

×

A

M

×

Q

M

is the transition relation

I

q

M

0

∈

Q

M

is the initial state.

QUESTION?

tugraz

Input Output Labeled Transition Systems

Input Output Labeled Transition System

An IOLTS is an LTS

M

= (

Q

M

,

A

M

,

→

M

,

q

0

M

)

with

I

Q

M

a finite set of states

I

A

M

=

A

M

I

∪

A

M

O

∪ {

τ

}

where

I

A

and

A

O

are input and output alphabets

I

τ

6∈

A

I

∪

A

is an unobservable, internal action

I

→

M

⊆

Q

M

×

A

M

×

Q

M

is the transition relation

I

q

M

0

∈

Q

M

is the initial state.

QUESTION?

tugraz

How to relate 2 LTSs?

I

Equivalence Relations (=)

Bisimulation

Trace Equivalence

Testing Equivalence

...

I

Preorder Relations (

≤

)

Trace Preorder

Testing Preorder

...

I

Input-Output Relations

ioconf

ioco

...

I

...

tugraz

(Weak) Bisimulation

I

Two states are bisimilar iff they simulate each other and go to

states which are bisimilar

tugraz

Trace Equivalence

I

A trace is an observable sequence of actions

I

Two states are trace equivalent iff they have the same traces

tugraz

Equivalence vs. Preorder Relations

I

Equivalence Relation (

R

)

I

reflexive (

s

R

s

)

I

symmetric:

i

R

s

→

s

R

i

transitive:

i

R

s

∧

s

R

t

→

i

R

t

I

Preorder Relations (

≤

)

I

NOT

necessarily antisymmetric:

I

i

R

s

↔

i

≤

s

∧

s

≤

i

simplifies testing

I

e.g.: Trace Preorder

tugraz

Some Notations: Transitions

I

q

→

a

M

q

0

=

(

q

,

a

,

q

0

)

∈→

M

I

q

⇒

q

0

=

(

q

=

q

0

)

∨

(

q

τ

→

M

q

1

∧ · · · ∧

qn

−

1

τ

→

M

q

0

)

I

q

⇒

a

q

0

=

∃

q

1

,

q

2

:

q

⇒

M

q

1

a

→

M

q

2

⇒

M

q

0

tugraz

Some Notations: Quiescence

I

δ

is used to represent quiescence

I

q

−

→

q

=

q

is a quiescent state.

I

Quiescent state = no edge labeled with an output or an internal

tugraz

Some Notations: Quiescence

I

δ

is used to represent quiescence

I

q

−

→

q

=

q

is a quiescent state.

I

Quiescent state = no edge labeled with an output or an internal

tugraz

Some Notations: Suspension Automaton

I

∆(

M

) = (

Q

M

,

A

∆(

M

)

,

→

∆(

M

)

,

q

0

M

)

where:

I

A

=

A

∪ {

δ

}

with

δ

∈

A

I

→∆(

is obtained from

→

by adding loops

q

δ

→

q

for each

quiescent state

tugraz

Some Notations: After

I

q

after

M

σ

=

{

q

0

|

q

σ

⇒

M

q

0

}

I

Q

after

M

σ

=

S

q

∈

Q

(

q

after

M

σ

)

.

tugraz

Some Notations: Out

I

OutM

(

q

) =

{

a

∈

A

M

O

|

q

a

→

M

}

I

Out

M

(

Q

) =

S

q

∈

Q

(

Out

M

(

q

))

tugraz

ioco

Definition: ioco

Let

IUT

= (

Q

IUT

,

A

IUT

,

→

IUT

,

q

IUT

0

)

be weakly input enabled with

A

IUT

=

A

IUT

I

∪

A

IUT

O

∪ {

τ

}

and

S

=

Q

S

,

A

S

,

→

S

,

q

0

S

be strongly

responsive with

A

S

=

A

S

I

∪

A

S

O

∪ {

τ

}

. Then:

IUT

ioco

S

=

∀

σ

∈

traces

(∆(

S

)) :

Out

IUT

(∆(

IUT

)

after

IUT

σ

)

⊆

Out

S

(∆(

S

)

after

S

σ

)

.

I

IUT ioco S iff outputs (and quiescences) of the IUT are possible in

S after an arbitrary suspension trace of S.

tugraz

ioco

Definition: ioco

Let

IUT

= (

Q

IUT

,

A

IUT

,

→

IUT

,

q

IUT

0

)

be weakly input enabled with

A

IUT

=

A

IUT

I

∪

A

IUT

O

∪ {

τ

}

and

S

=

Q

S

,

A

S

,

→

S

,

q

0

S

be strongly

responsive with

A

S

=

A

S

I

∪

A

S

O

∪ {

τ

}

. Then:

IUT

ioco

S

=

∀

σ

∈

traces

(∆(

S

)) :

Out

IUT

(∆(

IUT

)

after

IUT

σ

)

⊆

Out

S

(∆(

S

)

after

S

σ

)

.

I

IUT ioco S iff outputs (and quiescences) of the IUT are possible in

tugraz

P ioco S?

P

ioco

S

=

∀

σ

∈

traces

(∆(

S

)) :

tugraz

P ioco S?

P

ioco

S

=

∀

σ

∈

traces

(∆(

S

)) :

tugraz

P ioco S?

P

ioco

S

=

∀

σ

∈

traces

(∆(

S

)) :

tugraz

P ioco S?

P

ioco

S

=

∀

σ

∈

traces

(∆(

S

)) :

tugraz

P ioco S?

P

ioco

S

=

∀

σ

∈

traces

(∆(

S

)) :

tugraz

P ioco S?

P

ioco

S

=

∀

σ

∈

traces

(∆(

S

)) :

tugraz

P ioco S?

P

ioco

S

=

∀

σ

∈

traces

(∆(

S

)) :

tugraz

P ioco S?

P

ioco

S

=

∀

σ

∈

traces

(∆(

S

)) :

tugraz

P ioco S?

P

ioco

S

=

∀

σ

∈

traces

(∆(

S

)) :

tugraz

P ioco S?

P

ioco

S

=

∀

σ

∈

traces

(∆(

S

)) :

tugraz

P ioco S?

P

ioco

S

=

∀

σ

∈

traces

(∆(

S

)) :

tugraz

P ioco S?

P

ioco

S

=

∀

σ

∈

traces

(∆(

S

)) :

tugraz

Test Cases

I

A test case is an IOLTS

I

Inputs = Outputs IUT, Outputs = Inputs IUT

I

Equipped with verdict states (pass, fail)

I

In each state (except Pass, Fail):

I

Single output and

all inputs

tugraz

tugraz

tugraz

tugraz

tugraz

tugraz

tugraz

tugraz

tugraz

tugraz

tugraz

tugraz

A Complete Test Generation Algorithm

Given the suspension automaton of a specification as an LTS

S

= (

Q

S

,

A

S

,

→

S

,

q

0

S

)

1

Initially compute

K

=

q

S

0

after

S

2

Do non-deterministically, either:

I

Stop test case with verdict

pass

I

Let the test case produce an output (!a) with

K

=

K

after

a

6

=

∅

.

I

Also accept all inputs at the same time and add fail states for

unexpected results.

I

Accept all inputs (and quiescence) and add fail states for

unexpected results. Compute new

K

for valid inputs.

tugraz

A Complete Test Generation Algorithm

δ

tugraz

A Complete Test Generation Algorithm

δ

tugraz

A Complete Test Generation Algorithm

δ

tugraz

A Complete Test Generation Algorithm

δ

tugraz

A Complete Test Generation Algorithm

δ

tugraz

A Complete Test Generation Algorithm

δ

tugraz

A Complete Test Generation Algorithm

δ

tugraz

Tools

I

TGV: offline testing tool

I

jTorX: online testing

I

MoMut: offline model-based mutation testing tool, AIT and TU

Graz

I

SpecExplorer: uses

Alternating Simulation

(equivalent to ioco for

tugraz

References

I

Martin Weiglhofer, Bernhard Aichernig, and Franz Wotawa.

Fault-based conformance testing in practice. International Journal

of Software and Informatics, 3(2-3):375–411, June/September

2009. Copyright by Institute for Software, Chinese Academy of

Science.