Summary of Summary of Test Case Design Techniques
Brian Nielsen, Arne Skou
{bnielsen | ask}@cs.auc.dk
Development of Test Cases
Complete testing is impossible
ologi
Testing cannot guarantee the absence of faults
tekn o
How to select subset of test cases from all possible test cases
tions t
How to select subset of test cases from all possible test caseswith a high chance of detecting most faults ?
orma t
Test Case Design Strategies
Inf o
CISS
Whitebox Testing Overview
White box testing
ologi
White box testing
Flowgraphs
Test criteria/coverage
St t t / b h / d i i / diti / th
tekn o
Statement / branch / decision / condition / path coverage MC/DC coverage
Looptesting
tions t
Def-use pairs
Efficiency of different criteria
orma t
BEWARE: Expected outcome cannot be determined from
Inf o
code!CISS
White-Box : Statement Testing
Execute every statement of a program
Relatively weak criterion
ologi
Relatively weak criterion
Weakest white-box criterion
tekn o tions t orma t Inf o
CISS
Example : Statement Testing ((
result = 0+1+…+|value|, if this <= maxint, error otherwise)ologi
1 PROGRAM maxsum ( maxint, value : INT ) 2 INT result := 0 ; i := 0 ;
tekn o
3 IF value < 04 THEN value := - value ;
5 WHILE ( i < value ) AND ( result < maxint )
tions t
5 WHILE ( i < value ) AND ( result <= maxint )6 DO i := i + 1 ;
7 result := result + i ;
orma t
7 result := result + i ;8 OD;
9 IF result <= maxint
Inf o
9 IF result maxint10 THEN OUTPUT ( result )
11 ELSE OUTPUT ( “too large” )
CISS
12 END.
Flowgrah
12
Node = statement (blocks)
Edges = possible successor (control flow)
ologi
1 PROGRAM maxsum ( maxint, value : INT ) 2 INT result := 0 ; i := 0 ;
3 IF l 0
3 4
tekn o
3 IF value < 04 THEN value := - value ;
5 WHILE ( i < value ) AND ( result <= maxint )
6 DO i i + 1
5 6-7
tions t
6 DO i := i + 1 ;7 result := result + i ;
8 OD;
9 IF result <= maxint
orma t
9 IF result <= maxint10 THEN OUTPUT ( result )
11 ELSE OUTPUT ( “too large” )
12 END
9
Inf o
12 END.11 10
CISS
12
Flow graph: Cyclomatic complexity
#edges - #nodes + 2
Defines the maximal number of test cases needed to provide statement coverage
ologi
provide statement coverage
Mostly applicable for Unit testing
Strategy for statement coverage:
tekn o
gy g1. Derive flow graph
2. Find cyclomatic complexity #c
3 Determine at most #c independent paths through the
tions t
3. Determine at most #c independent paths through theprogram (add one new edge for each test case)
4. Prepare test cases covering the edges for each path
( ibl f th # )
orma t
(possibly fewer than #c cases)Inf o
CISS
Cyclomatic complexity?
1
y p y
1 PROGRAM maxsum ( maxint, value : INT )
2 INT lt 0 i 0
2
ologi
2 INT result := 0 ; i := 0 ;
3 IF value < 0
4 THEN value := - value ;
5 WHILE ( i < l ) AND ( lt < i t )
3 4
tekn o
5 WHILE ( i < value ) AND ( result <= maxint )6 DO i := i + 1 ;
7 result := result + i ;
8 OD;
5 6-7
tions t
8 OD;9 IF result <= maxint
10 THEN OUTPUT ( result )
11 ELSE OUTPUT ( “too large” )
orma t
11 ELSE OUTPUT ( too large )12 END.
9
Inf o
11 10
CISS
12
Example : Statement Testing
start
yes
ologi
value:= -value;
value < 0 yes no
tekn o
Tests for complete statement coverage:
i:=i+1;
result:=result+i;
tions t
gmaxint value
(i<value) and
(result<=maxint) yes
orma t
10 -1
0 -1
no
yes no
Inf o
result<=maxintoutput(result); output(“too large”);
yes no
CISS
output(result); output( too large );
exit
White-Box : Path Testing
Execute every possible path of a program, i e every possible sequence of statements
ologi
i.e., every possible sequence of statements
Strongest white-box criterion
tekn o
Usually impossible: infinitely many paths ( in case of loops ) So: not a realistic option
tions t
But note : enormous reduction w.r.t. all possible test cases ( each sequence of statements executed for only one value )
orma t
( eac seque ce o s a e e s e ecu ed o o y o e a ue )Inf o
CISS
White-Box : Branch Testing
Branch testing == decision testing
ologi
Execute every branch of a program :
each possible outcome of each decision occurs at least once
tekn o
Example:
IF b THEN s1 ELSE s2
tions t
IF b THEN s1; s2
CASE x OF
orma t
1 : ….
2 : ….
3
Inf o
3 : ….CISS
Example : Branch Testing
Tests for complete
start
yes
ologi
statement coverage:
maxint value
10 1
value < 0 yes value:= -value;
no
value < 0
tekn o
10 -10 -1
is not sufficient for
i:=i+1;
result:=result+i;
tions t
is not sufficient forbranch coverage;
(i<value) and
(result<=maxint) yes
T k
orma t
no
yes no
Take:
maxint value
10 3
Inf o
result<=maxintoutput(result); output(“too large”);
yes no 10 3
0 -1
for complete
CISS
output(result); output( too large );
exit
branch coverage
Example : Branch Testing
l l
l < 0
start
yes maxint value
ologi
value:= -value;
value < 0 yes no
maxint value
-1-1 -1-1
10 3
-1 -1
tekn o
i:=i+1;
result:=result+i;
10 3
But:
10 3
tions t
(i<value) and (result<=maxint)
no
yes But:
No green path !
orma t
lt< i t no
yes no Needed :
Inf o
result<=maxintoutput(result); output(“too large”);
yes o
Combination of decisions
10 -3
CISS
p ( ); p ( g );
exit
White-Box : Condition Testing
Design test cases such that each possible outcome f
ologi
of each condition in each decision occurs at least once
Example:
tekn o
decision ( i < value ) AND (result <= maxint )
consists of two conditions : ( i < value ) AND (result <= maxint )
t t h ld b d i d h th t h t l
tions t
test cases should be designed such that each gets valuetrue and false at least once
orma t Inf o
CISS
Example : Condition Testing
start
yes
ologi
( i = result = 0 ) :
maxint value i<value result<=maxint value:= -value;
value < 0 yes no
tekn o
-1 1 true false 1 0 false true
i diti
i:=i+1;
result:=result+i;
tions t
gives condition coveragefor all conditions
(i<value) and
(result<=maxint) yes
But it does not preserve
orma t
no
yes no
But it does not preserve decision coverage
Inf o
result<=maxintoutput(result); output(“too large”);
yes no
always take care that condition coverage
preserves decision coverage :
CISS
output(result); output( too large );
exit
preserves decision coverage : decision / condition coverage
White-Box : Multiple Condition Testing
Design test cases for each combination of conditions
Testing
ologi
Example:
( i < value ) (result <= maxint )
tekn o
( i value ) (result maxint )false false
false true
t f l
tions t
true falsetrue true
Implies decision- condition- decision/condition coverage
orma t
Implies decision-, condition-, decision/condition coverage But : exponential blow-up
Again : some combinations may be infeasible
Inf o
Again : some combinations may be infeasibleCISS
White-box: loop testing
Statement and branch coverage are not sufficient
Single loop strategy:
Zero iterations
ologi
Zero iterations
One iteration
Two iterations
l b f
tekn o
Typical number of iterations n-1, n, and n+1 iterations (n maximum number of allowable iterations)
d l
tions t
Nested loop strategy:
Single loop strategy often intractable
Select minimum values for outer loop(s)
orma t
Treat inner loop as a single loop
Work ‘outwards’ and choose typical values for inner loops
Concatenated loops:
Inf o
Concatenated loops: Treat as single, if independent
Treat as nested, if dependent
CISS
Example : Loop testing
start
yes
Tests for complete loop coverage:
ologi
value < 0 yes value:= -value;
no maxint value
15 0
tekn o
i:=i+1;
result:=result+i;
15 0 15 1 15 2
tions t
(i<value) and
(result<=maxint) yes 15 3
6 4
orma t
no
yes no
15 5
Inf o
result<=maxintoutput(result); output(“too large”);
yes no
CISS
output(result); output( too large );
exit
White-box testing: Data Flow Criteria
Basic idea: For each variable definition (assignment), find a path (and a corresponding test case), to its use(s). A
ologi
a path (and a corresponding test case), to its use(s). A pair (definition,use) is often called a DU pair.
Three dominant strategies:
ll d f ( ) f ll l h f h
tekn o
All-defs (AD) strategy: follow at least one path from each definition to some use of it All-uses (AU) strategy: follow at least one path for each DU
tions t
pair
All-du-uses strategy (ADUP): follow all paths between a DU pair
orma t
Complements the testing power of decision coverage
Inf o
CISS
Example: All-uses coverage
1 Dm,v1 PROGRAM maxsum ( maxint, value : INT ) 2 INT result := 0 ; i := 0 ;
3 IF value < 0 2 Dr,i
ologi
4 THEN value := - value ;
5 WHILE ( i < value ) AND ( result <= maxint )
6 DO i := i + 1 ; 3 4
Uv Uv;Dv
tekn o
7 result := result + i ;
8 OD;
9 IF result <= maxint
10 THEN OUTPUT ( lt ) Ui v r m 5 6-7
Ur,i;Dr,i
tions t
10 THEN OUTPUT ( result )
11 ELSE OUTPUT ( “too large” )
12 END.
5 6
i,v,r,m
orma t
Ur,m 9
Def-use pairs:
1 3 1 5 1 9 1 4
Tests for complete all-uses coverage:
i t l
Inf o
11 10
Ur 1-3,1-5,1-9,1-4
2-5,2-9,2-6 4-5
maxint value
0 0
0 -1
CISS
12
6-5,6-9,6-11 6-5-6
10 1 10 2
White-Box : Overview
statement coverage
ologi
condition coverage
decision (branch) coverage
tekn o
decision/
condition coverage
tions t
multiple- condition
orma t
conditioncoverage
Inf o
pathcoverage
CISS
White-Box : Overview
statement coverage
f decision
ologi
all defs coverage
decision (branch) coverage
tekn o
all uses coverage
tions t
all du paths coverage
orma t
path coverage
Inf o
pathcoverageCISS
Additional techniques: mutation and random testing
and random testing
Mutation testing:
Intended for evaluating the test cases
ologi
Intended for evaluating the test cases
Create at set of slightly modified mutants of the original program containing errors
R th t t i t th t t
tekn o
Run the test cases against the mutants Criteria
All mutants must fail (strong)
tions t
All mutants will eventually fail (weak)
Random testing:
Basic idea: run the program with arbitrary inputs
orma t
Basic idea: run the program with arbitrary inputs Inherent problems: How to define the oracle for arbitrary inputs and how to decide to stop?
Inf o
Advantage: The program structure can be ignoredCISS
Efficiency of white box techniques:
Efficiency of white-box techniques:
two studies
ologi
Strategy #test cases %bugs found
Random 35 93 7
tekn o Random 35 93.7
Branch 3.8 91.6
All 11 3 96 3
tions t
All-uses 11.3 96.3
orma t
Random 100 79.5
Inf o
Branch 34 85.5
All-uses 84 90 0
CISS
All-uses 84 90.0
What is MC/DC?
MC/DC stands for Modified
ologi
Condition / Decision Coverage
Main idea: Each condition must be h t i d d tl ff t th tekn o shown to independently affect the
outcome of a decision, i.e. the
outcome of a decision changes as a
tions t outcome of a decision changes as a
result of changing a single condition.
orma t Inf o
CISS
1. MC/DC Requirement
The decision has taken all possible outcomes at least once.
ologi If ((a) & (b) | (c)) then
tekn o If ((a) & (b) | (c)) then…
tions t
true false
orma t
We could also say we cover both the true d h f l b h (lik d i
Inf o and the false branch (like we do in Branch Testing).
CISS
2. MC/DC Requirement
Every condition in the decision has taken all possible outcomes at least once.
ologi
p
If ((a) & (b) | (c)) then…
tekn o If ((a) & (b) | (c)) then…
true false true false true false
tions t
Aims to cover compound conditions (like
orma t p (
Condition/Decision Coverage)
Inf o
CISS
3. MC/DC Requirement
Every condition in the decision
i d d tl ff t th d i i ’
ologi
independently affects the decision’s outcome.
tekn o
If ((a) & (b) | (c)) then…
tions t (( ) ( ) | ( ))
true false true false
orma t
Change the value of each condition individually while keeping all other
conditions constant.
Inf o
CISS
Creating MC/DC test cases
If (A and B) then…
ologi
(1) create truth table for conditions
tekn o
(1) create truth table for conditions.
(2) Extend truth table so that it
indicated which test cases can be used
tions t indicated which test cases can be used
to show the independence of each condition.
orma t Inf o
CISS
Creating MC/DC test cases cont’d
ologi
Show independence of A:
Take 1 + 3
tekn o
Take 1 + 3
Show independence of B:
tions t B:
Take 1 + 2
Resulting test cases
orma t g
are
1 + 2 + 3
(T T) (T F) (F T)
Inf o
(T , T) + (T , F) + (F , T)CISS
More advanced example
If (A and (B or C)) then…
ologi tekn o
Note: We want to determine the MINIMAL set of test cases H
tions t
Here:{2,3,4,6}
{2,3,4,7}
orma t
Non-minimal set is:
{1,2,3,4,5}
Inf o
CISS
Argument in the software industry
Federal Aviation Administration’s
requirement that test suites be MC/DC
ologi
q /
adequate (DO-178B).
tekn o
In the avionics domain, complex Boolean expressions are much more common
tions t expressions are much more common.
orma t Inf o
CISS
Argument in the software industry
Federal Aviation Administration’s requirement that test suites be MC/DC adequate.
Argument – Too expensive
ologi
Argument – Too expensive
“For example, one of our industrial partners reports that for one of its products of about 20,000 lines of code the MC/DC-adequate test suite requires seven
tekn o
code, the MC/DC-adequate test suite requires seven weeks to run.”
Counter Argument – Significantly more effective
A t i i l t d f d d i th l
tions t
A recent empirical study performed during the realtesting of the attitude-control software for the HETE-2 (High Energy Transient Explorer) found that the test cases generated to satisfy the MC/DC coverage
orma t
cases generated to satisfy the MC/DC coveragerequirement detected important errors not detectable by functional testing.
Inf o
CISS
Blackbox Testing Overview
Black-box testing (or functional testing):
ologi
Black box testing (or functional testing):
Equivalence partitioning
Boundary value analysis Domain analysis
tekn o
y y Cause-effect graphing
Behavioural testing / state transition testing
y
tions t
Random testing, Error guessing etc…
Others: Classification Tree Method, Pairwise testing, Use Case
orma t
Testing, Syntax testing, ..
Basics : heuristics and experience
Inf o
How to use black-box and white-box testing in combination
CISS
Black box testing
requirements
ologi
outputtekn o
input
SUT
tions t
put
events
y domain testing
orma t Inf o
x
CISS
x
Black-box: Equivalence Partitioning
ologi
Strategy :
Identify input equivalence classes
tekn o
y p q Based on conditions on inputs / outputs in specification / description
Both valid and invalid input equivalence classes
tions t
Both valid and invalid input equivalence classes Based on heuristics and experience
“input x in [1..10]” classes : x 1, 1 x 10, x 10
orma t
“enumeration A, B, C classes : A, B, C, not{A,B,C,}
……..
Define one / couple of test cases for each class
Inf o
Define one / couple of test cases for each class Test cases that cover valid eq. classes
Test cases that cover at most one invalid eq class
CISS
Test cases that cover at most one invalid eq. class
Example: Equivalence Partitioning
ologi
Test a function for calculation of absolute value of an integer
Equivalence classes :
tekn o
Equivalence classes :Condition Valid eq. classes Invalid eq. Classes
nr of inputs 1 0 > 1
tions t
nr of inputs 1 0, > 1Input type integer non-integer
particular abs < 0, >= 0
orma t
Test inputs :
Inf o
• x = -10, x = 100• x = “XYZ”, x = - x = 10 20
CISS
A Self-Assessment Test [Myers]
“A program reads three integer values. The three values are
interpreted as representing the lengths of the sides of a triangle.
ologi
The program prints a message that states whether the triangle is scalene (uligesidet), isosceles (ligebenet) , or equilateral
(ligesidet) ”
tekn o
(ligesidet).tions t orma t
Write a set of test cases to test this program
Inf o Write a set of test cases to test this program.
CISS
A Self-Assessment Test [Myers]
Test cases for:
ologi
1. valid scalene triangle ? 2. valid equilateral triangle ?
3 lid i l t i l ?
9. one side larger than sum of others ?
10 3 permutations of previous ?
tekn o
3. valid isosceles triangle ?4. 3 permutations of previous ? 5 side = 0 ?
10. 3 permutations of previous ? 11. all sides = 0 ?
12. non-integer input ?
tions t
5. side = 0 ?6. negative side ?
7. one side is sum of others ?
12. non integer input ?
13. wrong number of values ? 14. for each test case: is
orma t
7. one side is sum of others ?8. 3 permutations of previous ? expected output specified ? 15. check behaviour after
t t d d ?
Inf o
output was produced ?CISS
Example: Equivalence Partitioning
Test a program that computes the sum of the first value integers
l thi i l th i t Oth i
ologi
as long as this sum is less than maxint. Otherwise an error should be reported. If value is negative, then it takes the
tekn o
absolute value
Formally:
tions t
yGiven integer inputs maxint and value compute result :
orma t
result = |value|
k if this <= maxint, error otherwise
Inf o
K=0
CISS
Example: Equivalence Partitioning
Equivalence classes :
Condition Valid eq. classes Invalid eq. classes
N f i t 2 2 2
ologi
Nr of inputs 2 < 2, > 2
Type of input int int int no-int, no-int int
Abs(value) value 0, value 0
tekn o
maxint k maxint, k > maxint
Test Cases : maxint value result
tions t
Test Cases : maxint value resultValid 100 10 55
100 -10 55
10 10
orma t
10 10 errorInvalid 10 - error
10 20 30 error
Inf o
“XYZ” 10 error100 9.1E4 error
CISS
Black-box: Boundary Value Analysis
ologi
Based on experience / heuristics :
Testing boundary conditions of eq. classes is more effective
tekn o
i.e. values directly on, above, and beneath edges of eq. classes
Choose input boundary values as tests in input eq. classes
tions t
instead of, or additional to arbitrary values
Choose also inputs that invoke output boundary values ( values on the boundary of output classes )
orma t
( values on the boundary of output classes ) Example strategy as extension of equivalence partitioning:
choose one (n) arbitrary value in each eq class
Inf o
choose one (n) arbitrary value in each eq. class choose values exactly on lower and upper boundaries of eq. class
choose values immediately below and above each boundary ( if applicable )
CISS
( if applicable )
Example: Boundary Value Analysis
ologi
Test a function for calculation of absolute value of an integer
Valid equivalence classes :
tekn o
Valid equivalence classes :Condition Valid eq. classes Invalid eq. Classes particular abs < 0, >= 0
tions t
particular abs 0, 0• Test cases :
orma t
class x < 0, arbitrary value: x = -10 class x >= 0, arbitrary value x = 100
Inf o
classes x < 0, x >= 0, on boundary : x = 0classes x < 0, x >= 0, below and above: x = -1, x = 1
CISS
Example: Boundary Value Analysis
Given integer inputs maxint and value compute result :
|value|
ologi
result = if this <= maxint, error otherwise
|value|k
tekn o
K=0• Valid equivalence classes :
tions t
Condition Valid eq. Classes Abs(value) value 0, value 0
orma t
maxint k maxint, k > maxint
Sh ld l di ti i h b t i t 0 d i t 0 ?
Inf o
• Should we also distinguish between maxint < 0 and maxint >= 0 ?maxint maxint < 0, 0 maxint < k, maxint k
CISS
Example: Boundary Value Analysis
ologi
Valid equivalence classes :
Abs(value) value 0, value 0
tekn o
maxint maxint < 0, 0 maxint < k, maxint k Test Cases :
tions t
Cmaxint value result maxint value result
55 10 55 100 0 0
orma t
55 10 55 100 0 054 10 error 100 -1 1
56 10 55 100 1 1
0 0 0
Inf o
0 0 0 …. …. …-10 1 error
How to combine the boundary conditions of different inputs ?
CISS
Take all possible boundary combinations ? This may blow-up.
Black-box: Cause Effect Graphing
ologi
Black-box testing technique to analyse combinations of input conditions
tekn o
of input conditions Identify causes and effects in specification
tions t
inputs outputs
current state new state
orma t
Make Boolean Graph linking causes and effects
Inf o
Annotate impossible combinations of causes and effects Develop decision table from graph with in each column a particular combination of inputs and outputs
CISS
a particular combination of inputs and outputs
Transform each column into test case
Black-Box: Cause Effect Graphing
k maxint
k i t k
d xor
ologi
k maxint value 0
error
and
xor and
tekn o
C
value 0
xor
tions t
Causes k maxint 1 1 0 0inputs k maxint 0 0 1 1
value 0 1 0 1 0
orma t
value 0 1 0 1 0value 0 0 1 0 1
Inf o
Effects k 1 1 0 0
outputs error 0 0 1 1
CISS
Testcase
Black-box: Cause Effect Graphing
Systematic method for generating test cases representing combinations of conditions
ologi
combinations of conditions
Combinatorial explosion of number of possible combinations
S h i ti t d thi bi t i l l i
tekn o
Some heuristics to reduce this combinatorial explosion Starting point is effects (outputs) then working ‘backwards’
tions t
‘light-weight’ formal methods:
transformation into semi-formal Boolean graph
orma t
A technique : to be combined with others
Inf o
CISS
Black-box: Error Guessing
Just ‘guess’ where the errors are ……
Intuition and experience of tester
ologi
Intuition and experience of tester
Ad hoc, not really a technique
tekn o
Strategy:
Make a list of possible errors or error-prone situations
tions t
( often related to boundary conditions )
Write test cases based on this list
orma t Inf o
CISS
Black-box : Error Guessing
More sophisticated ‘error guessing’ :
Risk Analysis
Try to identify critical parts of program (high risk code sections):
ologi
Try to identify critical parts of program (high risk code sections):
parts with unclear specifications
tekn o
developed by junior programmer while his wife was pregnant …… complex code :
tions t
measure code complexity - tools available (McGabe, Logiscope,…)
High-risk code will be more thoroughly tested
orma t
( or be rewritten immediately ….)
Inf o
CISS
Black-Box Testing: Which One ?
Black-box testing techniques :
Equivalence partitioning
ologi
Boundary value analysis
Cause-effect graphing
tekn o
Error guessing ………
Whi h t ?
tions t
Which one to use ? None is complete
All are based on some kind of heuristics
orma t
All are based on some kind of heuristics They are complementary
Inf o
CISS
Black-Box Testing: Which One ?
Always use a combination of techniques
ologi
When a formal specification is available try to use it
Identify valid and invalid input equivalence classes
tekn o
Identify output equivalence classes
Apply boundary value analysis on valid equivalence classes
tions t
Guess about possible errors
Cause-effect graphing for linking inputs and outputs
orma t Inf o
CISS
White-Box testing : How to Apply ?
Don’t start with designing white-box test cases !
ologi
Start with black-box test cases (because requirements must be tested anyway)
tekn o
equivalence partitioning, boundary value analysis,cause effect graphing, test derivation with formal methods, …..
Check white box coverage
tions t
Check white-box coverage( statement-, branch-, condition-, ….. coverage )
U t l b bi d ith U it f k
orma t
Use a coverage tool – maybe combined with a Unit framework Design additional white-box test cases for not covered code
Inf o
CISS
Blackbox vs. Whitebox
ologi I II III
tekn o tions t orma t
Specified Behavior Implemented Behavior
Inf o (Blackbox)
(Whitebox)
I: Specified, not implemented II: Implemented and specified
CISS
II: Implemented and specified
III:Implemented, but not specified
A Coverage Tool : gcov
Standard Gnu tool gcov
O l t t t b t b li d ll f
ologi
Only statement coverage, but may be applied manually for multiple condition coverage
tekn o
Compile your program under test with a special option
Run a number of test cases
tions t
A listing indicates how often each statement was executed and percentage of statements executed
orma t
and percentage of statements executedInf o
CISS
A Coverage Tool : BullseyeCoverage
Supports decision/condition
ologi
Compile your program under test with a special option
tekn o option
Run a number of test cases
tions t