2.3 Using Tests for Verification
2.3.2 Test Design Techniques
Since software testing was first introduced, developers have thought about how to create tests systematically and therefore defined so-called test design techniques. Test design techniques allow the structured derivation of test cases by means of formal methods, which can be divided into two domains: black-box testing and white-box testing.
Techniques for Black-box Testing
The aim of black-box testing is to verify whether the test objective satisfies its specifi- cation or not. Therefore only the interface and the requirements of the test objective are taken into consideration for the definition of a test. The test’s result are evaluated
by analysing the output values with regard to the input stimuli and the specification. No internal knowledge about the implementation is used, the developer considers the test objective only from an external perspective – like a black-box [137]. Typical methods for this technique are:
Classification-Tree Method: The classification-tree method divides the input do- main of a test objective into classification and classes and represents them as a tree [140, 141, 142, 143, 144, 145]. A classification influences the functional behaviour of the system in general, whereas a class is a subset whose members result in the same kind of behaviour (equivalence partitioning). Each classifi- cation consists of one or more classes. A test case is generated by combining classes from different classifications.
Figure 2.12 shows an example classification tree for the controller of a colli- sion avoidance system with six example test cases.6 In addition to the textual
descriptions shown, classes can have additional attributes, especially the cor- responding signal or state name and a certain value or a range of values, e.g. [0.6, 0.8] for the value of the road friction at wet conditions. Moreover it is possible to define test sequences, which consists of a number of test cases and allow the transition between them, e.g. the transition between wet and dry friction.
Evolutionary Algorithms: Evolutionary algorithms are based on the evolution the- ory of Darwin. Test cases are generated from a “start population”, which changes through “recombination” and “mutation” in an iterative testing pro- cess [146]. The so-called survival of the fittest is the driving force to find defects of the test objective. With evolutionary algorithms, testing is transformed into an optimization problem, which formulates the test aim as a search for input values that fulfil the respective test aim, e.g. finding defects or verifying re- quirements [147, 148].
State Transition Testing: Different modelling notations were described in chapter 2.1.1, including statecharts. With state transition testing, test cases are derived from the states and transitions of these models, composing a state-event table and a transition tree [50]. While the state-event table consists of all possible and impossible combinations of states and events, the transition tree describes valid paths through this table starting from the initial state. The aim of the technique
6The classes and classifications marked with [..] are left out for simplification, the complete tree is more complex.
Object System Vehicle Environment Road Friction Speed Acceleration Speed Acceleration Distance [..] [..] [..]
None Low Medium High [..] None Low Medium High Dry Wet μLow
Figure 2.12: Example classification tree for a collision avoidance system
is to cover all of these paths and to test for the impossible combinations. State transition testing is only possible, if the test objective is or can be modelled as a statechart. A similar technique is control-flow testing. Here the test cases are derived from a flowchart of the test objective and are created in such a way, that they cover all possible combinations of actions (paths of the diagram) [149, 150].
It should be noted that state transition testing as well as control flow testing can be considered as both black-box and white-box techniques depending on their objective. They are used for black-box testing if the underlying model is the description of the specification of the system, and for white-box test- ing if it serves as the implementation of the system. Thus with model-based development the boundaries between both techniques can become blurred.
Statistical Usage Testing: The purpose of statistical usage testing is the simulation of the real situations of a system to detect defects concerning the operational usage. The probability of the different events is determined (according to the current state and the history of the events or states) to produce a relevant set of test cases [151, 152]. Events, which occur very infrequently (rare), are often relevant for safety aspects of the systems. In contrast to statistical usage
testing, but with similar methods, rare event testing, can be used to handle these events [50].
Techniques for White-box Testing
In contrast to black-box testing, white-box testing uses an internal knowledge of the system to design tests. Here, the tester defines input stimuli to exercise paths through the code and determines not (only) the appropriate outputs, but also how the elements of the control flow are exercised by the tests. Therefore, three major coverage criteria define the techniques for white-box testing [153]:
Statement Coverage: With statement coverage, each statement of a control flow is executed at least once. It ensures that no dead code exists within the system. Statement coverage is considered as the weakest coverage criterion, because the singular invocation of every statement does not guarantee the coverage of every logical path through the control flow and is in consequence only sufficient for the detection of simple defects [137].
Branch Coverage: The set of tests is defined in such a way that all decisions or jumps are tested at least once. Therefore both the then-branch and the else- branch of an if-statement have to be passed through, including also empty branches. In reality this is often complicated, because subsequent conditions are commonly dependent on prior decisions and the resulting paths. This might lead to complex or even impossible input constellations.
Path Coverage: Path coverage requires that all paths of a program are tested. In practise, this is often impossible as even small programs can have a huge or infinite number of paths due to conditional loops. Therefore several other struc- tural testing strategies have been developed to reach a high percentage of path coverage without the need of verifying all paths, e.g. boundary-interior path testing [154].
Figure 2.13 shows an example control-flow model of a simple activation algorithm for a collision avoidance system based on the time left until a collision would occur. The physical meaning of these variables is explained in Section 3.1. The transi- tions of this model are labelled with capital letters. 100 per cent statement coverage within this example can be realised through the input vectors (0, 0, 0), (5, 10, 10) and (5, 10, 5)7, which cover the paths ABCL, ABDEGJK and ABDEGHIEGJK. For
a complete branch coverage only the path ABDEGHIEFL is missing, therefore an- other input vector is defined (5, 10, 15). Finally the last possible path ABDEFL is addressed by (5, 15, 15) reaching 100 per cent path coverage. Please note that the path can never be executed by a correct implementation of the algorithm. Hence, the occurrence of this path indicates a wrong assignment of the variable ds.
ds = da t = ds / dv dv = 0? Start Stop t ≤ 2 ds = db ds = db? active = 1 active = 0 read dv, da, db A B C true false false true true false D E F G J H I K L
Figure 2.13: Example control-flow model with labelled transitions
The techniques commonly applied to reach a high percentage of these coverage cri- teria basically consist of the above mentioned state transition testing and control-flow testing, and another technique called elementary comparison testing. This technique evaluates the paths of a function by identifying its logical conditions in the source code and translating them into pseudo-code [155]. The pseudo-code is used to specify so called test situations, i.e. to find input values which satisfy the identified condi- tions and those which do not. Tests are created from these situations trying to cover each result of each condition at least once. In contrast to state transition testing and control flow testing, no model of the test objective is necessary.
Another group of white-box techniques is based on data-flow analysis. They eval- uate the interaction between the definitions of a program variable and subsequent references to it. The simplest type is called 2-dr interaction and consists of one definition and one reference reached by this definition [156]. Testing each of these interactions is similar to statement coverage, i.e. branch coverage can not be guaran- teed. Hence, the required pairs strategy creates a set of pairs for each 2-dr interaction depending on the occurrence of the definition or the references [157]. If a definition is placed for example within a loop, two iteration counts are used: one for exiting the loop as soon as possible, the other for a distinct number of iterations. Other data-flow based strategies are presented in [158, 159].