Assertions

The Assertion is the basic element for automatic testing as it is evaluating whether a result or output value meets the expectation or not and thereby checks, whether a test has passed or failed.

Generic Concept (xUnit)

Most xUnit frameworks provide a basic set of built-in assertion methods, divided into the following groups:

Single Outcome Assertions always behave in the same way, usually to let a test fail which is not finished yet, or within the try-branch of a try/catch-mechanism

2_{One adaptation of the xUnit architecture to message sequence charts as a model-based approach} has been developed by Yuefeng Zhang [131].

to detect a missing expected exception (see section 2.2.6). The only parameter is an error message – common to all assertion methods –, which is collected in the test result and displayed by the test runner. Example:

fail(’Unfinished test.’);

Stated Outcome Assertions accept a parameter which states whether the assertion will fail or not. The most widely used variant handles a Boolean input, e.g.:

assert(y > foo(x));

y is the expected value, foo(x) a function which is the test objective. Here, the assertion fails if the output of the test objective is smaller than or equal to y, i.e. if the Boolean expression is false.

Equality Assertions compare its two parameters, the expected value and the actual value, and fail if the outcome of the comparison is not true. The relational operator is usually specified through the name of the assertion, e.g.

assert_equals(y, foo(x));

checks whether the values of y and foo(x) are equal. The idea behind this is to display a more descriptive error message than with single or stated outcome assertions. Assuming that the value y was set to 2 and the function foo(x) returns 3, the error message will be automatically set to

Expected <2>, but was <3>.

Furthermore an optional third parameter allows to specify a tolerance, e.g.:

assert_equals(1.41, sqrt(2), 0.01);

will pass if the result of sqrt is greater than or equal to 1.40 and smaller than or equal to 1.42.

More complex assertions can be built using methods or functions of the correspond- ing programming language. The aim of these user-defined assertions is to prevent duplication in reusing code and to avoid conditional test logic.

If an assertion fails the test is stopped at the line of the assertion. However the execution of the next tests is continued as tests are independent of each other in

xUnit. After all assertions associated with a specified test have passed, the test passes which is typically indicated by a green bar. If one assertion fails, the test fails which is usually indicated by a red bar. With the red bar a list of failures including the expected result, the line of the failure or the optional message is shown, so that the developer is able to find the position and the reason for the problem.

Realisation (slUnit)

With slUnit, the Assertion is implemented as the Assert block. It has one Boolean input similar to the method assert, describing whether the assertion should pass – the Boolean input is true – or fail – the Boolean input is false. The state of the assertion is displayed as the background colour of the block – green for success, red for failure. This allows on the one hand the evaluation of the test without the need for an additional graphical user interface or an output on the command line, on the other hand it is easy to locate the failed assertions. In contrast to the xUnit assertion, the Assert block has to consider the time-dependent behaviour of its input signal. The Assert block is failing if the input signal becomes false and will not pass for the remainder of the test, but it will not stop the test. Therefore the assertions are independent from each other, i.e. more than one assertion can fail during a test in slUnit. This is typical for Simulink-based tests as they mostly analyse the behaviour of different signals. If consecutively executed Assertions are required they can be built through additional Simulink components such as enabled Subsystems or state machines.

An extension of the Assert block is the Assert State Change block, which evaluates whether the input changes from a value x to a value y during the simulation. It fails if this change does not happen. The difference to the Assert block is that the Assert State Change block can not fail until the end of the simulation, because until then it is not proved that the state change will not happen. Figure 2.4 shows the difference of both blocks for a number of static and dynamic inputs. In general, the Assert block only gets passed if it is stimulated by a constant Boolean input with the value true. In contrast, the requirements of the Assert State Change block are never fulfilled by a constant input, but by dynamic inputs which describe the specified number of state changes.

A similar block to the method assert_equals is not provided as the comparison of two time-dependent signals is in most cases much more complicated than a = b. Instead, the Propagate block provides the propagation of the background colour of the Assert blocks to the belonging subsystem. This allows the creation of user-defined

(a) Static Inputs

(b) Dynamic Inputs

Figure 2.4: Behaviour of the Assert and Assert State Change block for different inputs

assertions based on the two basic assertion blocks and all common Simulink blocks. Figure 2.5 shows an example assertion which checks the boundaries of its input signal. Finally it is also possible to automatically open Scope blocks or Figure windows for debugging purposes.

Figure 2.5: Example for a user-defined assertion