ITERA Test Coverage Criteria

Coverage criteria (see 2.2.1) can be applied at different levels of abstraction. On a level of abstraction close to implementation, they are usually based on source code. Nodes of the control flow graph reference individual statements in the source code. At the level of system testing, tests are concerned with the testing of a software against its specification, without the knowledge of the source code (black box test). Accordingly, the nodes of the control flow graph reflect the activities of the user with the SUT. This allows the SUT to be tested from the user’s perspective to see whether the SUT complies with the specification when using defined stimuli, e.g. RFID events.

A direct application of the existing test coverage criteria to system testing is not possible. System tests ensure that every function of the SUT that can be reached by the user is addressed at least by one test. However, this does not guarantee that every single statement in the source code is also checked by this set of tests. Consequently, statement coverage (C0 coverage, analogous

to C1 and C2-coverage), focused on statements in the source code of the application, cannot be

created in the previous steps are interpreted as a control flow graph and build the basis for the coverage metric.

To define coverage criteria at the abstraction level of UML activity diagrams, UML activity diagrams are defined as:

Definition 4.5.1 (Activity Diagram)

A activity diagram is a defined as a tuple AD = (N, C, nI, NF, F ) where

• N = {n1, n2, . . . , an} is a finite set of activities of type ActivityNodes

• T = {t1, t2, . . . , tm} is a finite set of transitions of type ControlNode

• C = {c₁, c2, . . . , cm} is a finite set of guard conditions, where ci is related to ti and

depends on ni.

• n_I ∈ N is the start node of the activity diagram of type InitialNode

• NF ⊂ N is a finite set of end nodes of the activity diagram of type ActivityFinalNode

• F ⊆ {N × T } ∪ {T × N } is the flow relation between the activities and transitions of type ActivityEdge

An activity diagram based on the plain meta-model of UML does not clearly show which nodes represent interactions with the user, depict outputs of the SUT or are context relevant. For this reason, the UML profile (Section 4.3.2) has been defined, providing stereotypes that allow for assigning corresponding attributes to elements of an activity diagram and include concrete semantics.

Within the scope of this thesis, the activities of the activity model are considered as elementary test steps. Therefore, the definition of the coverage criteria listed below refers to activities that are relevant to the execution of the test. Test relevant activities are in this sense direct subclasses of the meta-model defined in the previous section. These are in particular based on the stereo- types <<SystemAction>> or <<ActorAction>>, such as RFIDAddAction or RFIDRemoveAction (derived from ActorInteraction). Based on the definition of the Activity Diagram and in regard to elementary test steps five coverage criteria are defined:

Definition 4.5.2 (activity coverage)

An activity coverage test (C0AD test) for an activity diagram AD is defined as a test, during

which all test-relevant activities are executed at least once. C0AD tests are therefore based

on the classical definition of C0 tests, whereby the activity diagram of the SUT is used as

the control flow graph.

Definition 4.5.3 (activity branch coverage)

A C1AD branch coverage criterion for an activity diagram AD is defined as a coverage crite-

rion that requires all test-relevant branch alternatives (i.e. outgoing edges to DecisionNode elements) in the activity diagram to be covered by at least one test case.

Definition 4.5.4 (activity path coverage)

Path coverage C2AD criterion for an activity diagram is defined in analogy to the classical

C2 test as a test covering criterion, which requires that each path is covered by a test case

in the activity diagram AD.

Analogous to the classic C2 coverage criterion, however, the problem arises in practice that

even for models with relatively low complexity, the number of possible paths quickly increases to a level that is not manageable even when using computers. This problem was identified in the literature as a state-explosion-problem (combinatorial explosion) (cf. Valmari [126], Peled [127]). The approach examined in this thesis avoids this problem by not considering the C2

criterion, but the C2cAD criterion discussed below.

Definition 4.5.5 (activity basis path coverage)

A C2cAD baseline path coverage test for an activity diagram AD, is defined as a C1AD

test which, analogous to the classical C2c test, executes each cycle in the activity diagram

during test execution with at least a given number of passes, whereby concurrent control flow structures contained in the path are removed.

(a) cyclic structure (b) concurrent structure

Figure 4.12: Concurrent and cyclic structures of UML Activity Diagrams

The UML Activity Diagrams provide modelling elements that allow both cyclic and concurrent structures (see Figure 4.12). The classical definitions of the test coverage criteria are based on a control flow graph that does not contain concurrent structures. However, the activity diagram of the SUT is used as a control flow graph, in which concurrent structures are allowed. Whereas a test case intentionally does not include branch nodes, contain cyclic or concurrent structures to guarantee deterministic execution. In order to achieve this, the concurrent activities and loops must be interleaved when they are converted to test cases. If elements exist on concurrent control flow structures, only sequences of permitted permutations of these elements should be reflected by tests, assuming that a user can carry out exactly one interaction with the SUT at any given point in time.

Definition 4.5.6 (concurrent activity path coverage)

A Concurrent Activity Path CCAD is defined as a C2cAD (basis path coverage) test that

passes through each cycle at least once and additionally tests permitted sequences of per- mutations of test-relevant elements on concurrent control flow structures. A sequencing of elements of concurrent control flow structures by permutation is permitted if elements modelled on the same control flow structure are not interchanged with elements of concur- rent control flow structures in their execution sequence after permutation. To avoid the state explosion problem, the CCAD criterion is met if at least one permissible sequencing is

covered by a test.

The method examined in this thesis for generating system tests from activity diagrams implies that concurrent control flows in applications are described from the user’s point of view and can thus represent sequences of adjacent individual actions. The rationale is that a human actor is not able to interact concurrently with multiple components of an application at the same time. Due to human capabilities, there is are always an implicit sequential ordering. However, when multiple actors perform the same activity, as it might be the case in e.g. a warehouse, this is not modelled in the activity diagram directly, rather this is covered during the execution of the tests by running multiple tests at the same time.

In order to take a closer look at the path explosion and to explain the interleaving of the actions, the diagram shown in Figure 4.12b can be considered. Assuming that R ⊆ A × A is the partial order relation of two activities ai ∈ A and aj ∈ A. Precedence of activities, that is activity ai

has occurred before aj, is referred to as ai ≺ aj. The example shown in Figure 4.12b contains

4 concurrent activities and has the order relation R = {A1 ≺ A2, A1 ≺ A4, A2 ≺ A3, A4 ≺

A5, A3 ≺ A6, A5 ≺ A6}. This results in 24 permutations of which 6 are valid (permitted)

activity paths (denoted Pi):

P1 = A1 → A2 → A3 → A4 → A5 → A6

P2 = A1 → A2 → A4 → A3 → A5 → A6

P3 = A1 → A2 → A4 → A5 → A3 → A6

P4 = A1 → A4 → A5 → A2 → A3 → A6

P5 = A1 → A4 → A2 → A3 → A5 → A6

Note that if these would been test cases, every single (test-)path (P1− P6) already fulfils the

CCAD criterion, as only one valid sequence of the concurrent activities is required.