The basic idea behind model checker based testing is simple and elegant: construct a behaviour model of the system under test (SUT) and validate the behaviour using a model checker ( e.g., use Promela language for constructing the model and verify using Spin, [27]). Such a validated model can then be used for generating executable test cases for the SUT; the model also acts as an oracle, since it also indicates the expected outputs the SUT should produce under given conditions. A principal challenge here is the construction of a model that is sufficiently small (abstract, simple) to enable, as far as possible, exhaustive checking (full validation) by the model checker; at the same time, the model should be realistic enough to be able to generate test cases that exercise the SUT. Different techniques have been proposed in order to force a model checker to create traces suitable as test cases. The traces are generated as counterexamples for property violations; such traces are known as witness traces, which serve to illustrate that a system property is satisfied. Model checker based testing techniques have received wide attention in the software engineering community see [25,43, 46, 52].
4.6.1
Spin based test–case generation tool
A schematic view of test case synthesis tools such TGV [8] for the generation of test cases is shown in bold in Figure 4.14. Then, given a reference model, and some criterion, a set of test cases [54] is produced. The reference model is intended to
represent the behaviour of the system under test. The role of the synthesis tool (SPIN) is to select test cases from behaviours of the system specification. Thus, a second input which is a criterion or test purpose is required. This criterion is aimed to precisely match the system functionalities to be tested. The output of the tool is a set of test cases describing the behaviours of the system under test along with verdicts associated with those behaviours.
Synthesis tool SPIN Test case Counterexample Test purposes LTL propery CB2B formal model Reference ECA rules in extended
Promela Model of Contractual Business-to-
Business Environment
Promela VARS, CONS, Processes, Channels, etc...
to Execution Sequence input output e 1 e2 e3 en ( )
- - -
- - -
Figure 4.14: General structure of test case synthesis tool using CB2B model
4.6.2
Test–case generation steps
Generating test cases with the CB2B model and the general Spin model checker in- volves three steps. Firstly, an abstract reference model of the SUT is built. Then the test purpose concerning the property of interest such as when the SUT progresses from state Sito state Sj after being presented with input ei is formed as LTL formula.
Thirdly, the LTL is negated and presented to the model checker (the synthesis tool ) with the challenge to execute the abstract model to show that the negated LTL claim can be violated. As a result, the model checker then produces witnesses that include
transitions from Si to Sj when ei is provided. As explained in [44], counterexam-
ples (or witnesses) produced by model checkers contain abstract parameters that are meaningful to the abstract model but meaningless to the SUT. Thus they serve only as raw data to produce executable test cases that can be fed into the SUT to exercise a run. We note that a verified CB2B model (the reference) will by default generate sequences with events corresponding to the execution of contract–compliant opera- tions only. However, the model can be tuned to generate sequences which include unknown and non–compliant business events; different categories of non–compliant business events have been discussed in Chapter 3. Once a test–case is generated, a test verdict of pass or fail is then associated with it, so it can be used to test the correctness of behaviours of the actual implementation of the contract rules.
4.6.3
Limitations of Spin counterexamples
Model–checker based approaches, for test–case generation, offer many advantages: They are fully automated and flexible. However, because model–checkers in general were not originally designed as test–case generation tools, there are many limitations for using them for this purpose. In this section we list the main limitation of using Spin for generating test–cases. We note that the list of limitations discussed below might be applicable to a large class of model checkers not only Spin.
• The enormous state space of finite–state models of practical software specifica- tions often leads to the state explosion problem: Spin model checker might run out of memory or time before it can analyze the complete state space.
• Because Spin does a depth–first search (DFS) of the state–machine model, it produces very long counterexamples (which means very long test–cases). We note that Spin offers switch which finds the shortest counterexample, this might
• A large percentage of the test-cases produced using Spin model checker are redundant. Various heuristics approaches suggested to tackle the redundancy of test–cases (see for example [15–17]).
• Finally, a major weakness of this approach is the reliance of this method on a manual translation of the specification to the model–checker input language, which requires some skill and ingenuity. In our case, we tried to tackle this problem by developing an intuitive high–level notation (by extending Promela, the input language of Spin model checker) to model the contract rules.
The B2B contractual interactions can give rise to highly complex execution pat- terns, and it is quite unrealistic to assume that these can be produced manually for testing purposes. Testing tool support is therefore would help at design time to validate the consistency of the contractual clauses and later, to produce test case val- idation sequences to test the correctness of the actual implementation. This Section has briefly illustrated the idea of using the CB2B model and Spin model checker to generate test–cases for electronic contracts. An earlier version of the CB2B model [2] has been used to demonstrate this tool for test–case generation while testing an electronic contract system [51].