Conformance test generation methodologies are used to design and define the necessary conformance test suites for a protocol. The four most popular techniques used in generating conformance tests are as follows.
4.5.1 Transition Tour
The transition tour method was first suggested by Naito and Tsunoyama [78) for FSM-based sequential circuits. It is the most straight forward approach for conformance test generation. The state transitions defined by the FSM of the protocol specification are exercised at least once by applying the appropriate input sequence and observing the corresponding output.
Sarikaya and Bochmann first applied this method in protocol conformance testing in 1982 [79|. The major disadvantage of this method is the omission of the state verification step which severely limits the fault detection capability of the technique.
4.5.2 Distinguishing Sequence (DS) Method
The DS-Method is a two phase approach. The test on the first phase checks that each state defined by the specification exists in the implementation. In the second phase of the test, the DS-method checks for correct output and transitions in the remaining transitions defined by the specification. This two phase approach is also used by Voung’s UIOv method |8()|, the SW method 18 11 (Single Transition Checking using W set) and the partial W-method (Wp) 1821.
By S B Khoh l*agc 4 - 12
Chapter 4 - The Conformance Testing Conceptsand Methodologies
A distinguishing sequence of an FSM is an input sequence which generates an output sequence that uniquely identifies the state, Si of an implementation prior to the application of the input sequence (Figure 4-2).
Since the DS-Method was originally developed for sequential circuit testing
|75, 83] where all the states are fully specified, it suffers some drawbacks when applied to communication protocols that are typically partially specified. It will not be possible to generate every input sequence for every state due to the partially specified protocol. In addition, the number o f tests to be generated for a FSM of n states will be n , which limits the use of this method.
4.5.3 W-Method
This method is also known as the Characterising Sequences Method and was introduced for FSM specifications that do not posses a distinguishing sequence. It is very similar to the DS-method where a characterising set of input sequences are applied to the state S/ and the output sequences are observed. Each characterising set o f state S/ distinguishes state Si from a group o f states, hence uniquely identifying the state Si after applying all the characterising sequences. This results in very long test suites for most modern protocols as compared with those developed using other methodologies.
Input Sequence
Output Sequence
Figure 4-2 The DS-Method Testing
Il y SO. Khoh Page 4-13
Chapter 4 - The Conformance Testing Conceptsand Methodologies
In short, the Characterising Sequences Method is used as an alternative if the protocol specification does not have a distinguishing sequence. Another disadvantage of this method is that most real life protocols do not exhibit the distinguishing sequences due to partially specified specification. So, it will be difficult to find the distinguishing sequences in this situation.
4.5.4 Unique Input/Output (UIO) Method
The UIO method uses a unique sequence as an input for state, S/ such that its output sequence uniquely identifies the state, S/. The UIO sequence is designed with the knowledge o f what the new state will be. If an implementation under test does not return the expected output sequence, the test will be declared failed. The tester will not have any knowledge on the
Chapter 4 - The Conformance Testing Conceptsand Methodologies
4.5.5 Fault Coverage of test generation techniques
After reviewing the four most popular conformance test generation methods, the next step will be to find out which is the most efficient technique for DeviceNet. Sidhu and Leong did a Monte Carlo simulation [86] to estimate the fault coverage o f these techniques [87]. They devised a method of generating random faulty specifications and ranked them into 10 different classes of faults. For example, a Class 1 fault is formed by modifying at random the output of a state in the specification state machine. A Class 2 fault will be the same as Class 1 except the modification is made on the tail state that follows the previous modified output state, and so on.
Both Sidhu and Leong also defined the two levels of conformance, i.e.
strong conformance and weak conformance. Strong conformance requires the IUT to generate the same outputs for all the input sequences as defined in the specification. Conversely, weak conformance requires the IUT to have the same behaviour as that specified in the core edges o f the protocol state machine. The DUT is allowed to have unspecified behaviour for the input sequences on those non-core edges. The edges in a given protocol state machine are referred to as core edges. The non-core edges consist of unspecified state input sequences, and is assumed that the protocol entity will produce a null output or ignore those input sequences.
Sidhu et al ]87] then concluded that the three methods, i.e. DS, W and UIO methods are excellent for detecting faults in strong conformance testing.
In addition, Dahbura and Sabnani also reported similar findings on UIO method in another published paper |8 9 |. The Transition Tour method, however, could not detect all of the single faults presented for strong conformance testing as concluded by Sidhu et al. On the weak conformance
By S B Khoh Page 4-15
Chapter 4 - The Conformance Testing Conceptsand Methodologies
testing, Transition Tour method again could not detect all of the single faults injected.
In summary, the four methods presented will detect any output error of the implementation, provided the implementation follows the FSM specification. However, transfer errors, i.e. errors in the next state reached by a transition, will not always be found. Therefore, the W-Method and Distinguishing Sequence method will be used to find the transfer errors provided that the number o f states of the implementation remains within a certain boundary. The Transition Tour method mainly concentrates on the controllability issues while the rest of the methods address the observability issues. Controllability aims at bringing the IUT into the desired state where a test is to be conducted, i.e. test preamble. Observability aims at the identification of the IUT’s current state after a state transition has taken place.