Specification-based class testing: A case study
Ian MacColl ianm
Leesa Murray leesam
Paul Strooper pstroop
David Carrington davec
@csee.uq.edu.au
Software Verification Research Centre
Department of Computer Science & Electrical Engineering The University of Queensland
Abstract
This paper contains a case study demonstrating a com- plete process for specification-based class testing. The pro- cess starts with an abstract specification written in Object- Z and concludes by exercising an implementation with test cases and evaluating the results. The test cases are derived using the Test Template Framework for each individual op- eration. They are analysed to generate a finite state ma- chine that can execute test sequences within the ClassBench framework. An oracle is also derived from the Object-Z specification. The case study demonstrates how a formal specification contributes to the development of practical tests that can be executed by a testing tool. It also shows how a test oracle can be derived from a specification and used by the same testing tool to evaluate test results.
1. Introduction
The purpose of specification-based testing is to derive testing information from a specification of the software un- der test, rather than from the implementation. Although it is possible in theory to formally refine a specification into an implementation, this rarely happens in practice. However, when the implementation is developed informally from a formal specification, the specification can play a major role in the testing process.
Our method for specification-based class testing is shown in Figure 1. Testing involves three main tasks: test derivation, execution and evaluation. In our approach, test inputs are derived from a formal specification. The test in- puts are analysed to develop a finite state machine, which is transformed to control test sequencing in a test execu- tion environment. The specification is also translated into an oracle class for test evaluation in the test execution envi- ronment.
We use the Test Template Framework [24] to structure the test inputs and guide test derivation. In previous work we have extended the Test Template Framework to Object-Z [5], deriving a finite state machine for test sequencing from a class' s specification [17]. The Test Template Framework provides structure for testing but is not used for actually performing tests.
ClassBench [9] is an approach to automated class testing designed especially for testing collection classes. The class under test is (partially) modeled as a finite state machine.
The partial model, referred to as a testgraph, represents a subset of the states of the class and the transitions between them. ClassBench executes tests by traversing the testgraph and evaluates results by comparing the state of the class un- der test to the expected state supplied by an oracle class.
In this paper we present the first complete case study ap- plying all the steps of our approach to an example, deliber- ately kept simple to ease presentation. The case study is a part of a Dependency Management System. Section 2 con- tains an Object-Z specification of part of the Dependency Management System. In Section 3 we use the Test Template Framework to derive test cases and expected outputs, and in Section 4 we develop a finite state machine representation of the class under test. In Section 5 we transform the finite state machine into a testgraph, and in Section 6 we translate the specification into a ClassBench test oracle class. Testing of an independently developed implementation is presented in Section 7. In Section 8 we discuss related work and Sec- tion 9 concludes.
Throughout this paper we make extensive use of Z [21, 26] and Object-Z [5], an object-oriented variant of Z with its own semantics. We assume familiarity with Z and introduce Object-Z extensions as they arise. We adopt a convention of using emphasis for specification constructs andmonospacefor implementation constructs.
TTF
ClassBench
Development
Test Cases & Oracles
Analysis
Class FSM
Testgraph
Object-Z Specification
Oracle Class Class Implementation
Test Results Transformation
Oracle Generation
Figure 1. Our approach to specification-based class testing
2. Case study
The Dependency Management System (DMS) case study was a testbed for a project exploring software devel- opment methodologies [13]. The DMS is a critical compo- nent of a theorem-proving tool, tracking dependencies be- tween theorems and assertions in a proof, thus preventing circular reasoning. However, the concept of dependency management generalises to other problems (such as revision control systems), so the DMS is a reasonably generic com- ponent. It is comprised of a set of items under management and the dependency graph between them.
The DMS tracks dependencies between nodes. The ba- sic DMS maintains a set of nodes, direct dependencies be- tween nodes, and inferred transitive dependencies between nodes. The DMS provides operations for manipulating a dependency graph, from adding nodes and dependencies to calculating all the nodes dependent on some other node. In this paper we consider only features associated with the set of nodes.
2.1. Specification
We specify the features associated with the set of nodes of the DMS in Object-Z (ignoring the dependency informa-
tion). The complete specification is available as part of a technical report [14].
Items[X] xs:FX
INIT xs=? NoXs
result!:B result!,xs=?
IsX x?:X result!:B result!,x?2xs AddX
(xs) x?:X x?62xs xs0=xs[fx?g
RemoveX
(xs) x?:X x?2xs xs0=xsnfx?g
An Object-Z class is represented as a named box with optional generic parameters (here class Items with generic parameter X). A class contains an unnamed state schema, an initialisation schema (INIT) and zero or more operations (four in this case).
The state variable xs represents the aggregation of items stored in the system' s database. Initially there are no nodes.
Operation NoXs returns true if and only if xs is empty.
Operation IsX returns true if and only if the input node x? is a member of xs. NoXs and IsX do not change the state of the class.
In an Object-Z operation schema, state variables are im- plicitly equated to their primed after-state equivalents unless included in a-list. For example, state variable xs is in the
-list of operation AddX and so can be changed.
Node x?is added to xs by AddX. An Object-Z opera- tion is disabled outside its precondition, in contrast to Z in which precondition violation gives an undefined result (sat- isfying the precondition is interpreted as a client responsi- bility). AddX is disabled if x?is already in xs. Operation RemoveX removes node x?. The precondition is that the in- put value must be an existing node and the remaining con- junct specifies the removal of the designated node from the set of nodes.
3. Test cases and expected outputs
Our approach to specification-based class testing is based on the Test Template Framework [24]. The Frame- work is a formal, abstract model of testing, used to derive a hierarchy of test information from a model-based formal specification. The hierarchy includes test inputs and ex- pected outputs, but is not used directly for test execution.
The Test Template Framework provides structure to testing information and processes without mandating any particular
techniques or methods. Most of the work with the Frame- work has used the Z specification notation [21, 26]. In [18], we extend the Framework to accommodate object-oriented features such as those available in Object-Z.
We use the AddX operation to illustrate the Test Template Framework. We expand the operation as a Z schema (in- cluding expansion of the-list), and instantiate the generic parameter as the integers to simplify our presentation.
AddX xs;xs0:FZ x?:Z x?62xs xs0=xs[fx?g
The Test Template Framework uses the valid input space (VIS) of an operation as the source of all tests, since an Object-Z operation is disabled for inputs outside the valid input space. The valid input space for AddX is the precon- dition of the operation' s schema
VISAX=b pre AddX
(here, and in the remainder of the paper, we use two-letter abbreviations of operation names as subscripts) which ex- pands and simplifies to
VISAX=b [xs:FZ;x?:Zjx?62xs]
The basic unit for defining test data in the Test Template Framework is a test template (TT). A test template is a con- strained subset of the valid input space and is expressed as a Z schema. We define a type for all AddX test templates.
TTAX==PVISAX
A strategy in the Test Template Framwork identifies a particular technique for deriving test cases. The Framework encourages the use of multiple strategies to build the test template hierarchy, both traditional techniques such as in- put partitioning and boundary analysis [1], and specialised techniques that exploit the specification notation [23]. We take the set of all possible testing strategies as given.
[Strategy]
Test templates are organised into a hierarchy called the test template hierarchy (TTH). The root of this hierarchy is the valid input space. The hierarchy is created by applying testing strategies to existing test templates to derive addi- tional test templates. We identify the AddX test template hierarchy.
TTHAX:TTAXStrategy!7 PTTAX
A common, intuitive testing strategy is 0-1-many, based on the cardinality of a container type. We refer to this as
(one example of) type-based selection. We identify type- based selection (TB) as a particular testing strategy.
TB:Strategy
We use type-based selection to partition the valid input space into cases where xs is empty, a singleton, and contains more than one element, deriving three test templates as a result.
TTAX:1=b [VISAXj#xs=0] TTAX:2=b [VISAXj#xs=1] TTAX:3=b [VISAXj#xs>1]
TTHAX(VISAX;TB)=fTTAX:1;TTAX:2;TTAX:3g
We distinguish the new test templates by sequentially num- bering the subscript.
Test templates are derived until the tester is satisfied that the subdivisions of the valid input space are adequate to achieve satisfactory testing. A test template hierarchy is usually a directed acyclic graph with the leaf nodes parti- tioning the valid input space.
The formal specification can also be used to determine the success or failure of each test. We derive oracle tem- plates (OT) to describe suitable output for a given input.
Oracle templates can be derived at any point from test tem- plates but are commonly used for unique instantiations of test templates (referred to as instance templates). For ex- ample, the oracle templates for the above test templates are
OTAX:1=b[xs0:FZj#xs0=1] OTAX:2=b[xs0:FZj#xs0=2] OTAX:3=b[xs0:FZj#xs0>2]
The complete test derivation of the other operations is avail- able as part of a technical report [14].
The Test Template Framework provides a model and pro- cess for deriving testing information from a formal speci- fication, but it does not provide mechanisms for executing tests or checking results. We use test and oracle templates to derive states and transitions for a finite state machine rep- resenting the class under test. The finite state machine is transformed for use in ClassBench, which provides test ex- ecution and result checking.
4. Finite state machine
This section details the development of a finite state ma- chine from testing information derived from an Object-Z specification. We address the development in two steps:
first the derivation of the states of the finite state machine and then the transitions.
4.1. States
We use the Test Template Framework to derive test tem- plates and oracle templates for the class' s operations. We
derive the states of the class' s finite state machine from the class' s INIT schema and from each of the operations' test and oracle templates. The states derived from test templates only guarantee that some operation of the class can be exe- cuted in each state. However, there is no guarantee that the state reached after the operation terminates is a member of the set of states derived from test templates. Therefore it is also necessary to use the oracle templates to derive states for the finite state machine. We only consider oracle templates for operations that change the state of the class, as opera- tions that do not change the state cannot contribute any new states. The collection of states gained from the INITschema, the test and oracle templates gives us the set of states for the class' s finite state machine, and these are referred to as state templates (ST).
The state template derived from the INITschema of Items is
STINIT=b[xs:FZjxs=?]
We derive states from the leaf test templates of each op- eration' s test template hierarchy by using schema hiding to restrict the signatures of the templates to only the state vari- ables. Hiding involves removing the input variables from the template declaration and existentially quantifying them in the template predicate. For example, the state template derived from test template TTAX:1is
STTTAX:1 b
=TTAX:1n(x?)[xs:FZj#xs=0]
We derive ten test templates from the four operations of class Items (NoXs, IsX, AddX and RemoveX), which, in turn, provide ten state templates (subscript NX, IX, AX and RX respectively).
STTTNX:1 b
=[xs:FZjxs=?] STTTNX:2
b
=[xs:FZjxs6=?] STTTIX:1
b
=[xs:FZjxs6=?] STTTIX:2
b
=[xs:FZ] STTTAX:1
b
=[xs:FZj#xs=0] STTTAX:2
b
=[xs:FZj#xs=1] STTTAX:3
b
=[xs:FZj#xs>1] STTTRX:1
b
=[xs:FZj#xs=1] STTTRX:2=b[xs:FZj#xs=2] STTTRX:3
b
=[xs:FZj#xs>2]
We calculate oracle templates for operations that change the state of the class (AddX and RemoveX), to ensure that all post-states of operations are included in the finite state ma- chine. After calculation of the oracle templates, state tem- plates are formed by renaming the primed state variables to
their unprimed equivalents, and hiding any output variables.
The templates are then simplified so only state variables ap- pear in the templates' declarations.
STOTAX:1 b
=[xs:FZj#xs=1] STOTAX:2
b
=[xs:FZj#xs=2] STOTAX:3
b
=[xs:FZj#xs>2]
STOTRX:1 b
=[xs:FZj#xs=0] STOTRX:2=b[xs:FZj#xs=1] STOTRX:3
b
=[xs:FZj#xs>1]
Once the set of state templates is derived, we consider whether the templates are disjoint. If they are not, it is nec- essary to resolve any overlap of states so that we have a maximal partition1 of the class' s state space. To achieve this, we derive a canonical disjunctive normal form (DNF) to construct a partition, based on the technique described by Dick and Faivre [3], using equivalences of the form
A_B(A^B)_(:A^B)_(A^:B)
If the INIT schema is partitioned in the process of formu- lating disjoint templates, the subscript INIT is added to the names of the resulting templates to track those templates that are initial states of the class' s finite state machine. This does not occur in our example.
We form a set of disjoint state templates by remov- ing duplicates and considering overlap of remaining state templates. State templates STTTNX:1
;STTTIX:1, STTTAX:1 and STTTRX:1, and all those derived from oracle templates are du- plicates. For the remaining seven candidate templates we have four that form the equivalence classes of a partition, and three that can be partitioned.
STINIT equivalence class
STTTNX:2 hSTTTAX:2;STTTRX:2;STTTRX:3i
STTTIX:2
hSTINIT;STTTAX:2
;STTTRX:2
;STTTRX:3 i
STTTAX:2 equivalence class STTTAX:3
hSTTTRX:2
;STTTRX:3 i
STTTRX:2 equivalence class STTTRX:3 equivalence class
We rename the four remaining state templates as the states of our finite state machine. The subscript corresponds to the cardinality of xs: 0, 1, 2 or Many.
ST0=b[xs:FZj#xs=0] ST1
b
=[xs:FZj#xs=1] ST2=b[xs:FZj#xs=2] STM=b[xs:FZj#xs>2]
1By maximal partition, we mean that we maximise the number of STs in the partition, or, conversely, that we make the `granularity' of the par- tition as small as possible, i.e., that we minimise the amount of the state space covered by each ST.
RX
AX RX
AX RX AX
AX RX
ST ST ST
ST0 1 2 M
IX NX IX NX IX NX IX NX
Figure 2. Finite state machine representation ofItems
4.2. Transitions
In the previous section, we partitioned a class' s state into state templates to be used as the nodes of its finite state ma- chine. We derive the transitions of the finite state machine by considering each pair of states and checking whether the pair is related by an operation (see [17] for a formal model of this process).
A valid transition occurs when a pair of state templates,
(STi;STj0), is related by an operation. For each valid tran- sition identified, an arc labelled with the operation' s name, beginning in state STiand terminating in state STj, is added to the class' s finite state machine.
We choose two possible Items transitions to illustrate the derivation of valid and invalid transitions of a class. To show that (ST0;ST00
)are not related by AddX and do not form a valid transition in Items finite state machine, we prove2
9ISAX; OSAXST0
^AddX^ST00
9xs:FZ;x?:Z;xs0:FZ
#xs=0^
x?62xs^xs0=xs[fx?g^
#xs0=0
false
Conversely, to show that(ST0;ST10
)are related by AddX, we prove
9ISAX; OSAXST0^AddX^ST10
9xs:FZ;x?:Z;xs0:FZ
#xs=0^
x?62xs^xs0=xs[fx?g^
#xs0=1
true
ST0is the only initial state of Items as it was not parti- tioned during the derivation. We represent initial states by an incoming arc that does not originate in a state and has no label. The resulting finite state machine of class Items is shown in Figure 2. The finite state machine derived from
2The input space of an operation (ISOp) is the operation' s signature restricted to input and before-state variables. The output space of an oper- ation (OSOp) is the operation' s signature restricted to output and after-state variables. The expression9ISOp; OSOpmakes Op' s signature available within the quantification.
an Object-Z specification is now transformed into a partial model of the class under test, called a testgraph, for testing under ClassBench.
5. Testgraph
ClassBench [8, 9] is an approach to automated class test- ing designed especially for testing C++ collection classes.
For each ClassBench test suite, the tester must provide three components: a testgraph, anOracleclass, and aDriver class.
The testgraph is a finite state machine that models a sub- set of the possible states and transitions of the class under test (or CUT). The testgraph nodes correspond to states of the CUT and the arcs correspond to the transitions; there is one distinguished node, the start node, that represents the initial state of the CUT.
TheOracleclass provides the same operations as the CUT, but supports only the testgraph states and transitions.
TheOracleclass is discussed in the next section.
The Driver class is called by the framework as the testgraph is traversed. The Driver must provide two functions. The Driver::arc()function performs the state transitions associated with each testgraph arc on both the CUT and Oracle. The Driver::node() func- tion checks that the CUT behaviour is correct for the state corresponding to the current testgraph node. At run-time, the framework automatically traverses the testgraph, call- ing Driver::arc()each time an arc is traversed, and callingDriver::node()each time a node is reached.
To test the Items implementation with ClassBench, we must provide a testgraph. In [19] we detail the process for deriving a class' s testgraph from its formally derived finite state machine. Due to the abstractness of the finite state ma- chine, and the relative concreteness of the testgraph, many issues must be resolved for the derivation to be possible and successful. These issues and their resolutions are also pre- sented in [19]. The derivation of a testgraph is a four step process.
Step 1: States!Testgraph Nodes
To begin, we map each finite state machine state to a test- graph node. We choose concrete values for the class' s state variables that conform with the finite state machine' s state and types chosen for generic parameters in the class' s implementation. We map Z integers (Z) to C++ integers (int), and for the testgraph we choose the set of integers
f1;2;3;4;5g. The result of transforming the Items finite state machine states (shown in Figure 2) to testgraph nodes is shown in Figure 3. The value chosen for the Items state variable, xs, is shown above each node. Note that forTGM
we have chosen a set of four integers, but that any set with more than two integers would have been acceptable.
TG0 {}
TG2 {1,2}
TG1 {1}
TGM {1,2,3,4}
Figure 3. Testgraph nodes forItems.
Step 2: Choose a Start Node
A testgraph has one start node. The Items finite state ma- chine has one initial state, ST0. The testgraph nodeTG0is derived from ST0and so becomes the start node as shown in Figure 3.
Step 3: Reachable Nodes
In a testgraph, all nodes should be reachable. By reach- able, we mean that there is a path from the start node to every other node, via a sequence of arcs. We achieve this by identifying the transitions in the finite state machine that make its states reachable. For example in Figure 2, ST1 is reachable from ST0 by way of the AddX operation. In our testgraph, we identify the nodes derived from ST0and ST1,TG0andTG1respectively, and formulate a sequence of operation calls to transform TG0toTG1. We use the transition' s operation somewhere in the sequence, prefer- ably at the end. In this case, we achieve this with a call to AddX(1), and call this arcAX1. This process is continued until all derived nodes in the testgraph are reachable. The Items' testgraph is shown in Figure 4, which also gives the operation calls associated with each arc.
TG0 {}
TG1 {1}
TG2 {1,2}
AddX(1); AddX(2); AddX(4);
TGM {1,2,3,4}
AddX(3);
Figure 4. Reachable testgraph nodes forItems.
Step 4: Arc Coverage
Each transition in the finite state machine that changes the state of the class must map to a testgraph arc. We achieve this by the same process used in Step 3 to map transitions to arcs. However, there are two special cases. One is tran- sitions that begin and end in the same state, as AddX (AX) does for STMin Figure 2. Such arcs are not permitted in testgraphs. To overcome this problem, we derive another node from the state involved. That is, for the example just stated, we derive nodeTGMAfrom STMand give it a state that permits anAddXtransition fromTGMto it. The second
case arises when multi-arcs occur; that is, multiple transi- tions with different names linking the same states. These multi-arcs are also not permitted in testgraphs. We solve this problem in the same way: introduce a new node in the testgraph derived from the destination state. The complete, derived testgraph for Items is shown in Figure 5.
{}
TG0
{1,2}
TG2 {1}
TG1 TGM
{1,2,3,4}
RemoveX(3);
RemoveX(4);
AddX(5);
AddX(2);
AddX(1); AddX(4);
TGMA {1,2,3,4,5}
RemoveX(1); RemoveX(2); RemoveX(5);
AddX(3);
Figure 5. Testgraph forItems.
Transitions in the class' s finite state machine that do not change the class' s state are not mapped to arcs in the derived testgraph, but to code in Driver::node(). For Items this applies to all NoXs and IsX transitions in Figure 2. This process is described in [19].
6. Oracle class
In the original ClassBench framework, the behaviour of the implementation is checked by anOracleclass that is written by the tester. However, McDonald has developed a method for systematically deriving a test oracle from an Object-Z specification [15]. The method is sufficiently gen- eral that the generated oracle can be used in most testing frameworks, but in this section we show how to derive an oracle for Items for use in the ClassBench test suite.
The oracle we generate is a passive test oracle that checks the behaviour of the implementation, rather than re- producing this behaviour. The oracle essentially provides a wrapper class around the implementation that checks each time an operation is called that the operation behaves cor- rectly. The method follows the suggestion by H¨orcher [10]
to check the behaviour in the abstract state space of the specification rather than the concrete state space of the im- plementation. To accomplish this, the method relies on an abstraction function implemented by the tester.
There are two stages in mapping an Object-Z specifica- tion to a passive test oracle: optimisation and oracle trans- lation. Optimisation is the rearrangement of the specifica- tion to simplify translation to an implementation language.
Oracle translation maps the optimised specification to C++
code.
The optimisation step involves changes such as flattening inheritance, schema inclusions, schema enrichments (and performing any associated renamings), replacing concur- rency with equivalent sequential constructs (Griffiths [6]), and replacing constructs that are not finitely evaluable with equivalent finitely evaluable constructs (where possible). At the end of the optimisation step, if any constructs cannot
be transformed into finitely evaluable constructs, then the translation fails. For Items no optimisation is necessary.
To assist in producing oracles, McDonald has developed a library of classes that implement some of the standard types of the Z mathematical toolkit [15]. This library pro- vides implementations of the set, sequence, relation and function types. The implementations are in the form of generic classes. For example, the declaration Z Set<X>
xs;in a test oracle corresponds to the definition xs:FX in Object-Z.
To translate from an optimised specification to a passive oracle implemented in C++, the oracle inherits the imple- mentation and augments it with code to check its behaviour.
We use inheritance so that the oracle gains access to the private state of the implementation, to which it applies the abstraction function. The oracle provides the following op- erations:
abs(): an abstraction function that maps the imple- mentation state to the abstract state of the specifica- tion. It is the tester' s responsibility to implement the abstraction function, and the complexity of the abstrac- tion function depends on the complexity of the state of the implementation.
inv(): an invariant checker that is passed the abstract state calculated by the abstraction function and that checks whether that state satisfies the invariant pred- icate of the specification.
class constructor: a class constructor that checks the behaviour of the implementation' s constructor.
one operation for each operation in the implementa- tion: the operation checks the behaviour of the corre- sponding operation in the implementation.
For Items, the state space in the test oracle is declared as Z_Set<X> xs;
The abstraction function copies the implementation state to a parameter of the same type as the specification state (here toZ Set<X>) for subsequent checking. Since there is no invariant predicate in Items, the invariant checker does noth- ing.
The oracle constructor for Items is shown below.
template<class X>
DMSOracle<X>::DMSOracle():DepManSys<X>()
f
Z Set<X> xs = Z Set<X>();
abs(xs);
inv(xs);
check init(xs);
g
The oracle inheritsDepManSysand so calls the inher- ited constructor. It then declares a variable xs to repre- sent the abstract specification state and callsabs(xs)to instantiate this variable to the abstract specification state corresponding to the current concrete implementation state.
The call toinv()checks that the abstract state satisfies the class invariant, andcheck init()evaluates whether or not the implementation satisfies the INITschema of the spec- ification. In this case, it simply checks that the current state represents the empty set using the==operator provided by the Z toolkit library.
template<class X>
void DMSOracle<X>::check init(Z Set<X> xs)
f
CHECKTRUE(xs == Z Set<X>());
g
CHECKTRUE is a macro provided by the ClassBench framework that checks a boolean value and outputs an error message if the value does not represent the boolean value true(here checking the assertion that the state is an empty set).
The oracle replaces each of the implementation' s oper- ations by an augmented version that calls the original op- eration, checks its behaviour, and signals any differences between the specified behaviour and the actual behaviour.
For example, the implementation of the augmented version ofAddX()is shown below.
template<class X>
void DMSOracle<X>::AddX(X x)
f
Z Set<X> pre xs;
abs(pre xs);
inv(pre xs);
DepManSys<X>::AddX(x);
Z Set<X> post xs;
abs(post xs);
inv(post xs);
check AddX(pre xs,post xs,x);
g
Before calling the inherited operation, the oracle instan- tiates the abstract pre-state in pre xs and calls inv() to check that it satisfies the class invariant. After calling the inherited operation, it records the abstract post-state in post xs, callsinv()to check that the post-state satis- fies the class invariant, and callscheck AddX()to check that the pre- and post-states satisfy the specification of the operation. The invariant is checked both before and after calling the CUT operation because we view the invariant as an additional pre- and postcondition on each operation.
The check function forAddX() is shown below. It is a straightforward translation from the Object-Z specifica- tion for AddX to the corresponding expression using the Z toolkit library.
template<class X>
void DMSOracle<X>::check AddX(
Z Set<X> pre xs, Z Set<X> post xs, X x)
f
CHECKTRUE(post xs ==
(pre xs + Z Set<X>(x)));
g
For operations that do not modify the state (i.e., those with no-list), the operation checker additionally checks that the pre-state and post-state are the same.
The checks for other operations are similar. Note that al- though the oracle code for each operation is lengthy, most of this code is the same for all operations and can be generated automatically.
7. Implementation and testing
In this section we describe an independently developed implementation of the Items specification, and the results of testing it.
7.1. Implementation
The implementation that we tested implements the full Dependency Management System (DMS) specification as a generic C++ class with a complex inheritance structure involving nine classes. Even though we include the full im- plementation in our test suite, the testing described below tests only Items operations.
7.2. Testing
To test the implementation, we instantiate the generic DMS class tointeger. The implementation was tested using ClassBench, with the testgraph described in Section 5 and the passive oracle derived in Section 6.
During testing, each arc in the testgraph shown in Fig- ure 5 was traversed once, nodeTGMAwas visited once, and the other nodes were visited twice. No errors were detected in the implementation. However, initiallycheck AddX() from the oracle reported a problem that was traced to an er- ror in the oracle' sabs()function. Table 1 gives the num- ber of uncommented lines of source code in the driver, ora- cle and implementation classes. Note that the 752 lines for the implementation include the implementation of all of the DMS, not just Items.
8. Related work
In this section, we discuss related work on object- oriented testing based on specifications.
Bosman and Schmidt [2] use finite state machines to test classes. Their method uses state machines that result from
Driver Oracle Implementation
108 126 752
Table 1. Lines of code for Driver, Oracle and implementation ofItems.
Statecharts [7] used in object-oriented analysis and design.
A design fsm for a class is generated from the state diagrams and defines the expected outcomes of test cases. A repre- sentation fsm is an abstraction of the class' s implementation and is used to drive the testing of the class. It also provides a mapping between the design and implementation of the class. Their approach is similar to ours but starts from a behavioural rather than a model-based specification. Their tool support appears less developed than ClassBench.
Turner and Robson' s [25] state-based testing technique uses finite state machines for test case generation. Finite state machines are constructed to model the internal repre- sentation of a class, in contrast to Bosman and Schmidt, who use a state machine constructed for object-oriented analysis and design. Turner and Robson highlight the im- portance of considering state in object-oriented testing. In our approach, the Test Template Framework provides ac- cess to both the state of the class and the local parameters of an operation. Testing strategies can be applied to either.
ASTOOT, developed by Doong and Frankl [4], provides tools for automatic driver generation from a class interface specification and semi-automatic test case generation from algebraic class specifications. Each test case contains an original message sequence and a simplified message se- quence. The two sequences should leave objects of the class under test in observationally equivalent states. How- ever, it is left to the tester to provide a means of determin- ing whether the states are actually equivalent. ClassBench provides this functionality and reports if non-equivalence occurs.
Stepney [22] describes a method for formally specifying tests based on applying abstraction to a specification. The method uses the ZEST object-oriented variant of Z, and in- volves systematically introducing non-determinism into the specification. A firing condition interpretation of precondi- tion is used and testing information is structured as an in- heritance tree of specifications. The method is supported by the Zylva tool.
Kung et al. [12] describe the Object-Oriented Testing and Maintenance (OOTM) environment. OOTM uses a mathematically defined test model, derived from the source code. This highlights a limitation of the system as it only supports program-based testing.
Murphy et al. [16] describe testing of classes and clus- ters using the Automated Class Exerciser (ACE) tool. The
ACE tool was developed to support the testing of C++ and Eiffel classes. The tool takes a test script and generates a test harness. Each test script describes how to compile the class under test and provides code for the test cases and the expected results of the tests. The test harness runs the test cases and reports the results, any failures being entered into a problem management system for later correction.
Smith and Robson [20] present the Framework for Object-Oriented Testing (FOOT). The framework allows a class to be tested either manually or automatically, using either a test strategy or a previously defined test sequence.
FOOT provides several testing strategies, including exhaus- tive testing method calls, testing for memory leaks, and test- ing that sequences of operations that should be identities do actually leave the state unchanged.
9. Conclusion
Our method for testing, based on object-oriented formal specifications, has been demonstrated by a case study. The case study, although small, illustrates how the various tasks use information derived from the Object-Z specification to construct practical tests that can be executed in the Class- Bench framework. The Object-Z specification is also used to derive a test oracle for ClassBench.
Our method achieves a beneficial synergy between the two tasks of software testing: test generation and test execu- tion. It links the existing Test Template Framework for test generation with the ClassBench test execution framework.
Future work will investigate tool support for our method of testing from object-oriented formal specifications.
10. Acknowledgements
The Dependency Management System (DMS) was orig- inally specified in Object-Z by Gordon Rose in at least two different versions. Our Object-Z specification is based on a Z specification by Phil Stocks [23] which was derived from Gordon' s.
The implementation used in Section 7 was originally de- veloped by Wendy Johnston [11] under Unix from one of the original Object-Z specifications. Minor modifications have been made to handle a change of operating system to Microsoft Windows 95 and a compiler change to Borland C++ 4.5.
We also thank Jason McDonald for his assistance with ClassBench to help us execute our tests, and for reading drafts of this paper. We also thank the anonymous referees for their helpful suggestions.
This work is supported by Australian Research Council Grant A4-96-00282. Ian MacColl is supported by an Aus- tralian Postgraduate Award and a Telstra Research Labora- tories Postgraduate Fellowship.
References
[1] B. Beizer. Software Testing Techniques. Van Nostrand Reinhold, 2nd edition, 1990.
[2] O. Bosman and H. W. Schmidt. Object test coverage using finite state machines. In Technology of Object- Oriented Languages and Systems (TOOLS 18), pages 171–178. Prentice-Hall, 1995.
[3] J. Dick and A. Faivre. Automating the generation and sequencing of test cases from model-based specifica- tion. In J. C. P. Woodcock and P. G. Larsen, edi- tors, FME ' 93, volume 670 of LNCS, pages 268–284.
Springer-Verlag, 1993.
[4] R.-K. Doong and P. G. Frankl. The ASTOOT ap- proach to testing object-oriented programs. ACM Transactions on Software Engineering and Methodol- ogy, 3(2):101–130, 1994.
[5] R. Duke, G. Rose, and G. Smith. Object-Z: A specifi- cation language advocated for the description of stan- dards. Computer Standards & Interfaces, 17:511–533, 1995.
[6] A. Griffiths. From Object-Z to Eiffel: Towards a rig- orous development method. In Technology of Object- Oriented Languages and Systems (TOOLS 18), pages 293–307. Prentice-Hall, 1995.
[7] D. Harel. Statecharts: A visual formalism for com- plex systems. Science of Computer Programming, 8(3):231–274, 1987.
[8] D. Hoffman and P. Strooper. The testgraph methodol- ogy: Automated testing of collection classes. Journal of Object-Oriented Programming, 8(7):35–41, 1995.
[9] D. Hoffman and P. Strooper. Classbench: A method- ology and framework for automated class testing.
Software—Practice and Experience, 27(5):573–597, 1997. Also SVRC TR96-03.
[10] H.-M. H¨orcher. Improving software tests using Z specifications. In ZUM' 95, volume 967 of LNCS, pages 152–166. Springer-Verlag, 1995.
[11] W. Johnston and G. Rose. Guidelines for the man- ual conversion of Object-Z to C++. Technical Report TR93-32, Software Verification Research Centre, The University of Queensland, 1993.
[12] D. Kung, P. Hsia, J. Z. Gao, Y. Toyoshima, Chris Chen, Young-Si Kim, and Young-Kee Song. Devel- oping an object-oriented software testing and main- tenance environment. Communications of the ACM, 38(10):75–87, 1995.
[13] P. Lindsay. The dependency management system case study. Technical Report 94-01, Software Verification Research Centre, The University of Queensland, 1994.
[14] I. MacColl, L. Murray, P. Strooper, and D. Carring- ton. Specification-based object-oriented testing: A case study. Technical Report 98-08, Software Verifica- tion Research Centre, The University of Queensland, 1998.
[15] J. McDonald. Translating Object-Z specifications to passive test oracles. In International Conference on Formal Engineering Methods (ICFEM98), 1998. Also SVRC TR98-04.
[16] G. C. Murphy, P. Townsend, and P. Wong. Experiences with class and cluster testing. Communications of the ACM, 37(9):39–47, 1994.
[17] L. Murray, D. Carrington, I. MacColl, J. McDonald, and P. Strooper. Formal derivation of finite state ma- chines for class testing. In ZUM' 98, volume 1493 of LNCS. Springer-Verlag, 1998. Also SVRC TR98-03.
[18] L. Murray, D. Carrington, I. MacColl, and P. Strooper.
Extending Test Templates with inheritance. In Australian Software Engineering Conference (ASWEC97), pages 80–87. IEEE Computer So- ciety, 1997.
[19] L. Murray, J. McDonald, and P. Strooper.
Specification-based class testing with ClassBench.
In Asia-Pacific Software Engineering Conference (accepted), 1998.
[20] M. D. Smith and D. J. Robson. A framework for testing object-oriented programs. Journal of Object- Oriented Programming, 5(3):45–53, 1992.
[21] J. M. Spivey. The Z Notation: A Reference Manual.
Prentice Hall, 2nd edition, 1992.
[22] S. Stepney. Testing as abstraction. In ZUM' 95, volume 967 of LNCS, pages 137–151. Springer-Verlag, 1995.
[23] P. Stocks. Applying formal methods to software test- ing. PhD thesis, Department of Computer Science, The University of Queensland, 1993.
[24] P. Stocks and D. Carrington. A framework for specification-based testing. IEEE Transactions on Software Engineering, 22(11):777–793, 1996.
[25] C. D. Turner and D. J. Robson. A state-based approach to the testing of class-based programs. Software—
Concepts and Tools, 16:106–112, 1995.
[26] Z notation. ftp://ftp.comlab.ox.ac.uk/
pub/Zforum/ZSTAN/versions/, 1995. Ver- sion 1.2. Committee Draft of proposed ISO standard.
