T ECHNIQUE : THE ORDERED TEST GENERATOR

The case study generator illustrates a test system which works well against an API that does not require sequential ordering for the tests. But what about testing an API which requires that the functions or methods be called in a particular sequence—where val- ues from one call are passed on to subsequent calls? Sequence-based test systems can also be built using a code generator.

One approach is to use a test data file with an ordered set of data, in which special mark-up can be used to show where data should be stored in variables and passed on to subsequent routines. Here is an example of a test data set for a sequential test generator:

:ret1,add,1,2 5,add,:ret1,2

The file format is a comma-separated file format. The first field is the return value. If the return value is a number (e.g., 5), then the function should return that value. If the return value is specified using a :name, then the return value should be stored in a variable with the specified name.

Orders the arguments properly

TECHNIQUE: THEORDEREDTESTGENERATOR 151

The second field is the function name. In both cases in the example, the function name is add. The remaining fields are the arguments to the function.

Here is the code generated for the input file:

// Test 1

int ret1=add(1,2); // Test 2

if ( add( ret1, 2 ) != 5 ) { return 0; }

To implement the first line of the file, we have run the add function with the argu- ments 1 and 2. The output of the function is stored in the ret1 variable. The second line also calls add, but this time it passes in ret1 and the value 2. We then check the output against the value 5 to make sure we are adding properly.

This is a simplistic example, but it demonstrates the basic technique for building a sequential test data set and lets you see what the resulting code will look like.

7.4.1 Roles of the generator

The roles of the ordered unit test generator are the same as the roles of the unit test case study generator, as described in section 7.3.1.

7.4.2 Laying out the generator architecture

To build the test code that fits within the test framework, you’ll use a tier generator model. We chose the tier model because you are building fully self-reliant code from an abstract model—the test data set. You take the ordered test data set as input. Then, using a set of invocation templates, you build the test code, which sits inside a test framework (e.g., JUnit, CPPUnit, Ruby/Unit). Figure 7.4 shows the block I/O architecture.

In the next section, we will walk through the steps of this ordered generator as it executes.

Generator Invocation_Templates

Test Code Test Data

Test Framework

Figure 7.4

The I/O flow for a sequential test generator

7.4.3 Processing flow

Here is the process flow for the ordered generator:

● Reads the test data input file and stores it locally. ● Creates a buffer that will hold the test code.

● Iterates through each test in order and follows these steps:

❍ Inspects the return value to see if you are going to check against the return

value or store the return value. If you are storing the return value, it runs the template to create the variable and selects the proper template to handle the invocation. It stores the variable creation code, if you created it, in the test code buffer.

❍ Runs the invocation handler that builds the call to the function with the right

arguments and either stores or inspects the output.

❍ Stores the result of the template in the test code buffer.

● Uses a template that provides a framework for the test code and adds the test code

to the framework.

● Stores the final product in the output file.

At the end of the process, you will have test code that allows for sequential API access.

7.5

TECHNIQUE: THE

TEST

DATA

GENERATOR

Testing a function or class well means running a lot of tests on it with a variety of data. Ideally you would use data that stresses the target code in a number of directions. The data would test the positive function of the API by sending in valid data. In addition, it would test negative conditions by sending in bad data or by calling the API out of sequence. The mechanism should also test the API under load by sending in large amounts of data.

The test data generator randomly generates test data that can then be used to test against the target class or function. Ideally, you would create the random output once and store it so that it could be predictably used over and over again against the same target. Randomly generating data for each test is troublesome for two reasons. First, you will not get the same coverage of test cases each time. Second, you will get unpre- dictable errors as a result of differing test data each time the test is run.

7.5.1 Roles of the generator

In order to define the physical architecture of the test data generator, you have to start by defining the role of the generator and declaring with certainty what the generator is responsible for—and what it is not responsible for. It has one main responsibility: creating test data sets for the application.

TECHNIQUE: THETESTDATAGENERATOR 153

And here are the tasks this generator is not responsible for: • Building the test data loader.

• Building I/O functions or libraries to read the test data. • Validating test data.

With these guidelines in mind, you can now design the architecture of the generator.

7.5.2 Laying out the generator architecture

You can define the test data generator as a tier generator because it uses an abstract model of the test data and the test definition, along with some data and data templates to create the finished test data.

The generator takes as input a test definition file. This file contains information about how much test data should be created, how many test data types there are, and what each test data type looks like.

To create the data, the generator uses a set of data pools, from which it randomly selects data elements. The generator then merges the input data with data templates, which are used to format the data for output.

The data pools can be sets of names or addresses, or just different types of strings or numeric values—any set of data from which you could derive some interesting tests. The block I/O flow for the test data generator is shown in Figure 7.5.

The next section shows how this generator works.

7.5.3 Processing flow

Here are the process steps for the test generator:

● Reads the test definition and stores it locally ● For each test data type:

❍ Builds enough data for the amount of tests specified by selecting data

randomly from the data pools for each of the fields

● Sends the data to the data templates for output formatting ● Stores the output in the test data file

Random

Generator TemplatesData

Test Data Data

Pool

Test Definition

Figure 7.5

The I/O flow for the generator, which builds data for test sets

7.5.4 Using the test data generator and the sequential test generator together

Figure 7.6 shows how the test data generator can be used in conjunction with the sequential test generator. The test data generator builds test data sets, which are then fed to the sequential test generator to create test code within the test framework.

You could argue that the generators should be merged, but by separating the data generation system from the test implementation system you extend the value of both systems. The test data generator can be used to build data for other systems, such as database loaders, while the sequential test generator can be used to find particular edge cases in either random data sets or test sets that are hand-coded.