Test Generation Method

For each stage of test generation and reduction, we classify the stage into three components. The first component informally explains the process of that stage. The next component gives input and output formats of each stage. The final component gives implementation level details of that stage with algorithms and data structures used in implementation.

gramming language. The main function of the PERL script has steps as in Algorithm 1. Throughout this thesis, we refer to this algorithm as the main algorithm.

Inputs to the algorithm are: a C file (F ), name of the selected func- tion under test (f uncN ame), the list of input variables (Ilist), the list of

output variables (Olist), the initial test suite (T S) and the timeout value

(T Mval). For simplicity of explanation, the algorithm shown here accepts

only a single C file. This, however, is not a limitation of the procedure. The algorithm can trivially be modified to accept multiple files as input. The algorithm first prepares a wrapper function to test the selected func- tion. Next, an executable file is prepared to execute the test suite over the selected function. A loop executes the set of statements of recording pro- gram behaviours (recProgBehaviour), preparing FSMs from program traces (prepareFSMsfromTraces), merging the FSMs to generate a single specifica- tion FSM (mergeFSMs) and generating test data from the specification FSM (genTestDataFromSpec). The loop terminates when no new test data can be generated from the specification with our technique (errorFlag) or when timeout has occurred (lT imeV ar >= T Mval). Once the loop has executed,

we have T S with initial and new test cases. The test suite (T S) is optimized using a reduction criteria (reduceTestSuite) to generate a reduced test suite T Sreduced. The algorithm outputs the test suite T Sreduced.

In practice, prepareFSMsfromTraces and mergeFSMs are part of a single stage. For simplicity, they are explained as separate stages.

Each of these are explained below in detail. 1. Prepare program environment

Preparing a program environment includes preparing a driver function and using the driver function to prepare a program executable.

Algorithm 1 PseudoPERL script for test generation

Input: F, Ilist, Olist, T S, T Mval

driverF unc = prepareProgramDriver (F, f uncN ame, Ilist, Olist) execF ile = prepareExecutable (driverF unc, F )

testGenF lag, lT imeV ar, F SM spec = (TRUE, START, () ) T Snew = T S

while testGenF lag = TRUE do

traceF iles = recProgBehaviour ( execF ile, T Snew ) F SM s = prepareFSMsfromTraces ( traceF iles ) F SM spec = mergeFSMs ( F SM s, F SM spec )

newT estData, errorF lag = genTestDataFromSpec ( F SM spec ) if errorF lag = FALSE then

T S = appendNewTestData( T S, newT estData ) T Snew = newT estData

if lT imeV ar >= T Mval then testGenF lag = FALSE end if

else

testGenF lag = FALSE end if

end while

T Sreduced = reduceTestSuite ( T S ) Output: T Sreduced

Initially, we prepare a driver for the program. This driver simulates an environment to the actual program function. The driver is similar to the program shown in Listing 3.1. Driver contains a reactive loop where input values are accepted, the program function is called and the output is displayed. The driver can be generated with the availability of input and output variables.

Once the driver function is generated, we prepare an executable file with the program and driver function.

Implementation detail

To generate the driver for the program, we need

• input and output variables with their datatypes, and • signature of the program function.

With this information, the driver can be prepared as shown in Figure 4.1. Function prepareProgramDriver from main algorithm produces this driver function. The driver function is written to a separate file. The driver in the figure is for programs in pseudo C language. A similar driver can be prepared for most such languages.

In the initial part of the driver, file pointers and temporary variables are declared. This declaration is with respect to the datatypes. Next, a driverFunction is written which simulates the actual reactive envi- ronment. The driverFunction opens the input test file for reading and executes the reactive loop till all tests in a test file are read. It also calls functions which read inputs and print outputs in a desirable format. Once the driver is generated, an executable file is prepared. We use the GCC compiler to generate the executable file. The function prepare-

Executable calls the GCC compiler with the C files and driver function as input and outputs an executable file.

2. Run initial test suite on the program

The initial test suite T S is executed on the program to produce a set of program runs.

The driver in step 1 is prepared such that each test case is input to the program. The function is executed for that test case. Result, namely the values of output variables, are printed to a file. This process of reading test vector, executing the function and capturing outputs is sequentially performed for each test vector in the test case. This process is repeated for each test case in the test suite.

Each test case produces a program trace or a program run, which con- tains information regarding the program’s run for a set of inputs, exe- cuted in sequence. We prepare this trace as shown in Figure 4.2 The program run has information of the values of input variables and program states in the sequence of execution. First, the initial values of output variables are recorded. This is followed by the values of input variables. Next, the values of output variables after execution of inputs on the function are noted. This pair of related input-output values is recorded for all test vectors in a test case. Execution of all test cases in the test suite result in a set of such program runs.

Implementation detail

The executable generated in the previous stage is used to prepare pro- gram runs. The function recProgBehaviour produces these program runs by running the test suite (T S) over the executable. The Pseu- doPERL algorithm for recProgBehaviour is shown in Algorithm 2.

// T h i s d r i v e r i s p r e p a r e d f o r f u n c t i o n <programFunction >. FILE ∗ f p I n p F i l e ; char ∗ i n p F i l e ; // Temporary v a r i a b l e s p r e p a r e d a s p e r t h e i r d a t a t y p e . <dt1> r e t V a r ; // r e t u r n v a l u e o f program f u n c t i o n <dt2> glbForParam1 ; // p a r a m e t e r s o f program f u n c t i o n <dt3> glbForParam2 ; void d r i v e r F u n c t i o n ( ) { unsigned i n t i t e r a t i o n V a r = 0 ; f p I n p F i l e = o p e n F i l e ( ‘ ‘ i n p F i l e ” ) ; p r i n t f ( ‘ ‘ INITIAL OUTPUTS\n” ) ; r e p o r t O u t p u t s F r o m F u n c t i o n ( ) ; while ( f p I n p F i l e != NULL ) { g e t I n p u t s F o r F u n c t i o n ( ) ;

r e t V a r = <programFunction >(glbForParam1 , glbForParam2 , . . . ) ;

r e p o r t O u t p u t s F r o m F u n c t i o n ( ) ; }

}

void g e t I n p u t s F o r F u n c t i o n ( ) {

f s c a n f ( ‘ ‘ % d %f %d %f . . . \n” ,& g l b I n p 1 , &g l b I n p 2 , & glbForParam1 , &glbForParam2 , . . . ) ; p r i n t f ( ‘ ‘ INPUTS\n” ) ; p r i n t f ( ‘ ‘ % d %f %d %f . . . \n” , g l b I n p 1 , g l b I n p 2 , glbForParam1 , glbForParam2 , . . . ) ; } v o i d r e p o r t O u t p u t s F r o m F u n c t i o n ( ) { p r i n t f ( ‘ ‘OUTPUTS\n” ) ; p r i n t f ( ‘ ‘ % d %f . . . \n” , glbOut1 , glbOut2 , . . . ) ; }

INITIAL_OUTPUTS <O1>_<O2>_<O3>_ ... INPUTS <I1>_<I2>_<I3>_ ... OUTPUTS <O4>_<O5>_<O6>_ ... INPUTS <I4>_<I5>_<I6>_ ... OUTPUTS <O7>_<O8>_<O9>_ ... ...

Figure 4.2: Program Trace Format

Algorithm 2 PseudoPERL function for recProgBehaviour

Input: execF ile, T S traceF iles = ()

for testCase in T S do

opF ile = prepareTraceFile ( )

opF ile = execute ( execF ile, testCase )

traceF iles = addToSetOfFiles ( traceF iles, opF ile) end for

O1,O2,O3 O4,O5,O6 O7,O8,O9 I1,I2,I3 I4,I5,I6

Figure 4.3: FSM of sample program trace

The function recProgBehaviour takes the executable file (execFile) and the test suite (T S) as input to produce a set of trace files traceF iles. Each test case in the test suite is executed over the executable file. The driver function present in the executable file prepares an output trace file recording behavioral information of the test case. This information is captured in the opF ile file. These files are collected in a set which we call traceF iles.

3. For each program run, prepare a specification in the form of a FSM

An individual program run is converted into a FSM. Each output be- comes a state in the FSM and the input becomes the guard on some transition. Each run produces a FSM. To prepare this, we sequentially traverse the program run. The values of initial output form the initial state of the FSM. The value of next occurring inputs form the guard on the transition to the state formed from the following values of out- put variables. This process continues until all the states have been exhausted.

For the program run in Figure 4.2, Figure 4.3 gives the FSM so pro- duced.

Implementation detail

FSMs from program runs. The PseudoPERL algorithm for prepareF- SMsfromTraces is shown in Algorithm 3.

We iterate over the set of program traces to get each program trace. A program trace thus selected is converted into a FSM. For preparing a FSM from a program trace, we read pairs of lines from the program trace and classify every pair as either an input or an output. The first line of the pair denotes whether the pair is an input (INPUTS) or an output (INITIAL OUTPUTS or OUTPUTS). The second line of the pair contains the respective values. In case of input, the values are added to the input array (addToInputArray). In case of output, they are added to the state array (addToStateArray). After every output pair denoted by (OUTPUTS), the sequence of inputs and states are stored in transition array (addToTransitionArray). Thus, after reading the complete trace file, we obtain three array of arrays: inputArray, stateArray and transitionArray. These arrays store the entire informa- tion of the trace file.

Arrays inputArray and stateArray are of the same format. For ex- ample in Figure 4.3, stateArray will be ((O1,O2,O3...), (O4,O5,O6...), (O7,O8,O9...)) and inputArray will be ((I1,I2,I3...), (I4,I5,I6...)). Each element of stateArray will contain an array of output values. Similarly, each element of inputArray will contain an array of input values. For both arrays, the values are added to the respective arrays after per- forming a duplicate check. It may be possible that values to be added as element are previously present. In that case, the new element is not added, but the index of previously present same array element is returned.

Algorithm 3 PseudoPERL function for prepareFSMsfromTraces

Input: traceF iles F SM s = ()

for each runF ile in traceF iles do

inputArray, stateArray, transitionArray = (), (), ()

while (line = readN extLine(runF ile)) not end of file do if line = IN IT IAL OU T P U T S then

opLine = readNextLine(runFile)

prevStateIndex = addToStateArray ( stateArray, opLine ) else if line = IN P U T S then

opLine = readNextLine(runFile)

prevInputIndex = addToInputArray ( inputArray, opLine ) else if line = OU T P U T S then

opLine = readNextLine(runFile)

stateIndex = addToStateArray ( stateArray, opLine )

transitionArray = addToTransitionArray ((prevStateIndex, prevInputIn- dex, stateIndex))

prevStateIndex = stateIndex end if

end while

FSMs = addToFSMs ( F SM S, (inputArray, stateArray, transitionArray) ) end for

of ((s0, i0, t0), (s1, i1, t1) ...), where sx and tx represents index of state

array elements while ix represents index of input array. For example in

Figure 4.3, transitionArray will contain ((0,0,1), (1,1,2), ...).

Thus, these three arrays form a FSM represented in a specific format. Using a combination of these three arrays, one can easily generate a representation as in Figure 4.3. For each trace file, the three arrays are produced and collected in FSMs. The FSMs are output from this stage of the algorithm.

4. Merge individual FSMs to form a single FSM

Each FSM depicts a behaviour of the program for a specific input. The program may show either total identical behaviour or partial identical behaviour on two or more inputs. This identical behaviour can be detected and the corresponding FSMs merged, to reduce redundancy of behaviours in the FSMs. Thus, all FSMs are merged into a single FSM to make it easier to work with a single FSM representing complete behaviour of program for a test suite than many individual FSMs. The individual FSMs are combined into a single FSM by merging states with same values. If two FSMs have a state with same values, then the two states form a single state. All incoming transitions to the two states are now sink into the merged state. Also, all outgoing transitions from the two states have a single source state which is the merged state. Figure 4.4 depicts this state merging process. For simplicity of ex- planation, we represent the state values and input values with q and i respectively. Consider FSM 1 as a FSM produced from a program run and FSM 2 as another FSM from an other program run and state qB and qY to be the same states. The merged FSM is shown in the

qA qB qC FSM 1 qX qY qZ FSM 2 qA qC qX qZ qB-qY Merged FSM i1 i2 i3 i4 i1 i3 i2 i4

Figure 4.4: State merging process

diagram where the states are merged and the transitions are adjusted. All other non-identical states retain their existence in the merged FSM along with their transitions.

Implementation detail

The function mergeFSMs in the main algorithm merges individual FSMs into a single FSM specification. The PseudoPERL algorithm for mergeFSMs is shown in Algorithm 4.

The F SM spec contains the specification in the form of a FSM. Format of F SM spec is same as the format of F SM s, i.e. collection of three

arrays of inputs, states and transitions. To prepare F SM spec, we

merge the F SM s. Thus, we want to merge all respective arrays of inputs, states and transitions into three arrays of inputs, states and transitions which will represent the complete specification.

We pick each singleF SM from set of F SM s and merge it with F SM spec. Initially, F SM spec will be empty and thus will be directly assigned the

Algorithm 4 PseudoPERL function for mergeFSMs

Input: F SM s, F SM spec

for each singleF SM in F SM s do

(inputArray, stateArray, transitionArray) = singleF SM if F SM spec = () then

F SM spec = singleF SM else

setOf SimilarStates = compareFSMsForState( F SM spec, stateArray ) F SM spec = addAllUniqueStates ( F SM spec, setOf SimilarStates ) setOf SimilarInputs = compareFSMsForInput( F SM spec, inputArray ) F SM spec = addAllUniqueInputs ( F SM spec, setOf SimilarInputs ) for each transition in transitionArray do

( source, input, target ) = transition

source = getStateIndexFromFSMspec ( F SM spec, stateArray[source] ) input = getInputIndexFromFSMspec ( F SM spec, inputArray[input] ) source = getStateIndexFromFSMspec ( F SM spec, stateArray[target] ) F SM spec = addTransToSpec (( source, input, target ))

end for end if end for

values of singleF SM . For all further cases, we first determine all pairs of similar states (compareFSMsForState) and inputs (compareFSMs- ForInput) in singleF SM and F SM spec. All unique states and inputs are added to the F SM spec. For each transition in transitionArray of singleF SM , the tuple (source, input, target) is updated with new

index values from similar arrays of F SM spec. This updated tran-

sition is added to F SM spec using function (addTransToSpec). If a particular transition is found to be duplicate, it will not be added by (addTransToSpec).

Iteratively performing this action for all FSMs, we will get a merged FSM in F SM spec. This specification will be used for test generation process in the next step.

5. Generate additional test cases using the merged specification Additional test cases are generated such that each newly generated test case explores unexecuted runs of the program. This is achieved by taking runs that end in a state that has no outgoing transition and extending it by an input test vector that was generated earlier. This will lead to a new transition to either a new state or an existing state. The motivation to choose a terminal state is with the assurity that any input executed from a terminal state will generate a new state or a new transition or both. The specification in the form of a FSM represents the behaviour of the program for a test suite. If we are able to prepare test cases such that new states and transitions in the specification are generated, then we may have new test cases which probe previously unexplored behaviours of the program. If we execute a new test vector from a non terminal state, we might generate another transition to the

same state of the FSM. Thus, we might have two inputs which have same source and target states. This would mostly mean taking the same program path with different inputs, which is unlikely to increase code coverage. Intuitively, it is easier to achieve more program coverage by generating new states in the program. Thus, we target test generation from terminal states than other intermediate states.

For example, referring to the merged FSM in Figure 4.4, states qC and qZ are two states from which there are no outgoing transitions. One such terminal state is randomly selected, say state qC. Also, an input vector from the test suite is randomly selected, say ix. Next, existing test case to reach state qC from initial state is selected and the newly selected test vector of ix is appended to it. So the new test case formed is h...,i1,i2,ix i.

Generating a test case by selecting qB-qY as the selected state, may result in an input such that new transition is from state qB-qY to state qC or qZ. Thus we select a terminal state for test generation.

Implementation detail

The function genTestDataFromSpec generates new test data using FSM- spec. The PseudoPERL algorithm for genTestDataFromSpec is shown in Algorithm 5.

The steps in the algorithm are as per the technique explained above. First a random terminal state (state) is selected (getRandomTerminal- State) from the specification. We used PERL provided rand function to generate random values. Next, a list of input vectors (inputList) to reach state from the initial state is deduced (getListOfInputToRe- achState). A random test vector from the available set of test vec-

Algorithm 5 PseudoPERL function for genTestDataFromSpec

Input: F SM spec

state = getRandomTerminalState ( F SM spec )

inputList = getListOfInputToReachState ( F SM spec, state ) inputSelected = getRandomInputVector ( F SM spec )

newInput = prepareInput ( inputList, inputSelected ) Output: newInput

tors is selected (getRandomInputVector) and is appended to inputList (prepareInput) to produce a new input test case as newInput. This newInput is the test case generated by our technique.

The new test case is executed on the actual code of the program. This program execution produces a new program run. The process of gen- erating a FSM from program run is repeated. This FSM is merged with the existing specification. The new program run produces a new program transition and a new program state in the specification. This step of test generation is repeated until there are no terminal states or the process times out. At the end of this step, a new test suite is generated which is the collection of all test cases generated in this step. 6. Test suite reduction

The test suite generated from step 5 is reduced as follows:

(a) Sort the test suite in descending order with respect to number of test vectors in a test case. This helps us ensure that a test case completely contained within another is removed from the final test suite.

test vectors at the beginning of the test suite and the number of test cases with lesser number of test vectors at the end of the test suite.

(b) Using the sorted test suite regenerate the merged specification as a FSM as in the test generation technique. Each test case is individually executed and FSMs are prepared from their program runs. Then, the individual FSMs are merged as per the merging process explained in the test generation technique. The order of merging the individual FSMs is determined by the order of test cases in the sorted test suite. Thus, the FSMs prepared by the first two test cases are merged first. This new FSM is next merged with the third test case to produce a new merged FSM. This process continues until all test cases in the test suite are exhausted. While merging the test cases, we want to discard all test cases which do not generate a new state or which generate only duplicate transitions between source and target states with different inputs. A test case is not discarded when,

• the test case generates a new state in the FSM, or

• the test case generates a transition to an existing state in the FSM where there is no direct transition between the source and target state.

Considering Figure 4.5, assume that we have an initial merged FSM formed from a test suite and three new test cases TC1,TC2 and TC3. Successive merging of FSMs produced by TC1, TC2 and TC3 with the initial merged FSM produces a new merged