Testing strategies and approaches

6.4.5.1 Introduction

For the above test objectives, different testing strategies can be used. Some testing strategies focus on the method used for the selection of input data. For both white box and black box techniques, the selection of input data and objectives of the test cases drives the testing activities to be performed.

Thus black box testing is testing against the specification and to better discover the part of the specification that was not fulfilled. White box testing is testing against the implementation and to better discover the part of the implementation that is faulty. In order to fully test a software product both black and white box testing should be used.

White box testing can be planned and produced based on low level software functionality (e.g.

pseudo-code). White box testing is much more laborious in the determination of suitable input data (when not using a tool) and the determination if the software is or is not correct.

Test planning should start with a black box test approach as soon as the specification is available.

White box testing is a test case design method that uses the control structure of the procedural design to derive test cases. Test cases can be derived depending on the criticality of the software (see ECSS-Q-ST-80C requirement 6.2.3.1) and for example to:

a. guarantee that all independent paths within a module are exercised at least once.

b. exercise all logical source code structure decisions on their true and false sides.

c. execute all loops at their boundaries.

d. exercise internal data structures to ensure their validity.

For dependability and safety testing, there are more specific techniques to be highlighted, being either white box or black box. Among these techniques, the following ones can be highlighted:

a. Fault injection b. Stress testing

c. Equivalence class and input partitioning d. Boundary value testing

6.4.5.2 Testing techniques 6.4.5.2.1 Fault injection

Fault injection is focused on reducing the impact of any residual fault by first determining how badly the software can behave if it is faulty and see how effective its fault-tolerance mechanisms are.

Potentially specific software failures can be removed; though it is a strategy still very much hardware- fault-injection oriented handling the software product as a black box. Fault injection technique can be used to support robustness testing objectives.

Nevertheless, fault injection is recognized as a key element in the assessment and validation of critical software systems, as it is a practical approach to perform stress testing, i.e., raising conditions to trigger rarely executed software operations such as error handling and recovery. Fault injection is used to assess the FDIR mechanisms of any software system. Fault injection can be seen as a special case of testing, with faults becoming the main input in addition to its outputs. Fault injection is quite effective to spot the “interesting” faults, i.e., software faults that have a high probability of escape test case design. It is this inherent ability to trigger unexpected system behaviour that places fault injection technology as a definitive “should have” for achieving more confidence on accomplishment of dependability and safety requirements of critical software.

Fault injection should be used in a more comprehensive way where the fault models to be used address both hardware and software faults.

Hardware faults or software faults can be supported by three fault injection techniques:

1. Use of hardware injection support is a unique way to assess the containment and error propagation into low-level (software directly controlling hardware) critical software, and to achieve evident confidence that dependability and safety requirements are fulfilled.

2. Use of compile time software injection support, e.g. by use of mutation or fault seeding techniques. These two white box techniques are done with intrusion, which can be a disadvantage.

These two techniques are performed by inserting or changing the original source code. They evaluate the software product from two different perspectives: the first one intended to locate faults and the second one intended to evaluate the effect of single changes to the original code. These techniques as modifying the source code, are to complement black box testing, and should be carefully used since in the insertion or change of the original code, other non-intended faults can be introduced. To be effective, these two techniques need good automated tools and significant amount of human analyst time and good insight of the software.

3. Use of run-time software injection support, e.g. by modifying the content of the memory, register, etc.

The use of fault injection is based on skilled personnel and good automation tools, which can be a disadvantage.

6.4.5.2.2 Stress testing

Stress or avalanche testing are black box testing, intended to burden the test object with an exceptionally high workload in order to show that the test object stands normal workloads easily.

Under these test conditions the time behaviour of the test object can be evaluated. The influence of load changes can be observed. The correct dimension of internal buffers or dynamic variables, stacks, etc. can be checked. There are a variety of test conditions which can be applied for avalanche stress testing. Some of these test conditions are:

a. if working in a polling mode then the test object gets much more input changes per time unit as under normal conditions;

b. if working on demands then the number of demands per time unit to the test object is increased beyond normal conditions;

c. if the size of a database plays an important role then it is increased beyond normal conditions;

d. influential devices are tuned to their maximum speed or lowest speed respectively;

e. for the extreme cases, all influential factors, as far as is possible, are put to the boundary conditions at the same time.

Under these test conditions the time behaviour of the test object can be evaluated. The influence of load changes can be observed (these tests are sometimes called volume testing). Throughput analysis is considered part of this king of stress tests.

Requirements for the usage of resources such as CPU time, storage space and memory can be subject of these resource stress tests. The best way to verify these kinds of requirements is to allocate these resources and no more, so that a failure occurs if a resource is exhausted. If this is not suitable (e.g. it is not always possible to specify the maximum size of a particular file), alternative approaches are to:

a. use a system monitoring tool to collect statistics on resource consumption;

b. check directories for file space used. The correct dimension of internal buffers or dynamic variables, stacks, etc. can be checked.

As part of the main advantages of stress testing is that it is often the only method a) to determine that certain kind of systems are robust when maximum numbers of users are using the system, at fastest rate possible (e.g., transaction processing); and b) to identify that contingency actions planned when more than maximum allowable numbers of users attempt to use system, when volume is greater than allowable amount, etc.

One of the disadvantages is that is requires large resources.

6.4.5.2.3 Input partitioning

This test technique is intended to test the software using a minimum of test data. The test data is obtained by selecting the partitions of the input domain to exercise the software.

This testing strategy is based on the equivalence relation of the inputs, which determines a partition of the input domain (orthogonal groups- FTA can help to select inputs where to focus on robustness issues).

Test cases are selected covering all the partitions previously specified. At least one test case is taken from each equivalence class.

There are two basic possibilities for input partitioning which are:

a. equivalence classes derived from the specification – the interpretation of the specification can be either input orientated, for example the values selected are treated in the same way, or output orientated, for example the set of values lead to the same functional result.

b. equivalence classes derived from the internal structure of the program – the equivalence class results are determined from static analysis of the program, for example the set of values leading to the same path being executed.

Equivalence classes can be defined according to the following conditions:

a. If an input condition specifies a range, one valid and two invalid equivalence classes are defined.

b. If an input condition requires a specific value, then one valid and two invalid equivalence classes are defined.

c. If an input condition specifies a member of a set, then one valid and one invalid equivalence class are defined.

d. If an input condition is Boolean, then one valid and one invalid equivalence class are defined.

Test cases from equivalence partitioning are complemented with test cases from Boundary Value Analysis and, applied in conjunction with other test practices to ensure the specified test coverage.

Equivalence partitioning used to test dependability and safety of software focuses the test cases on the limit values or ‘out of bound’ values of each class identified.

The advantage of Test Cases from Equivalence Partitioning is to analyse that program behaves correctly for any class of input by selecting a representative value, reducing the total number of test cases that are developed.

No significant disadvantages are identified in the application of Test Cases from Equivalence Partitioning.

6.4.5.2.4 Boundary Value testing

The purpose is to provide test cases to detect and remove faults occurring at parameter limits or boundaries. The input domain of the program is divided into a number of input classes. The Test Cases should cover the boundaries and extremes of the classes. The tests check that the boundaries of the input domain of the specification coincide with those in the program.

Test cases from boundary value testing are complement with test cases from equivalence partitioning and applied in conjunction with other test practices to ensure the specified test coverage.

The use of the value zero, in a direct as well as in an indirect translation, is often error-prone and demands special attention:

a. zero divisor;

b. blank ASCII characters;

c. empty stack or element list;

d. full matrix;

e. zero table entry;

f. maximum or minimum values;

The boundaries for input have a direct correspondence to the boundaries for the output range. Test cases should be written to force the output to its limited values. Consider also if it is possible to specify a test case which causes the output to exceed the specification boundary values. Extreme values depend on the structure of the data.

If the output is a sequence of data, for example a printed table, special attention should be paid to the first and last elements and to lists containing none, 1 and 2 elements.

Examples of types of fault detected with the use of this technique:

a. Calculation faults.

b. Array size.

c. Null pointers.

d. Loop iterations

The advantages of the test cases definition from the boundary value testing, is that it analyses that the program behaves correctly for any permissible input or output

No significant disadvantages can be highlighted to the test cases definition from the boundary value testing in itself, but for programs with many types of input, all combinations of input cannot be tested

and therefore problems resulting from unexpected relationships between input types cannot be identified.

Table 6-2 summarizes the suitability of each of the above testing techniques to cover the different testing objectives described at the beginning of this annex.

Table 6-2: Relation between the testing objectives and the testing strategies

Testing techniques Test objective

Interface Robustness Performance

Fault injection X

The marked cells of the table mean that the testing technique suits best to achieve the respective testing objective. This does not deny that other techniques can contribute to achieve the other objectives.