(Test Oracle) A source to determine expected results to compare with the actual result of the software under test.

3.2 t e s t i n g t e c h n i q u e s

In software quality assurance different categories of techniques exist. Figure 8 presents a schematic overview according to [Lig09]. The first level is the differentia- tion between static and dynamic methods.

Static Dynamic Analyzing Formal Verification Reviews, Inspections,... Examples: Model-Checking, Theorem Proving,... White Box Black Box Control flow Function oriented Data flow Diversifying Statistical

Statement coverage, Branch coverage, Modified Condition/Decision Coverage,... Defs-/Uses - criteria

Functional Classification tree method, State-based testing,...

Mutation Testing, Regression Testing,...

Random Testing,...

Figure 8: Classification of test techniques

3.2.1 Static and Dynamic Techniques

Static analysiscollects information about the system without executing it to find faults within the implementation. Typical methods are: Audits, Reviews, Walk- throughs, and Inspections, which are executed on the source code of the system under test (SUT).

Various tools support static analysis on source code. Those tools examine [Wal01] e.g.:

• Syntactical Information (Complexity, Dependency Graphs) • Semantical Information (Anomalies)

• Lexical Information (Length of Procedures)

Thus, static analysis focuses on complexity and semantical faults.

In formal verification, properties are defined on the basis of the system require- ments or specification that the system must satisfy. For each property, formal

3.2 testing techniques 35

verification checks whether it holds or not. If the property does not hold, a situation is shown in which this property is violated. If all properties hold, it is proved that the system meets the requirements formulated by the properties. Complete formal verification of a system is rarely performed in Software Engineering, since the extraction of the system properties might be very time consuming. Nevertheless, the industrial importance and awareness of formal methods has increased during the past decade and is growing [Tre99]. Well-known examples of formal verification are Theorem Proving and Model Checking [Lig09].

Dynamictesting of software involves the selection of at least one software testing technique to:

• generate test cases • execute test cases

• compare the result of the test execution with the expected system behavior defined in the system specification

A tester always starts with the construction of a formal or informal (depending on the test strategy) model that captures certain properties e.g. behavior of the SUT. This model is then used for test case generation, which can then be executed in the SUT.

Within this thesis, we intend to test an SPL without testing each possible product individually. Since our domain of research is more or less the embedded domain, the major advantage of testing is that testing can take Hardware/Software interre- lationships into account, including effects on the environment and safety [Lig09]. Since testing is a dynamic technique, we focus on dynamic approaches.

Dynamic Software Testing can be either white box, black box or a combination of both. White box approaches take the internal structure of the SUT into account. Thus, the test engineer requires information about the internal behavior, the logic, and the structure. Structural testing is an often-used synonym for white box testing in the testing community. This information is typically presented via control flow and data flow graphs of the SUT. The control flow graph represents possible execution paths within the SUT, whereas the data flow graph describes the flow of data and relationships between variables in the SUT. The control flow and the data flow graph can both be used for test case generation [Lig09].

Black box testing techniques test the execution of the SUT against its require- ments or specification. Therefore, it is also called specification based testing. The test engineer uses only the requirements or the specification for testing, without know- ing the internal structure of the SUT. In abstract terms, valid and invalid inputs are used to check whether the expected output has been calculated. Strategies such as classification trees [GG93] and combinatorial testing [Bei90] provide heuristics to reduce the space of input parameters. On the one hand, this technique has the advantage of being close to realistic conditions but, on the other hand, one

important disadvantage is the lack of internal information, which is useful for generating tests and for fault tracing. For example, testing against the specification of a system does not guarantee that every line of code or part of the system is tested adequately.

Cai et al. recommend combining black box and white box techniques, since the correlation between fault detection and code coverage depends on the black box coverage criteria [CL05].

3.2.2 Test Levels

Testing can take place in every phase of the development process. These phases extend from requirement analysis to the implementation. A very prominent model representing the different phases of the development process is the V-Model [dIB97] shown in Figure 9. On the left, the development process is presented, whereas on the right, the location of the corresponding test levels is shown. The development process is described top-down, starting at the very left with the requirements analysis. On the basis of the requirements, the system specification is defined followed by a design phase. Afterwards, the system is split into units that are then implemented. The test levels are described bottom-up, starting with Unit Testing, which tests the different units of the system e.g. classes. The components of the design of the system are then tested via Integration Testing. The entire product is then tested against the system specification during system testing. Finally, the stakeholder requirements are tested via Acceptance Testing.

Figure 9: V-Model

3.3 m o d e l-based testing

Model-based testing is derived from the concept of developing software based on models. This means explicitly modeling the structure and behavior of the system to a certain extent by using models on the basis of (semi-)formal modeling approaches.

3.3 model-based testing 37

The difference to non-model-based approaches is achieved by replacing the informal model with an explicit representation [PP04].

Definition 19(Development Model). A model represents the behavior and the structure