RQ2: Accuracy

For our approach to be effective, the tool has to be able to identify all constraint violations in the input and accurately apply the oracle without reporting any false positives. For each of the 32 test cases, we know from the test documentation what violations of the input constraints should be reported. We expect no violations of the oracle constraints because the system is correct with regard to the test suite based on the manually written system tests.

When we validated the 32 test cases using our tool, we found that every expected violation of input/configuration constraints was correctly reported with no false positives. We also found, as expected, that no violations of the oracle constraints were reported. Because all the constraints are checked on each test case, any number of test cases can be added to the test suite without the need to manually check the expected output or write assertions for the new test cases (the current practice). This reduces the resources and effort needed to test the system more extensively.

When we validated the real transmission files, we found that in some files many input constraints were violated. Investigating these violations revealed that some of the specifications of the input file were changed by the client but had not yet been implemented in the system. For example, a constraint on ISPs (modelled in Fig. 3.2) specifies that their length should be a multiple of four. This constraint was violated in the real transmission file as the client decided to remove this restriction causing a violation to be reported for each packet that violated this constraint. This change was discovered by SES through running the real transmission on the DAQ system and then manually examining the output log files. This shows that our input data validation approach could help identify changes in specifications without the need to execute the transmission file on the system, which takes considerably more time and effort than input data validation due to the need to analyse many test case failures, as opposed to a subset of violated constraints.

Answer to RQ2: The results of our empirical evaluation show that our approach is accurate in validating test cases and applying the oracle. The results also show that our approach is able to identify implicit changes in specifications (from the client) of the input file and the DAQ system without the need to execute any test cases on the DAQ system.

4.2.4

Threats to validity

Internal threats

The internal threats to validity in this study are related to the test cases and transmission files used to evaluate the system. We used all the test cases and transmission files provided by the system testers to avoid experimenter bias.

External threats

The external threats are related to the choice of case study system and the ability to generalise results. Although we only used one system in the case study, it is representative of data processing systems. Our results might only be relevant in this application domain; nevertheless, this domain is important and widely used. Many other types of systems have similar characteristics as they process very complex inputs, detect input errors, and extract data. In other words, our approach might also be applicable to any system where the complexity lies in the input, output and their mappings.

4.3. Conclusion

Construct threats

For studying scalability, we used the execution time of the test validation and applying the oracle processes. Though execution time depends to some extent on the content of the files, and not just their size, the used transmission files are not only real but representative and, in any case, we are mostly interested in the order of magnitude of the processing, not exact figures.

4.3

Conclusion

In this chapter, we evaluated our proposed automated test validation and oracle approach for data processing systems with complex input structures and mappings between input and output. Many systems share these characteristics, including DAQ systems common in the satellite communications industry. In such systems, generating valid test inputs is challenging and checking the output manually is time-consuming and error-prone. Requirements change on a regular basis and the input and output files are large and complex. Our approach is driven by models of the input and output structure and content, and to support it, we defined a specific modelling methodology using UML class diagrams and OCL constraints as a notation. We developed a tool to automate our approach and evaluated the approach on a real industrial DAQ system. Though there is substantial work in the area of Model- Based Testing (MBT) (as reported in related works, Chapter 8), none of the existing approaches match the needs for DAQ systems and other systems with similar characteristics.

Our empirical evaluation showed that we were able to properly validate input data and automate the test oracle by correctly identifying failing and successful test inputs, as well as effectively val- idating real transmission files. Additionally, the results of our empirical evaluation show that the approach is scalable as the input and oracle validation process executed within reasonable times on real transmission files for a real satellite DAQ system. Though input files can be validated in a few minutes, it can take up to 50 minutes for applying the oracle. The main reason is that transmission files in our case study generated large numbers of violations due to changes to specifications from the client that had yet to be addressed. Furthermore, oracle constraints are more expensive to check as they are significantly more complex, though such durations are fine in practice since such oracle vali- dation processes can be run in batch mode, at night for example. Furthermore, results on existing test suites show that the level of abstraction of the model is not an impediment to the accurate evaluation of oracles. In terms of scalability, the relationship between execution time and input file size is linear, suggesting that much larger files can be handled in the future using our approach.

Chapter 5

Automatic Test Input Generation

In this chapter, we focus on the problem of testing the correct behaviour of data processing systems in the presence of faulty input data. The proper identification of faulty input data is particularly impor- tant in our case study since a satellite transmission is often subject to alterations that may introduce faults, such as channel noise, which should be properly handled to prevent the generation of erro- neous results. This problem generalises to other types of data processing systems. For example, web crawling engines, which process complex data structures (i.e. web pages) and need to determine the presence of faulty data (e.g. tags) not properly closed (a common mistake of novice web developers and bloggers).

Most model-based approaches that relate to the problem above focus on generating data structures for unit testing [Boyapati et al., 2002, Senni and Fioravanti, 2012]. In addition, the few approaches that focus on system testing and generate faulty data are based on Context-Free Grammars (CFGs) and, as a result, cannot generate data that presents complex relationships between the data fields [Hoffman et al., 2009, Zelenov and Zelenova, 2006].

The approaches that are particularly appealing for automatically testing data processing systems for input data faults are the ones based on the adoption of mutation operators that alter existing test inputs to generate faulty data [Shan and Zhu, 2009, Bertolino et al., 2014, De Jonge and Visser, 2012]. The main limitation of these approaches is that the mutation operators are specific for the particu- lar kind of input data used by the SUT. In contrast, we would like software engineers to define a fault model, capturing their experience of the domain, and then provide a way to tailor generic mutation operators to implement this fault model on a specific data model of inputs and outputs.

This chapter highlights the following research contributions:

1. A model-based technique that automatically generates faulty test inputs, by relying upon generic mutation operators that alter data collected in the field.

2. The use of UML stereotypes and OCL queries to configure the mutation operators to implement a fault model of the SUT.

3. An empirical evaluation on an industrial data processing system to evaluate the effectiveness of our proposed approach.

Chapter 5. Automatic Test Input Generation

1. Data Modelling

3. Data Mutation Data Model

(class diagram +

OCL constraints) 2. Data Loading

Field Data

Data Model Instance matching the Field Data

4. Data Writing Mutated

Field Data

3.2 Identify the class instance/feld to mutate. 3.5 Apply the mutation operator.

3.6 Update affected attributes annotated with the stereotype «Derived». 3.4 Select mutation targets according to OCL queries.

5. System Execution

6. Output Validation

Mutated Data Model Instance

3.3 Select a mutation operator.

Outputs

3.1 If stopping condition is satisfed, terminate; otherwise, proceed.

Violated Constraints

Legend:

1. Step Step automated by the technique 1. Step Step performed by the software engineer

Data fow Loop

Figure 5.1. Steps for generating complex and faulty test data.

The remainder of the chapter proceeds as follows. Section 5.1 describes the data mutation tech- nique presented in this chapter. Section 5.2 details the data mutation operators that are used to mutate field data. Section 5.3 describes how mutation operators can be configured to comply with the fault model of the SUT. Section 5.4 describes how we enhanced the modelling methodology to enable automatic data mutation. Section 5.5 provide details about the data mutation strategies considered in this chapter. Section 5.6 presents the empirical results obtained. Finally, Section 5.7 concludes the chapter.

5.1

Description of the Approach

The mutation approach requires two inputs: field data and a data model (i.e. a class diagram modelling input and output data, annotated with UML stereotypes and constraints, which in our case are written in the OCL). Field data is used to generate new test inputs by means of six different data mutation operators that alter existing data on the basis of its data model. The data model drives the usage of the generic mutation operators in order to implement the fault model specified for the SUT. The characteristics of the fault model are captured by means of UML stereotypes and OCL expressions. UML stereotypes are used to select the data fields to mutate, the kind of mutation operators to apply, and the data fields to update after mutation to preserve data invariants. OCL expressions are used to configure mutation operators by selecting the targets of the mutation through appropriate OCL queries.

Further, the technique makes use of automated test oracles that, in our context, consist in the detec- tion of faulty inputs that remain undetected by the SUT. Oracles work by checking OCL constraints that capture the relationships between faulty inputs and expected outputs, hereafter input/output con-

5.1. Description of the Approach

straints. These constraints are relatively simple to define, since they indicate the output expected (e.g. an error message) in the case of invalid input data.

The approach works in six steps: data modelling, data loading, data mutation, data writing, system execution, and output validation. Fig. 5.1 provides an overview of the approach.