Figure 9.3 shows the tool developed to generate test inputs via the mutation of valid field data. The following subsections describe the functionalities developed to implement the tool.
9.3.1
Data loading
The same data loading functionality developed for the Input Validation and Oracle tool of Section 9.2 is used. In this case, the tool requires the System Configuration files and Valid Field Data (unlike the input validation and oracle tool, the inputted transmission data must contain only valid data).
Sometimes the input field data may be too large to fit in memory or we might simply wish to consider only limited chunks of a field data file. For this reason, our parser enables software engineers to specify both the amount of data to load (e.g. 50 CADUs) and the location of the data to load. Software engineers can specify two possible locations: beginning, which indicates that the data is loaded from the beginning of the stream, or random, which indicates that the parser randomly chooses a position in the stream, then identifies the first synchronisation block that follows that position and starts loading the data. Section 2.1 provides an overview of the format of the transmission data.
9.3.2
Data mutation
Given the Model Instance, data mutation applies mutation operators according to the fault model (captured within the Data Model). The tool supports single or multiple data mutations.
With no parameters, the mutation proceeds as follows: a mutation operator is randomly selected and then applied to one of the elements within the Model Instance to which it can be applied (also selected randomly).
Optionally, the Mutation Operator Configuration can be used as follows:
(1) It can specify the mutation operator to be executed against the Model Instance—in this case, one of the elements to which the mutation operator can be applied is selected randomly. If there are no elements mutatable by the operator, an error is returned.
(2) It can specify the mutation operator and the type of element (i.e. the specific class instance type or attribute type) to target—one of the specified elements to which the mutation operator can be
9.3. Test Input Generation
applied is selected randomly. If there are no elements of the specified type that are mutatable by the operator, an error is returned.
(3) It can provide a list of htarget, operatori pairs exactly specifying an ordered list of mutation operators to use and the exact model elements (i.e. the specific class instances or attributes) to mutate. If the operations cannot be completed, an error is returned.
(4) It can provide a list as specified in (3), followed by a final list item that matches the function of list item (1) or list item (2). If the operations cannot be completed, an error is returned.
The Data mutation step returns Mutation Results detailing (1) an ordered list summarising the successful application of mutation operator(s) and their target(s) within the Model Instance or (2) in the event of an error, information pertaining to the attempted operation(s).
9.3.3
Data writing
The generation of concrete Mutated Field Data from the Mutated Model Instance can be handled by using the same solution adopted for data loading. XStream APIs, for example, handle both the loading and saving of information from and to XML files [XStream, 2016]. For the case of SES-DAQ, we implemented a data writing tool that implements the same logic as the parser (i.e. using the same stereotypes introduced in Section 3.4) but for writing data instead of reading it.
Chapter 10
Conclusions and Future Work
This chapter is organised as follow. Section 10.1 summarises the contributions of this dissertation. Section 10.2 discusses potential future work.
10.1
Summary
In this dissertation, we addressed the challenges of testing data processing systems. In data processing systems, inputs are complex and the current practice is that software test engineers have to manually write test cases; they have to carefully handcraft input data, while ensuring compliance with the multiple constraints between data fields to prevent the generation of trivially invalid inputs. Assessing test results often means analysing complex output and log data. Software engineers typically write scripts to evaluate whether the generated outputs for a given input are correct (i.e. the test oracles are written manually); such an approach requires a high development effort per test case and should system specifications change, the scripts must be reassessed to ensure they are still valid.
We presented four approaches to enable the automated testing of data processing systems using model-driven approaches. Each of the approaches addresses distinct and important problems related to testing. These techniques are supported by the data models generated according to our modelling methodology introduced in Chapter 3. Our modelling methodology uses UML class diagrams and OCL constraints as a notation; we also created a profile to define stereotypes to support data parsing and test input generation. The results of our empirical evaluation showed that the application of the modelling methodology is scalable as the size of the model and constraints was manageable on a real data processing DAQ system.
The first approach is a technique for the automated validation of test inputs and oracles based on the application and tailoring of MDE technologies. The results of our empirical evaluation showed that the approach is scalable as the input and oracle validation process executed within reasonable times on real transmission files. For a transmission file of 2 GB, it took less than one hour to apply input data and oracle validation. In terms of scalability, the relationship between execution time and input file size is linear, suggesting that much larger files can be handled in the future using our approach.
Chapter 10. Conclusions and Future Work
for the purpose of robustness testing, by relying upon generic mutation operators that alter data col- lected in the field. An empirical evaluation performed with a data processing system, shows that our automated approach achieves slightly better instruction coverage than the manual testing taking place in practice, based on domain expertise.
The third approach is an evolutionary algorithm to automate the robustness testing of data pro- cessing systems through optimised test suites. The empirical results obtained by applying our search- based testing approach to test an industrial data processing system show that it outperforms the pre- vious approaches based on fault coverage and random generation: higher coverage is achieved with smaller test suites. Furthermore, we show that fitness functions based on models alone (i.e. not re- quiring test case execution) can achieve good coverage results, while significantly reducing test suite size. This is of practical importance as test generation is much quicker and often more practical when no test execution is required.
Finally, the fourth approach is an automated, model-based approach that reuses field data to gen- erate test inputs that fit new data requirements for the purpose of testing data processing systems. The empirical study shows that the proposed approach scales in the presence of complex data structures. In particular, the study shows that the input generation algorithm based on model slicing and constraint solving can produce, in a reasonable amount of time, test input data that is over ten times larger in size than the data that can be generated with constraint solving only. To evaluate the effectiveness of the proposed approach, we generated test inputs to stress software robustness and we measured the code covered when executing the generated test inputs against the software. The results show that the generated test inputs cover most of the source code instructions of the updated software written to implement new data requirements. The generated test inputs also achieve more code coverage than the test cases implemented by experienced software engineers.