Test Execution and Test Report

The test execution does not demand any further effort other than the simulation of any SL model. The test assessment is already included in the test design, thus the verdicts are immedi- ately obtained. This is possible due to the existence of the test oracle and arbitration mechanism in the TSpec unit. Additionally, the quality metrics for evaluating the test model are calculated and the test report is generated after the test execution (see Chapter 7).

Although producing the reports is rather an implementation issue, its basic design has been pro- vided in this work and reports have been generated for the case studies presented in Chapter 6. The MATLAB® Report Generator™ [MRG] and Simulink® Report Generator™ [SRG] are used for defining and generating compact, customizable documents automatically after test execution. They include the applied test data, their variants, test cases, test control, test results, and the calculated quality measures. The presentation form is constituted by a number of tables, plots of signals, graphs, verdict trajectories, and their textual descriptions [Xio08]. The generic design of a report template enables it to be adapted to the actual test model and SUT configura- tion. Technically, both report generation and test system need to be coordinated. Many func- tions contribute to the report contents depending on the available models.

Before generating the test reports, the test results have to be recorded in the ML workspace. Also, the verdicts need to be ordered according to the delays they were identified with during the test so as to provide the test results in a correct manner.

5.8 Related Work

The model-based test approaches have already been reviewed in Chapter 3. Similarly, related work on properties of signals has been discussed in Section 4.5. However, a few issues deserve a special attention since they either contribute to the results achieved in this thesis or constitute the ongoing efforts that are based on the experience gained here.

5.8.1 Test Specification

Time Partitioning Testing (TPT) [Leh03] is the primary example using the concept of SigF in an evaluation system. Although the test assessment is mainly offline – due particularly to the real-time constraints at the HiL level – TPT concepts are a basis for the TSpec unit described in this thesis.

Similarly, the Classification Tree Method [GG93] illustrates how to construct test stimuli sys- tematically. Even if the realization proposed in MTest [Con04b] enables test data to be only created manually, the concept constitutes the fundamental principles according to which the test data generator embedded in TDGen unit is built.

Also, the synchronization mechanism for the test evaluation system of MiLEST realized in [MP07] is a considerable contribution to this thesis. As discussed in Chapter 3, the transforma- tions of [Dai06] contribute to the MiLEST automation. [SCB, EmbV] help to conceive of the TSpec. [CH98, WCF02] underline the test evaluation problems. [SLVV] gives the technologi- cal solution for building the prototype of MiLEST.

Further on, the progress achieved in MiLEST is the foundation for the ongoing efforts towards UML Testing Profile for Embedded Systems (UTPes) [DM_D07, Liu08].

5.8.2 Transformation Possibilities

The background knowledge for the MiLEST transformations has been gained from the experi- ence on Model Driven Architecture (MDA) artifacts [MOF] applied in the context of testing [ZDS+05, CBD+06, GCF06, Dai06]. This practice enabled the creation of generic transforma- tion functions that may be applied to any model39.

The reasoning on the MDA-related transformation for testing proposed by [DGN04, Dai04, Dai06] may be followed for the framework provided in this thesis in an analogue way. Figure 5.30 shows a layered metamodel hierarchy applicable to MiLEST. It has been resigned from developing the detailed metamodels for both ML/SL/SF and MiLEST since the direct ML- based solution appears to be feasible and performs well. The continuation of this topic, e.g., along the application of Query/View/Transformation (QVT) techniques is, however, not ex- cluded from the future research as indicated in [ZSF06].

SL/SF Metamodel MiLEST TSpec Metamodel MiLEST TDGen Metamodel Transformation Rules Transformation Rules SL/SF Model MiLEST TSpec Model MiLEST TDGen Model Transformations Transformations M3: Meta-metamodel M0: Data M2: Metamodel M1: Model

<<instance of>> <<instance of>> <<instance of>>

<<instance of>>

<<instance of>> <<instance of>>

<<instance of>> <<instance of>> <<instance of>>

SL/SF

Instance TSpec InstanceMiLEST

MiLEST TSpec Instance including TDGen MOF Model

<<instance of>>

Figure 5.30: The Metamodel Hierarchy of MiLEST.

5.9 Summary

In this chapter, the concepts of testing in MiLEST have been introduced and explained follow- ing its development phases. Technically, MiLEST extends and augments SL/SF for designing and executing tests. It bridges the gap between system and test engineers by providing a means to use the SL/SF language for both SUT and test specifications, similarly to the way [Dai06] did for the Unified Modeling Language [UML] world. Moreover, it allows for the reuse of sys- tem design documents for testing. By that, the test development is early integrated into the software production.

This chapter adresses the third set of the research questions given in the introduction to this the- sis as well. It has been illustrated that the test development process can be automated by appli- cation of test patterns and transformations. In particular, the test data and their systematically selected variants can be generated automatically from the formerly designed test specification. Then, the test control patterns can be applied. The manual workload during the test design phase cannot be fully excluded, though. The concrete VFs must be added by the test engineer, even if many hierarchically organized patterns for the test specification ease this process con- siderably.

Regarding the test execution, the test evaluation runs automatically on the fly, which allows for an immediate analysis of the test results. Test reports are obtained automatically as well.

In particular, in this chapter, the classification of signal features presented in Chapter 4 has been recalled to describe the architecture of the test system. In Section 5.1, different abstraction lev- els of the test system have been provided. They were denominated by the main activities per- formed at each level. The test harness level includes patterns for TSpec, TDGen, and test con- trol. Then, the test requirements level has been followed by test case and validation function levels. Afterwards, the feature generation and feature detection have been elaborated on. In Ap- pendix C, an overview of all the hierarchy levels is given. Also, different options for the speci- fication of a test have been reviewed, revealing the challenges and limitations of the test design. The importance of the test evaluation has been emphasized. Furthermore, principles for the automatic generation of test data have been presented. By means of the concrete generic trans- formation rules, the derivation of test signals from the VFs has been formalized. Similarly, the generation of signal variants has been investigated. Combination strategies for test case con- struction have been outlined and sequencing of the generated variants at different levels has been reported. The concepts of reactive testing and of test control have been summarized too. Section 5.6 presents considerations on the integration level testing. An advantage of using the ML/SL/SF framework is the possibility to execute both the system and test models. Thus, in Section 5.7, the test execution and test reporting have been discussed.

Finally, in Section 5.8, related work on test design, transformation approaches, and ongoing work towards UTPes [DM_D07, Liu08] have been elaborated.