Test Development Process for the Proposed Approach

The signal generation and signal evaluation in isolation, no matter how sophisticated, neither test the system automatically nor make testing systematic or completed. These activities as such should be embedded in a clear and well-defined test process. Moreover, an appropriate frame- work has to deliver easy means for test specification, generation, and execution. For these rea- sons a method specific test development process is introduced in this section and a hierarchical architecture of the resulting test system in Chapter 5. Both are facilitated by application of test patterns in different constellations.

The origin of any test specification is the document that specifies the SUT requirements. Usu- ally, they are available in a textual form. System requirements are often hierarchical, starting from high level, down to concrete technical specification. In some cases formalized versions are provided. From such textual documents, test requirements (here also called test objectives) should be derived. According to the general test process these can be classified as a set of the

abstract30 test scenarios. These test scenarios can be described by a conditional form which re- lates incoming SUT stimulation to the resulting SUT behavior by IF–THEN rules (4.8).

A technically-oriented MiLEST test development process proposed in this thesis is shown in Figure 4.36. The MBD paradigm assumes that the SUT model is already available and that the input/output interfaces are clearly defined and accessible.

Besides the analysis of the SUT specification, a proper functional, dynamic testing also requires a systematic selection of test stimuli, appropriate test evaluation algorithms, and obviously a test execution or simulation environment. Thereby, if the above assumptions hold, pattern for the generation of test harness31 model can be applied to the SUT model as denoted by step I in

Figure 4.36. This is done automatically with a MiLEST transformation function, giving an ab- stract frame for test specification. This phase together with the definition of IF-THEN rules is called the test design. Further on, the test specification and test implementation phase is carried out in step II, where the test definition in MiLEST is concretized based on the test require-

ments. The test engineer manually refines the test specification using the concept of validation function patterns, which include the test scenarios. This issue has been already mentioned in Chapter 3 and will be explained later in depth too. Afterwards, in step III, structures for test

stimuli and concrete test signals are generated. This step occurs automatically with application of the transformations. The test control design can be added automatically too. In that case, step

IV would be omitted. However, if the advantages of the test reactiveness are targeted, it should

be refined manually. Finally, in the test execution and test evaluation phase in step V, the tests

(i.e., test cases) may be executed and the test results obtained in the form of verdicts. At the same time the quality of the produced test system specification is also assessed.

SUT as a Model

automatic generation – step I

Test Harness Pattern Application

manual refinement automatic generation

Test Data Generation

manual refinement

Test Control Generation Test Specification – step II – step III – step IV automatic execution Verdicts Analysis – step V

Figure 4.36: Test Development Process.

30 _{An abstract test scenario means that it is valid for a certain generic description of the SUT behavior, without including the}

concrete data.

31 _{In [ISTQB06] a test harness is defined as a test environment comprised of stubs and drivers needed to execute a test. In}

this thesis, the test harness pattern refers to the design level and is specified as an automatically created test frame includ- ing the generic hierarchical structure for the test specification. Together with the test execution engine (i.e, SL engine) it forms a test harness.

In Figure 4.37, a generic pattern of the test harness is presented. The test data (i.e., test signals produced for the test cases) are generated within the test data generator shown on the left-hand side. The test specification, on the right-hand side, is constructed by analyzing the SUT func- tionality requirements and deriving the test objectives from them. It includes the abstract test scenarios, test evaluation algorithms, test oracle, and an arbitration mechanism. The structural details and functionalities of the corresponding units will be elaborated in the next chapter32.

SUT test reactiveness InOut Bus Test Specification Verdict Test Control Test Data Generator

Figure 4.37: A Test Harness Pattern.

The test specification is built by applying the test patterns available in the MiLEST library. It is developed in such a way that it includes the design of a test evaluation as well – opposite to a common practice in the automotive domain, where the test evaluation design is considered last. Afterwards, based on the already constructed test model, the test data generators are retrieved. These are embedded in a dedicated test data structure and are derived from the test design automatically. The generation of test signals variants, their management, and their combination within a test case is also supported, analogous to the synchronization of the obtained test stim- uli. Finally, the SUT model stimulated with the previously created test data is executed and the evaluation unit supplies verdicts on the fly.

The first step in the test development process is to identify the test objectives based on the SUT requirements. For that purpose a high-level pattern within the test specification unit is applied. The number of test requirements can be chosen in the graphical user interface (GUI) that up- dates the design and adjusts the structural changes of the test model (e.g., it adjusts the number of inputs in the arbitration unit). The situation is illustrated in Figure 4.38.

32 _{Parts of the test process have been published in [ZSM07b], the test evaluation in [ZSM06, MP07], the test signals genera-}

tion in [ZMS07a, ZXS08], whereas the test control and reactive testing in [Zan07]. This thesis is the most up to date and presents the most current progress.

a)

b)

Figure 4.38: A Pattern for the Test Requirement Specification. a) Instantiation for One Test Requirement.

b) Instantiation for Three Test Requirements.

Next, validation functions (VFs) [ZSM06, MP07] are introduced to define the test scenarios, test evaluation, and test oracle in a systematic way. VFs serve to evaluate the execution status of a test case by assessing the SUT observations and/or additional characteristics/parameters of the SUT. A VF is created for any single requirement according to the conditional rules:

set assertions THEN set ons preconditi IF (4.8)

A single informal requirement may imply multiple VFs. If this is the case, the arbitration algo- rithm accumulates the results of the combined IF-THEN rules and delivers an aggregate ver- dict. Predefined verdict values are pass, fail, none, and error. Retrieval of the local verdicts for a single VF is also possible.

A preconditions set consists of at least one extractor for signal feature or temporally and logi-

cally related signal features, a comparator for every single extractor, and one unit for precondi- tions synchronization (PS).

An assertions set is similar, it includes, however, at least one unit for preconditions and asser-

tions synchronization (PAS), instead of a PS.

VFs are able to continuously update the verdicts for a test scenario already during test execu- tion. They are defined to be independent of the currently applied test data. Thereby, they can set the verdict for all possible test data vectors and activate themselves (i.e., their assertions) only if the predefined conditions are fulfilled.

An abstract pattern for a VF (shown in Figure 4.39) consists of a preconditions block that acti- vates the assertions block, where the comparison of actual and expected signal values occurs. The activation and by that, the actual evaluation proceeds only if the preconditions are fulfilled.

Figure 4.39: Structure of a VF – a Pattern and its GUI.

The easiest assertion blocks checking time-independent features are built following the schema presented in Figure 4.40.

Figure 4.40: Assertion Block – a Pattern.

They include a SigF extraction part, a block comparing the actual values with the expected ones and a PAS synchronizer. Optionally, some signal deviations within a permitted tolerance range are allowed. Further schemas of preconditions and assertions blocks for triggered features are discussed in [MP07] in detail.

A further step in the test development process is the derivation of the corresponding structures for test data sets and the concretization of the signal variants. The entire step related to test data generation is completely automatic by merit of the application of transformations. Similar to the test specification side, the test requirements level for the test data is generated. This is possible because of the knowledge gained from the previous phase. The pattern applied in this step is shown in Figure 4.41.

Figure 4.41: Test Requirement Level within the Test Data Unit – a Pattern.

Moreover, concrete SigFs on predefined signals are produced afterwards. The test signals are generated following a conditional rule in the form (see 4.9):

set s generation THEN set ons preconditi IF (4.9)

Knowing the SigFs appearing in the preconditions of a VF, the test data can be constructed from them. The preconditions typically depend on the SUT inputs; however, they may also be related to the SUT outputs at some points in time. Every time a SigF extractor is present for the

assertion activation, a corresponding SigF generator may be applied for the test data creation.

Giving a very simple example: for detection of a given signal value in a precondition of a VF, a

signal crossing this value during a default time is required. Apart from the feature generation, the SUT output signals may be checked for some constraints if necessary (cf. Figure 4.42). The feature generation is activated by a Stateflow (SF) diagram sequencing the features in time ac- cording to the default temporal constraints (i.e., after(time1)). A switch is needed for each SUT input to handle the dependencies between generated signals. Initialization & Stabilization block

enables to reset the obtained signal so that there are no influences of one test case on another.

Figure 4.42: Structure of the Test Data Set – a Pattern.

The patterns in Figure 4.42 and the concrete feature generators are obtained as a result of the

automatic transformations. The general principle of the transformation is that if a given feature or feature dependency extraction is detected in the source (i.e., preconditions part of a VF),

then the action to generate the target (i.e., feature generator in the test data structure) is per-

formed. A set of transformation rules has been implemented. Afterwards, the concrete test data variants are constructed based on the generators obtained from the transformations. The as- sumption and necessary condition for applying the variants generation method is the definition of the signal ranges and partition points on all the stimuli signals according to the requirements or engineer’s experience. Equivalence partitioning and boundary value analysis are used in dif- ferent combinations to then produce concrete variants for the stimuli.

When a test involves multiple signals, the combination of different signals and their variants have to be computed. Several combination strategies are known to construct the test cases –

minimal combination, one factor at a time, and n-wise combination [LBE+04,GOA05]. Combi- nation strategies are the selection methods where test cases are identified by combining values of different test data parameters according to some predefined criteria. In this thesis, the first two strategies have been used.

A similar sequencing algorithm like for the test data applies for ordering the test cases on a higher hierarchy level while dealing with a number of requirements. This aspect is called test control. A traditional understanding of the control makes it responsible for the order of test cases over time [ETSI07]. It may invoke, change, or stop the test case execution due to the in- fluence of the verdict values coming from the test evaluation. Thus, the test cases are sequenced according to the previously specified criteria (e.g., pass verdict). An extended definition of the test control is considered in Section 5.5.

The test patterns used for realizing a test specification proposed in this thesis are collected in a library and can be applied during the test design phase. In Table 4.4 two patterns are presented,

each of them corresponding to a selected testing activity. Since they will be applied in the case studies part in Section 6.4, their meaning will be explained afterwards. The full collection of MiLEST patterns enabling the entire test system to be built is attached to this thesis in Appen- dix B.

Table 4.4: Illustration of Reasoning about Patterns.

Test Activity

Test Pattern Name

Context Problem Solution

Instance

Test evaluation

Detect SigF Test of a control

unit

Assessment of a control unit behavior in terms of a selected SigF

Test data generation Generate SigF Evaluation of a step response function

Generation of the proper signal to stimu- late a selected feature on the SUT output signal

After the test specification has been completed, the resulting test design can be executed in SL. Additionally, a report is generated including the applied test data, their variants, test cases, test results, and the calculated quality metrics.