Results and Discussion

Figure 10.2 shows the result of the rst part of our analysis. The goal was to nd out how many fault scenarios are detectable by task-based test case generation. The result shows that 88% of all feature interaction faults could have been detected by task-based test case generation and 12% could have been not.

An example of a fault that could not be found by task-based test case gener- ation is the following: when the ignition is rst set to on, then to o and then set back on before the bus transitions into the sleep mode, the audio functions were inactive even though they should be active. This fault could not be cov- ered because it was caused by internal system behavior (the sleep mode of the

10.3 Results and Discussion 145 Not Covered

12%

Covered 88%

Fig. 10.2. Faults covered by task-based test case generation.

Task Coverage 21% Interleaving Coverage 22% Enable Coverage 4% Specific Sequence 28% Suspend Coverage 25%

Fig. 10.3. Faults per test selection criterion.

bus) which would not be modeled by a task model. Another example of a fault which resulted from a feature interaction was the animation that did not run smoothly enough during the switch from the interactive map to the digital versatile disc (DVD) menu. This fault-scenario would have been modeled in a task model, however, the fault could not be found by an automated test oracle.

Nevertheless, our result clearly indicates that task models are an appropriate means for test case generation. Our previous analyses in chapter 3.2 showed that 40,9% of all severe faults result from feature interactions. In combination with our results from this case study we can conclude that task-based test case generation could cover 30% of all severe faults. This study reveals that task-based test case generation holds huge potential.

The second goal was to analyze, the number of faults found for each test se- lection criterion. The diagram in Figure 10.3 shows the results of the analysis. The diagram shows that Suspend Coverage, Interleaving Coverage, and Task Coverage nd the most faults. 28% of the faults could not be found by the proposed test selection criteria and only 4% could be found by the Enabling Coverage Criterion.

The fact that the task criterion nds such a large fraction of faults is inter- esting, because it is a rather trivial test selection criterion. This results from the fact that a task sequence describes for each active task all modes of the

other tasks. Many faults occurred in situations where a feature should have disabled another feature, but did not. An example of such a fault found by task coverage was that the GUI showed the CD selection dialog every time the voice guidance was active. This shows that unintentional feature interferences can be found by the task coverage criterion as well.

Another interesting result was the large number of faults found by the sus- pend coverage criterion. This was the case because there were many dierent interruption scenarios. Switching the ignition on and o while performing an- other task is especially error prone. An example of a fault resulting from an interruption between tasks was when the driver started the car while dialing a number. In some countries it is disallowed to dial a number while driv- ing, therefore the system should have suspended the dialing until the car had stopped. The fault was that the driver could wrongfully dial a number while driving. This occurred when the driver started the car in the midst of dialing the number.

However, 28% of the faults could not be found by the proposed test selection criteria. These faults where classied as specic scenarios. These were scenar- ios in which a specic task sequence led to a fault which was not covered by one of the proposed test selection criteria. On the one hand, this indicates that there might be a potential for new test selection criteria that cover parts of these faults and on the other hand this advocates the application of random based test case generation.

Nevertheless, the study shows that more than two-thirds of all faults could be found by the proposed test selection criteria. 50% of the faults are covered by the straightforward test selection criteria Suspend Coverage, Enable Cover- age, and Task Coverage. This is promising, because these three test selection criteria create relatively small test suites in comparison to the Interleaving Coverage criterion.

However, caveats need to be mentioned that can aect the validity of the result:

ˆ The assumption that appropriate test oracles will be available might not be reasonable. Automated test case execution is gaining importance and hence future systems are designed to support automated testing by pro- viding corresponding test interfaces. However, detecting graphical errors automatically is often not possible, especially in the context of animations. Hence, we classied graphical errors conservatively as non detectable. ˆ The sequences that lead to a fault are more complex than indicated by the

bug descriptions. In order to avoid this situation we additionally included the descriptions of the corresponding bug x in our evaluation. However, when we were not sure whether a task sequence was covered by a test selection criterion we classied the fault rather as not covered by a test selection criterion.

In this study, we focused only on faults that resulted from feature interactions. In addition, task-based test case generation could also be used to generate

10.4 Summary 147 feature-specic test cases. We expect then the number of detected faults to be even higher.

10.4 Summary

The study shows that 88% of all error scenarios that resulted from feature in- teractions could be described by a task model. Furthermore, the study showed that the proposed test selection criteria cover a large fraction of faults. To- gether with the previous two case studies, this shows that task-based test case generation can be applied in practice and is able to systematically test feature interactions.

11

Summary and Conclusions

Infotainment systems are examples of multi-functional systems that integrate a large number of features. Over the last vehicle generations the number of features increased continuously. For example, an actual infotainment system in a BMW vehicle integrates up to 1000 dierent features. The number of features is expected to increase even more in future vehicle generations. These features are realized in multiple electronic control units that are con- nected by communication busses. They share common resources, such as the user interface or communication channels. The shared usage of resources re- sults in intentional and unintentional interactions between these features. In practice these interactions are often error prone. A study showed that 40% of all faults that occurred during the development of an infotainment system at the BMW Group resulted from erroneous feature interactions. In order to cope with feature interactions and to detect erroneous interactions early, the interactions between features must be tested thoroughly. However, up to now there are no approaches that enable a systematic and automated test of feature interactions.

In this thesis we targeted this situation and presented an approach for test case generation that enables the systematic coverage of feature interaction scenarios in order to cope with the complexity of future infotainment systems.

11.1 Summary

Model-based test case generation involves multiple steps. First of all, an ap- propriate test model is required that describes critical system properties and that omits unnecessary detail. We introduce task models as means to model features and their interactions. Task models provide a simple graphical no- tation for the explicit modeling of feature interactions. Such a task model describes the space of all possible feature interaction scenarios.

The second step in model-based test case generation is to generate appro- priate test cases. We propose multiple test selection criteria that derive task sequences from a task model. These criteria select task sequences that sys- tematically cover intentional and unintentional feature interaction scenarios.

Task models describe the interaction between environment and system on a high abstraction level. This reduces the state space of large systems to a fea- sible size. Furthermore, they focus on critical feature interaction scenarios by omitting unnecessary details. However, the generated task sequences lack two elementary parts of a test case: they describe no input and no output behav- ior. Hence, the third step of model-based test case generation, the test case instantiation, is one of the most important steps. Test cases are instantiated into executable test scripts in order to execute them automatically. This en- ables a continuous testing process during development. Furthermore, this is an important prerequisite in order to cope with a potentially large number of generated test cases.

As a solution to this problem, we propose two approaches for test case instan- tiation: task sequence renement and aspect-oriented test script generation. The rst approach is specic to task models. Our approach of task sequence renement transforms task sequences into test cases by enriching them with additional input and output behavior. The missing information is derived from behavior models on a lower abstraction level.

The second approach can be applied to generate test scripts from abstract test cases that are dened in arbitrary modeling languages. The problem is that test script generation varies depending on dierent testing concerns, such as test goal, test setup and test phase. Thus, for each testing concern a new transformation must be dened. We propose aspect-orientation as means to modularize these testing concerns. The modularization of testing concerns reduces test case instantiation eort by enabling their reuse in dierent testing contexts.

To conclude, in this thesis we propose an integrated approach for test case generation that involves all steps to systematically test feature interactions. We showed the feasibility and suitability of our approach by multiple case studies in the automotive domain. Furthermore, we created an integrated tooling environment that seamlessly combines all approaches to enable their application in practice.