Mathematical Test Case Generation

One approach for the generation of test cases is the use of mathematical and combinatoric techniques to define new test cases for autonomous vehicle systems systematically. Inputs and outputs of autonomous vehicle systems are analyzed and combinatorially combined as test cases which cover a large quantity of the problem space for autonomous vehicle systems.

Schuldt et al. present in [Sch+13] a modularized virtual test tooling kit for the generic generation of test scenarios for simulation-based verification. The tooling kit outlines the process from the verified ADAS over the identification and analysis of relevant parameters, definition of test scenarios, verification in XIL tests, and evaluation of test results by metrics. The tooling kit further defines a four-layer model for the flexible combination of the underlying road topologies, obstacles, traffic situations, and weather conditions in test scenarios.

In [Ber10], Berger models test scenarios from system requirements and metrics from customers’ acceptance criteria for automatic acceptance testing of autonomous vehicle systems in simulations. The test scenarios are defined in a domain specific language (DSL) called ScenarioDSL.

Some approaches generate tests scenarios by classifying traffic situations. Saust et al. define in [Sau+09] an approach for the test case generation based on the classification of critical traffic situations from crash data databases. Sippl et al. derive test cases from probabilistic environment-sensitive behavior simulations data (cf [Sip+16]). The simulation data is filtered for relevant situations in which other traffic participants impact the behavior of the automated ego vehicle. The remaining situations are rated based on developer specified factors and stored in a situation catalog. Test scenarios for are defined in a textual DSL from the situations in this situation catalog.

Berger et al. address in [Ber+14; Ber+15] the systematic generation of EuroNCAP conform test scenarios1 _{for simulation-based verification of active safety systems under}

the consideration of tolerance ranges for a specific system and environment parameters. Possible variances of parameters over time are modeled as a graph. Trajectories are derived from the graph as test cases for the simulations. The authors enhance the evaluation of test results by introducing tolerance ranges in [Ber+14].

The authors of [Gäf+08; Tat15; TMJ12] present a tool-based generation of test cases for simulation-based testing of autonomous vehicle systems. The tool TestWeaver uses intelligent search in order to automatically analyze and classify the behavior of autonomous vehicle systems in simulations for the automatic generation of different test scenarios. TestWeaver controls specified parameters and inputs of the system and monitors the system behavior in simulations on system requirements and quality criteria. Khastgir et al. present in [Kha+17] an automated constrained randomized approach for the definition test scenarios and test cases using the tool Vitaq2_{. The approach applies}

randomization to test scenarios and test cases for the variation of vehicle trajectories, environment, and traffic in the simulations by intervening the real-time communication between simulation and SUT. Constrained randomization enables the intelligent explo- ration of the problem space in simulations in order to find the corner cases for which an ADAS and automated systems are likely to fail.

Aniculaesei et al. present in [Ani+18a] an automatic requirements-based test-case genera- tion for an adaptive cruise control system. Natural language requirements are formalized in LTL. The LTL formulas are negated as trap properties under consideration of code coverage metrics. A formal system model is used by the model checking tool NuSMV to generate counterexamples for the trap properties (cf. [Cim+02]). The counterexamples represent traces through the system model which allow the derivation of test cases. The test cases verify the system requirement for different mutants of the initial system. The authors of [Bau+07; Bau+08; Sie+11] introduce usage models for the risk- and model-based testing of software-based (embedded) systems. Annotated UML diagrams from the requirements analysis enable the definition of state-based usage models. These

1_{https://www.euroncap.com/en/for-engineers/protocols/ (Accessed 12/05/2018)}

4.1. Related Work

usage models represent the risk profile of each SUT. Transitions in the usage models are annotated with the corresponding risks. Test cases for the SUT are derived from valid paths through these usage models. For risk-based testing, generated test cases are prioritized in risk-based test plans based on the accumulated risks by all transitions in each test case’s path.

In [Oli+16], Olivares et al. use Markov Chain and Markov Chain Monte Carlo methods to generate road topologies for test scenarios. Essential road parameters, e.g., geometry, type, and the number of lanes are deduced from OpenStreetMap (cf. [HW08]). All relevant road parameters are repented in a stochastic model by probability density functions and conditional probabilities. This stochastic model allows generating test scenario with critical combinations of road parameters which have a high possibility to reveal unsafe system behavior.

Zhao et al. introduce in [Zha16] an approach to accelerate the evaluation of automated vehicles by eliminating repeating and uncritical parts in naturalistic driving data for simulations. Stochastic models for the behavior of traffic participants are derived from real-world naturalistic driving data and optimized by reducing its non-safety-critical portion. Monte Carlo simulations (cf. [Moo97]) use the optimized models in order to evaluate interactions between the automated ego vehicle and other traffic participants with higher criticality. Results from the Monte Carlo simulations help to understand the performance of the automated ego vehicle under naturalistic driving conditions. In [HLZ17], Huang et al. incorporate the Kriging model as statistic models for the behavior of traffic participants.

Mathematical and combinatorial approaches for the generation of test scenarios and test cases in simulation-based tests are used to cover large scopes of the input/output space of systems whose input space has been completely specified. Autonomous vehicle systems incorporate inputs with complex data, e.g., object-oriented description of the systems’ environments, that makes it difficult — if not impossible — to fully cover the input/output space of autonomous vehicle systems by mathematical and combinatorial techniques. It remains questionable if mathematical and combinatorial techniques able to generate test scenarios and test cases that sufficiently model relevant and critical real-world traffic situations.

Nevertheless, general aspects of these approaches, e.g., the layered model for scenarios in [Sch+13], can be incorporated by more-elaborated approaches. This thesis considers the layered model for test scenarios in [Sch+13] for the definition of test scenarios (cf. Section 7.2.3).