Test Case Generation from Real World Data

Complementary approaches for the generation of test scenarios and test cases incorporate data from real-world test drives.

The test scenario and test case generation by mathematical and combinatoric approaches can verify the behavior of autonomous vehicle systems in a vast variation of traffic situations, but some modeled traffic situations might not be representative for the operation of the autonomous vehicle system in the real world or be completely unrealistic.

Other approaches incorporate data from real-world driving for the generation of test scenarios and test cases.

In [PHK17], critical situations are identified in data from naturalistic driving studies based on their occurrences in crash databases. The critical situations are stored with their parameters in a situation catalog. Scenarios are defined from the situations in the situation catalog for the qualitative and quantitative evaluation of vehicle safety functions.

Lages et al. briefly describe in [LSK13] the definition of test scenarios from real-world data of reference sensor system. Zofka et al. use in [Zof+15] real-world data for the creation of critical traffic scenarios. The real world data is recorded in test drives by a reference sensor system. The road layouts and trajectories of traffic participants from the real world test drives are recreated in the test scenarios. Spatial and temporal modifications of the recorded trajectories enable the creation of additional test scenarios. The approach is limited by the fixed, derived, and parametrized vehicle maneuvers and allows only for open-loop testing (cf. Definition 2.21).

Peters et al. analyze in [PHR16] recordings from real-world test drives to represent the specific states and transitions in these test drives as automaton models. Segments of consistent behavior are identified in the recorded data and classified to known segments in a knowledge base. The sequence of classified segments is transformed into an automaton model. Classified segments represent the states of the automaton model. Transitions between classified segments in the test drives are mapped to transition in the automaton model. The model is annotated with additional information about the number of instances for each segment and each transition as well as the average duration of each segment. Bach et al. describe in [Bac+15; Bac+17a; Bac+17b] a reactive replay approach for utilizing recorded test data during in the virtual verification of ADAS. The reactive replay approach identifies casual dependencies of inputs for ADAS in order to identify system inputs which can be represented by conventional plant model and system inputs which have to be stimulated by data from conditioned data records. Coherent and reactive stimuli for the SUT are enabled by changing the inadequate domains of recorded data to match the input domains of the plant model.

Driving scenarios are specified by a domain model which abstracts from the real world and characterizes temporal and spatial information on a logical level in an omniscient view composed of sequential different acts and maneuvers (cf. [BOS16]). Graph-based rule models ensure the consistency of specified scenarios.

In [Bac+17c], Bach at al. present a two-step approach for the selection of test scenarios in verification of ADAS with the reactive replay approach. The two-step approach consists of a specification-based classification and a data-driven reduction. Available real-world scenarios are categorized based on criteria and properties from system-level requirements for an initial scenario selection. The initial set of scenarios is further reduced based on their cross-parameter coverage of system inputs and outputs to minimal scenario subsets with significant diversity under avoidance of repetitive situations. The usage of the reactive replay approach limits the combinations of parameters in the selection of scenarios to recorded real-world driving data.

4.1. Related Work

In [Luc+16], Lucchetti et al. automatically identify most common driving scenarios in data from on-road experiments based on spatial relations of vehicles in relation to the ego-vehicle in recorded situations.

The approaches in [Bac+15; Bac+17a; Luc+16; Zof+15] are similar to the engineering approach presented in this thesis (cf. Section 4.3). The approaches use data from real-world drives for the definition of simulation-based tests. However, none of these approaches validates the verification results from simulations in the real world.

The approaches of [Bac+15; Bac+17a; Zof+15] vary recorded system parameters for simu- lations and do not create an abstract representation of the system state and environment state for universal usage in various simulation frameworks. The approach of [Luc+16] incorporates a description of the environment similar to the environment description for the lane change assistant in the case study (cf. Chapter 8). They require the modeling of reference scenarios before the analysis for their identification in real-world test drives. Their approach does not address an iterative extension of these reference scenarios. Simulation-based tests are an efficient approach to verify and validate autonomous vehicle systems in many variations of traffic situations under reasonable costs. However, it is unlikely that simulation-based tests will verify autonomous vehicle systems and their decision making for all possible real-world situations (cf. Section 3.8.3). The following section presents related work for the verification of autonomous vehicle systems during operation.