Evaluation of the Research Hypothesis

This thesis was motivated by the need to improve the validation process of SAA algorithms required for the safe integration of UAVs into civilian airspace. By building on ideas from SBST, this thesis explored the use of agent-based simulation and evolutionary search to support the validation process of UAV SAA algorithms, with the research hypothesis as follows:

The validation of UAV SAA algorithms requires identifying challenging situations that the algorithms have difficulties in handling. It is possible to identify such situations using an evolutionary-search-based approach and the process can be partially automated. The evolutionary-search-based approach is more effective and efficient than some plausible rivals.

As noted in Section 1.4, the first sentence of the hypothesis is an assumption — we assume that the identification of challenging situations that the tested UAV SAA algorithms have difficulties in handling is a part of the validation work. Firstly, according to the common practice of software testing, which heavily involves finding counterexamples showing the tested software is not valid in all situations, this assumption is clearly sound. Secondly, if the tested SAA algorithms are moderately good, the challenging situations are actually very rare, which is evidenced by that, in all the case studies, the random search either could not, or took a lot of trials to, find even one challenging situation. This is the precondition that it is necessary to develop new approaches to support the validation of SAA algorithms, otherwise, conventional techniques (e.g. random- search-based simulation analysis) would be capable of identifying such situations.

Four propositions can be identified in this hypothesis:

1) Feasibility: it is possible to identify challenging situations for the selected SAA algorithms using the proposed evolutionary-search-based approach;

2) Partial automation: the process of identifying challenging situations for supporting the validation of SAA algorithms can be partially automated if using the proposed evolutionary-search-based approach;

3) Effectiveness: the proposed evolutionary-search-based approach is more effective than some plausible rivals in identifying challenging situations for the selected SAA algorithms.

156

4) Efficiency: the proposed evolutionary-search-based approach is more efficient than some plausible rivals in identifying challenging situations for the selected SAA algorithms. The research described in this thesis explored and evaluated these four propositions as follows. Feasibility is positively supported.

Evidence:

1) In Chapter 4 the proposed evolutionary-search-based approach was used to find mid-air collision situations for SVO either under perfect sensing ability or with sensor noise. Results (Section 4.4.3) showed that the proposed approach can identify some required situations;

2) In Chapter 6 the proposed approach was used to find high-accident-rate situations for ACAS XU. The results showed that it can indeed find some, with a type of

encounter combining the overtaking-overtaken form and the climbing- descending form to be very noteworthy, since it was also found by the random search and the deterministic global search;

3) In Chapter 7 the proposed approach was used to find the violation of safe- separation situations that are most likely to happen in the real-world environment for ORCA-3D. By formalizing the problem as a multi-objective search problem, the proposed approach successfully found the required situations.

Partial automation is positively supported.

Evidence:

1) In all the case studies, having built the simulations and defined the evolutionary search processes, the evolutionary search can then automatically search for the required situations;

2) An open-source tool was developed to support the proposed approach. The tool was used in the case studies as presented in Chapter 6 and Chapter 7. With this supporting tool, the process of identifying challenging situations for SAA algorithm validation can be partially automated.

Effectiveness is positively supported.

Evidence:

1) In Chapter 4 the proposed evolutionary-search-based approach was empirically compared with a random search approach. Results showed that the proposed approach can effectively identify some very subtle situations that random search cannot find in reasonable time;

157

2) In Chapter 6 the proposed approach was empirically compared with a random- search-based approach and a deterministic-global-search-based approach. The results showed that the proposed evolutionary-search-based approach can find high-accident-rate encounters more effectively than the random-search-based approach, and even though it is a little less competitive than the deterministic- global-search-based approach in the relatively easy case, it is more effective in more difficult cases, especially when the objective function becomes highly discontinuous.

3) In Chapter 7 the proposed approach was empirically compared with a random- search-based approach. The results showed that the proposed approach can effectively find the low-cardinality encounter situations that can cause violations of safe separation, while the random-search-based approach has difficulty in finding them.

Efficiency is positively supported.

Evidence:

1) In Chapter 6 the proposed approach was empirically compared with a random- search-based approach and a deterministic-global-search-based approach. The results showed that the proposed evolutionary-search-based approach can find high-accident-rate encounters more efficiently than the random-search-based approach, and it is also more efficient than, or at least comparable with, the deterministic-global-search-based approach, especially when the objective function is highly discontinuous.

2) In Chapter 7 the proposed approach was empirically compared with a random- search-based approach. Since the random-search-based approach failed to find the required situations with a specified number of searches, it is obvious that the proposed evolutionary-search-based approach is more efficient.