experienced enough to combine multiple 2D displays and other information into a higher dimensional mental picture. The technological drive resulting from the gaming industry, especially in the realm of computer graphics, has made it possible to generate more realistic displays of the GBAD environment, while still allowing for sufficient rendering speed.
(a) View of several simulation ob- jects, including missile trails and radar domes
(b) Visualisation of the radar dome and virtual cones in the GBAD environment
Figure 7.10: South African 3D GBAD system visual analysis tool displays [63].
7.5 HMI Designed in Matlab
A preliminary HMI was designed using Matlab’s graphic user interface tool. This design should not be seen as a final design, but rather as the foundation for a more detailed HMI that may be used to validate and evaluate the final TEWA DSS in conjunction with its intended end-users (the operators). Furthermore, this design also serves the purpose of demonstrating the capabilities of the Matlab graphic user interface design environment.
Matlab is viewed by many users as both a high-performance language for technical computing as well as a convenient environment for the development of HMIs [179]. The majority of military HMI designs continuously change during the project life-cycle, as the customer requirements of the intended end-users are reiterated [88]. It is therefore unavoidable that the system designers need to adopt an evolutionary development process. In addition, the preliminary design should account for these possible future changes in order to save time, cost and improve flexibility, thereby allowing the design to better adapt to changing customer requirements.
The layout of the preliminary designed HMI is shown in Figure 2.1(a)–(d). The demonstrated scenario is the hypothetical GBAD scenario introduced in §4.6. The four different figures cor- respond to a different TEWA-cycle (scenario time), and the four selected times correspond to the TEWA-cycle times of 25, 50, 75 and 100. The two DAs are depicted by black-filled circles, whereas the two WSs are depicted by the black-filled triangles. The maximum ranges of the two WSs are visualised by the red circles. The three threats, on the other hand, are depicted as squares, their past tracks are indicated by grey dotted lines and their current velocity vectors are depicted by lines protruding from the squares. Furthermore, the system threat values of these three threats are visualised by the bar-chart in the top right-hand corner and the results of a number of binary flagging models are indicated for the currently selected threat. In addition, the values of specific assessment cues are listed in the table. Some operator controls are also included to be used for detailed validation purposes in the future.
(a) TEWA cycle 25
7.5. HMI Designed in Matlab 133
(c) TEWA cycle 75
(d) TEWA cycle 100
7.6 Chapter Summary
A number of considerations were provided in this chapter for inclusion in the detailed design of the TEWA DSS HMI. The chapter opened with an overview of the required DSS functionality in §7.1. After understanding the requirements and functioning of the required DSS, decision support pertaining to a GBAD environment was described in general in §7.2. Special attention was afforded to the operators in a GBAD environment (§7.2.1) and the types of information management strategies (§7.2.2).
After understanding the context and requirements of the DSS, the importance of facilitating germane decision support for the operators was detailed in§7.3. Special attention was afforded to the complexities associated with the provision of germane decision support to the FCO. Furthermore, the effect of operator stress on system performance, and the uncertainties present in a TEWA DSS were described in§7.3.2 and §7.3.3, respectively. An existing HMI interface was subsequently described in §7.4.1 and a number of design suggestions were made for the detailed design of a HMI in§7.4.2. The chapter closed with an overview of a preliminary designed HMI and demonstrated the capabilities of the Matlab graphic user interface design environment in §7.5.
PART III
PERFORMANCE EVALUATION
CHAPTER 8
Performance Evaluation Framework
True genius resides in the capacity for evaluation of uncertain, hazardous and con- flicting information.
— Winston Churchill
Contents
8.1 Performance Evaluation of TEWA Systems Overview . . . 138 8.2 The Concept of System-of-Systems Analysis . . . 139 8.3 Adopted System of Systems Approach . . . 140 8.4 Performance Evaluation Approaches . . . 142 8.4.1 Prototype Evaluation in Conjunction with End-Users . . . 142 8.4.2 Single Scenario Evaluation . . . 143 8.4.3 Batch-simulations . . . 144 8.5 Performance Evaluation Metrics . . . 145 8.5.1 Survivability Metric . . . 145 8.5.2 Economy Metric . . . 146 8.5.3 Engagement Effectiveness Metric . . . 146 8.5.4 Adaptability Metric . . . 147 8.6 Practical Simulation Characteristics . . . 147 8.7 Chapter Summary . . . 148
As mentioned in Chapter 1, the aim in this project is to integrate the available TE and WA algorithms in order to develop a simulation of a working TEWA system. After developing the simulation, it is required to evaluate the validity and quality of the results returned by the system. Because of the many elements in the system, in practice, this can become a very complex and iterative process that intersects many disciplines.
This chapter opens by first providing an introduction to the evaluation of TEWA systems, after which the underlying concepts of a System of Systems (SoS) approach is explained. This is followed by a review of available performance evaluation approaches for TEWA systems as identified from literature and, finally, four metrics are proposed to be used within the developed simulation paradigm.
8.1 Performance Evaluation of TEWA Systems Overview
Before the 1980s, military systems were largely stand-alone, analogue and mechanically con- trolled [194]. Today, most military systems are highly-complex1 in the sense that numerous on-board subsystems typically interact to provide the system capability in a collective fashion. As systems become unavoidably more complex, as is the case with NCW (see §2.1.2), it is im- portant to recognize the complexity inherent in defining the common system capabilities as well as to understand the constraints and requirements of all the potential users, thoroughly. This understanding is crucial in order to ensure that the final commissioned system functions in such a way that the user requirements and constraints are met.
The only way to ensure that a system satisfies all the requirements and accommodates all the constraints of its end-users, is through rigorous testing of the system, both in a simulated- environment (as advocated and implemented in this thesis), and in a physical environment where the system is tested in concert with the constituent systems (namely the operators, WSs and meteorological conditions etc. in the case of a TEWA system) [65]. This Testing and evaluation is an iterative process of performance measurement, correction of deficiencies and remeasuring of the resulting performance [194]. This testing process should commence as early as possible during the design process of a system and should be conducted throughout system development, as advocated by the International Council of Systems Engineering [49]. Testing and evaluation are not separate from the design process, but rather an inherent, intrinsically critical part of it. Testing and evaluation should also not end when the system is commissioned, the performance should be verified throughout its operational phase until the system is retired [49].
The ability to influence a system’s characteristics diminishes rapidly as the development of the TEWA system progresses through its life-cycle stages [187]. It is therefore of utmost importance to ensure that diligent attention is afforded to during the early life-stages when developing a new system. This concern is partly the reason for the adoption this preliminary performance evaluation through this study.
The main purpose of the preliminary testing of a TEWA system, as applied in this thesis, is to identify general design deficiencies and specific conflicts present in the internal algorithms of the system and, consequently, highlighting required corrective action. By following a bottom- up testing approach2, it is possible to reduce the risks associated with the final commissioned system. According to Sparrius [187], the only way to demonstrate risk reduction in terms of reducing the number of predicted system failures — as a prerequisite for an increase in resource commitment (time and capital) — is through testing and evaluation.
In view of this, M&S may be seen as the key enabler for effectively focussing and executing the testing and evaluation of a complex system such as a TEWA DSS. An M&S approach may be used to predict system performance, identify risk areas, review technology suitability and support the evaluation of the system’s effectiveness (i.e. suitability and survivability) [194]. Despite the lack of academic references for analysis methods applied within the defense-domain, the authors know that M&S are common tools used for the evaluation of TEWA systems.
1For an explanation of what a complex system entails, the reader is referred to§4.1.
2Bottom-up refers to an integrated testing approach in which the lowest level components (TE and WA) are tested first, before testing of the higher-level system components [188].