The integrated fire control system was benchmarked against the same initial 80 trajectories generated for the Fire and Forget performance evaluation in Chap- ter 3. The results of which would then indicate whether the performance of the AAAW was improved against manoeuvring targets if a data link and integrated fire control system were utilised.
The results of the simulated engagements are shown in Figure 5.19 (in subse- quent chapters, the system will be referred to using the abbreviation of S.A.S.S, short for Simulated Annealing and Simple Search)
0 10 20 30 40 50 60 70 80 0 2 4 6 8 10 12 14 16 x 105 Trajectory Number
Area due to target deviation from LOS m
2
Hit Miss F.M.1
Figure 5.19: S.A.S.S results
The area due to the target deviation shown in Figure 5.19 is calculated using equations 2.51 and 2.54 with R~in either taken as the final location of the target
at the point of interception or at T=40s. Out of the 80 engagements simulated, the weapon successfully intercepted 54 targets and failed to intercept 26. Just as in Chapter 2, missed targets by the weapon in the integrated fire control system can be considered as either Failure Mode 1, 2 or 3. However Failure Mode 1 in this case describes a failure of the integrated fire control system to calculate a trajectory which would result in the target being detectable within the weapon scan area at some instant in time.
The performance of this system against each of the three types of target tra- jectory generated by the model is presented in Table 5.1 along with the Fire and Forget (F.F) results obtained in Chapter 3.
System N.M I.B.M M.T S.A.S.S 12/14 1/6 41/60
F.F 14/14 6/6 27/60 Table 5.1: S.A.S.S and F.F Results
Each of the 26 failed intercepts are classed as Failure Mode 1. Of the 41 ma- noeuvring targets intercepted in the S.A.S.S system, 18 of the targets where also intercepted by the F.F system. The S.A.S.S system intercepted 23 manoeuvring targets which the F.F system failed to intercept and failed to intercept 9 targets which the F.F system successfully intercepted.
Though the F.F system intercepted 27 manoeuvring targets, the constraints for a successful intercept in this system are very limited, as discussed in Chapter 3. The constraints must be considered when comparing the results with the inte- grated system proposed in this Chapter.
In each of the successful intercepts in the S.A.S.S system, the trajectory has been shaped in response to target manoeuvres in order to improve the probabil- ity of a successful detection and engagement. Therefore the results of the S.A.S.S system indicate that this system overall offers a superior performance against manoeuvring attacks compared to the F.F system. The performance against non manoeuvring targets remains excellent as out of the 14 non manoeuvring target trajectories generated, 12 targets where successfully intercepted. However the performance of the F.F system against non manoeuvring targets was superior. The AAAW in the case of the F.F system successfully intercepted all 14 of the non manoeuvring targets.
5.10.1
Test for Statistical Significance - Chi Square
χ
2Test
The integrated system was developed to improve the performance of the weapon against manoeuvring targets, Though the results of the trials indicate that supe- rior performance against manoeuvring attacks is obtained using this system, the possibility that the observed results are due to chance alone must be considered.
In order to determine whether this is the case, a test for statistical significance can be used. In this thesis, a Chi square test is used due to its simplicity and its recognised reliability in a variety of applications from manoeuvre detection in target tracking to drug trial analysis [72].
A Chi test requires the formation of a null hypothesis and the calculation of the χ2 value. The probability of that value can then be obtained from look up
tables or calculated in Matlab. Normally, a probability of 0.05 is then used for a comparison. If the computed probability is less than this value, then there is little statistical evidence to suggest that the results are due to chance. A probability value higher than this would indicate that there is statistical evidence to suggest that the results are due to chance alone.
The following statistical assumptions are made when using this test:
1. The sampling population is random, e.g. the patients of a clinical trial are selected from the wider population at random.
2. The variables of the trial are independent.
The assumptions are satisfied as the trajectories are randomly generated and in each simulated engagement the weapon can either hit or miss the target (in- dependent variable). As only two possible outcomes exist for each simulated engagement, the engagements can actually be classed as Bernoulli trials.
The process for completing the Chi square test is as follows :
1. The results of the F.F and S.A.S.S system are collected into an observed table (O):
System Hits Misses Total F.F 27 33 60 S.A.S.S 41 19 60 Totals 68 52 120
Table 5.2: Observed Data
2. The expected values are then calculated as shown in the expected table (E) below :
System Hits Misses Total F.F 34 26 60 S.A.S.S 34 26 60 Totals 68 52 120
3. The difference between the observed values and expected values is the cal- culated, the sum of which produces the χ2 statistic.
χ2 = 4 X n=1 (On−En)2 En (5.19)
4. The calculation of the probability of the null hypothesis being true requires the number of degrees of freedom (v) to also be determined which is simply defined as :
v = (m−1)×(n−1) (5.20) where m is the number of rows in the observed table and n is the number of columns in the observed table. The number of degrees was calculated as 1.
5. The probability associated with this Chi squared value for the given number of degrees of freedom can be looked up in a table [72]. The Chi-square probability value for the S.A.S.S system was calculated as 0.01.
As the calculated probability is below that of statistical significance (p), the results of the S.A.S.S trials can be considered to be a reliable representation of the performance of the system. The result of the χ2 test therefore supports the
initial conclusion that the integrated system discussed in this chapter will improve the performance of the AAAW to intercept manoeuvring targets. However it is worth exploring the possible reasons why the system failed to intercept the 2 non manoeuvring targets, 5 I.B.M targets and 19 manoeuvring target trajectories.