4. RESEARCH METHODOLOGY
4.5. Validation of Software Implementation and Simulation Results
To validate all software code implementations and the trustworthiness in the comparative simulation results, code correctness checking techniques and statistical significance tests for the results were used throughout the development and performance evaluation cycles of the new framework. The specific correctness check techniques used, in this thesis referred to as software code validation, will be described in the next subsection, while the rationale for the adopted statistical significance test is described in Section 4.5.2.
4.5.1. Software Code Validation
As ns-2 was used as a platform for the simulation environment, all generic MANET related functionalities including the AODV routing protocol were already implemented, so the only new software implementations were those relating to the custom ns-2 plugin and the constituent framework algorithms i.e., TTHCA, TTpHA, and ∆TVE extensions, together with the wormhole and time tampering attack functions. Throughout the development of these implementations both static and dynamic tests were undertaken to validate the code. A static analysis tool called Cppcheck (Sourceforge, 2015), was applied for detecting coding and design errors, like divide by zero, integer overflow and exception handling.
The correct behaviour of the code implementation of each contribution and comparator was manually checked by designing a number of test cases. For these dynamic tests, nodes were assigned static positions in the simulation environment and thus predicted results relating to the PTT, RTT and routing packet processing time (∆Ti) measurements as well as wormhole/time tampering detection algorithm output could be calculated prior to each simulation run and compared to the corresponding simulation output. To rigorously validate each wormhole/time tampering detection algorithm implementation a number of targeted scenarios were generated, in which the outcome for each algorithm was known a priori as to whether it would fail and/or succeed. Finally, the functionality of the custom ns-2 plugin
was validated in a similar way by manually pre-calculating a number of packet processing times based on a given set of TS and ⍴ values, before comparing these with the corresponding output values generated by the getPPTime subroutine.
4.5.2. Statistical Significance Tests
A series of statistical significance tests was undertaken to verify that an adequate number of samples were used for the critical performance analysis in comparison with other state-of- the-art wormhole detection solutions. The output of each tested wormhole detection algorithm is either detected or not detected and these can be seen as two categorical values, i.e. the algorithm and its output. Thus, the statistical significance can be verified if there is a clear association between these two variables, in other words if the choice of algorithm has an effect on the output. In contrast, if these two variables are independent, i.e. if the choice of algorithm has no effect on the output then the statistical significance cannot be verified. For statistical significance analysis, results are typically presented in a contingency table. For this purpose such a table can be structured analogously to Table 4.3 where C1 and C3 are the number of correctly/falsely detected routes by the framework and the comparator respectively, while C2 and C4 are correspondingly, the number of undetected routes.
Table 4.3: Wormhole and FP detection simulation results plotted in a contingency table for statistical significance analysis.
Detection algorithm Output
Detected Not detected
Framework C1 C2
Comparator C3 C4
To analyse such 2 x 2 contingency tables for statistical significance, techniques including
Chi-square test of independence, Fisher’s exact test, and the G-test can be applied
(McDonald, 2014). For this purpose, there were some restrictions relating to the choice of test technique as there are many occasions where a given cell frequency (C1, C2, C3 or C4)
may be small, for example C2 = 0 in cases where the detection rate is 100% for the framework contribution. For this reason, neither the G-test nor Chi-square test can be applied because they require a minimum cell frequency of 5, otherwise at lower cell frequencies the results become inaccurate. In contrast, the Fisher’s exact test is applicable regardless of cell frequency and consequently this is the rationale for applying this particular test in the results analysis. The wormhole detection and FP rates were tested separately and the hypotheses for these tests defined as:
H0: Wormhole detection/FP rate is independent of the applied WH detection algorithm
H1: Wormhole detection/FP rate is not independent of the applied WH detection algorithm
H0 indicates that regardless of the applied detection algorithm, wormhole detection rate/FP
rate will still be similar. Therefore to prove the statistical significance of the performance
improvements of the two wormhole detection algorithms (TTHCA and TTpHA) compared to other state-of-the-art solutions, TTHCA and TTpHA were separately tested against each comparator algorithm. The implementation of Fisher’s test is described in detail in Chapters 5 and 6.
4.6. Summary
This Chapter has provided the detailed methodology for this research. A simulation environment developed in ns-2 has been used for critical evaluation of the performance of all thesis contributions. A detailed description of this environment has been presented together with key parameter settings, outputs, and metrics used for performance evaluation. State-of-the-art packet delay based wormhole detection algorithms, i.e. DelPHI, M-TTM, and MHA, have been implemented as comparators in the simulation environment to demonstrate the performance improvements of the new wormhole detection framework. The
statistical significance of the comparative test results have been analysed by applying
Fisher’s exact test and its implementation has been presented. The correctness of each thesis
contribution and comparator code implementation in the simulation environment has been validated through both static and dynamic tests. The next chapter will introduce the first research contribution in the new wormhole attack detection framework.