Analysis and Interpretation of Results

5.10.1 Experiment 1: Finding an Optimal Configuration. The analysis of this experiment consists of performing a series of one-variable statistical computa- tions and two-variable comparison tests to prove or disprove the hypothesis that a system equipped for a single peer-to-peer protocol (in this case, BitTorrent) can be constructed and optimized such that a packet of interest is detected and recorded with at least 95% probability. In addition, by analyzing the results of the probability of packet intercept (all-peer-to-peer workload) test for each configuration, a figure of merit is calculated for the probability of successfully intercepting multiple sequential packets of interest on a high utilization network.

5.10.1.1 Calculating Packet Processing Time. For each combination of CUT configuration and workload, a one variable t-test is performed to determine the mean packet processing time in CPU cycles, the standard deviation, the standard error of the mean, and a 95% confidence interval for the mean. Then, for each workload, the mean packet processing time of each CUT configuration is compared to the packet processing time of the Control configuration, and the reasons for any increase or decrease in processing time is analyzed.

5.10.1.2 Calculating Probability of Intercept Under Non-Peer-to-Peer and All-Peer-to-Peer Loads. For the configuration and workload combinations outlined in Section 5.9.1, a one proportion confidence interval analysis is performed on the bi- nomial variable to determine the probability of packet intercept and a 95% confidence interval for the proportion. This generates a series of basic statistics from which to perform the two proportion hypothesis testing.

Next, a one-sided statistical hypothesis test using two proportions and a 95% confidence interval is performed for each of the five modification CUT configurations against the Control configuration. The results of these hypothesis tests determine which of the modification CUT configurations show statistically significant improve- ment over the Control configuration.

Finally, a one-sided statistical hypothesis test using two proportions and a 95% confidence interval is performed for the Combined configuration against the other four modification CUT configurations and against the Wireshark software-based packet sniffer. The results of these hypothesis tests determine if the improvement of the combined software configuration over each individual modification is statistically sig- nificant, and also determine if the hardware-based SUT is at least as effective as the software-based Wireshark system in intercepting single BitTorrent packets in a heavy non-peer-to-peer traffic environment and back-to-back BitTorrent packets of interest.

5.10.2 Experiment 2: Expanding the System. The analysis of this exper- iment consists of performing a series of one-variable statistical computations and two-variable comparison tests to prove or disprove the hypothesis that the system can be expanded to include a second peer-to-peer protocol (in this case, SIP) while maintaining at least a 95% probability of intercept for a packet of interest from either peer-to-peer protocol. By analyzing the results of the probability of packet intercept (all-peer-to-peer workload) test for each workload, a figure of merit is calculated for the probability of successfully intercepting multiple sequential packets of interest on a high utilization network.

5.10.2.1 Calculating Packet Processing Time. For each combination of CUT configuration and workload, a one variable t-test is performed to determine the mean packet processing time in CPU cycles, the standard deviation, the standard error of the mean, and a 95% confidence interval for the mean. Then, for the “Non- P2P”, “BT On List”, and “BT Off List” packet types, the mean packet processing time of the Optimized (BT + SIP) configuration is compared to the packet processing time of the Combined configuration from Experiment 1, and the reasons for any increase or decrease in processing time are analyzed.

5.10.2.2 Calculating Probability of Intercept Under Non-Peer-to-Peer and All-Peer-to-Peer Loads. For the configuration and workload combinations outlined in Section 5.9.2, a one proportion confidence interval analysis is performed on the bi- nomial variable to determine the probability of packet intercept and a 95% confidence interval for the proportion. This generates a series of basic statistics from which to perform the two proportion hypothesis testing.

Then, a one-sided statistical hypothesis test using two proportions and a 95% confidence interval is performed for the Optimized (BT + SIP) configuration from Experiment 2 against the Combined configuration from Experiment 1. The result of this hypothesis test determines if there is any statistical difference in the probability of packet intercept of single BitTorrent packets in a heavy non-peer-to-peer traffic

environment and back-to-back BitTorrent packets of interest, when the system is expanded to included SIP functionality.

5.11 Summary

This chapter discusses the methodology used to evaluate the performance of the digital forensic tool under various workloads and network utilization scenarios. Performance is evaluated using a real-world experimental design and is based on two performance metrics: packet processing time and probability of packet intercept. Two partial-factorial experiments using three different tests are performed to measure the impact of varying the software configuration and the peer-to-peer input workload on overall system performance. An analysis is then performed on the data through a series of statistical tests to determine the effectiveness of the system using various configurations and workloads.