Statistical Analysis

From the boxplots in Figures 3.8 to 3.10, we notice that statistical means and respective medians are similar (quantile-quantile plots show the same) which implies a normal distribution of the results. We are able to perform Welch’s two-sample t-test significance testing on our results. Tables 3.1

to 3.3 show the statistical significance analysis of our null hypotheses (see

§3.3.1).

We will reject the null hypothesis if NC64 defence did not perform similar to SYN Cookies ,i.e, if p-value is less than 0.05. If the p-value is equal to 0.05 then we can neither reject nor accept the null hypothesis. And, if the p-value is greater than 0.05 then we will accept the null hypothesis.

3.3.3.1 LAN Analysis

LAN Setup With No Packet Loss NC64 vs SYN Cookies NC64 vs No Solutions SYN Cookies vs No Solutions

p-value <2.2e-16 <2.2e-16 <2.2e-16

Confidence Interval (99%) 18621.92 to 19772.08 4865.33 to 5400.19 23820.64 to 24838.88 Mean Values (ppr) 24329.76 and 5132.76 5132.76 and 0.00 24329.76 and 0.00 Evidence For Acceptance

(Similar True Means) WEAK STRONG STRONG

LAN Setup With 10% Packet Loss NC64 vs SYN Cookies NC64 vs No Solutions SYN Cookies vs No Solutions

p-value 0.95 <2.2e-16 <2.2e-16

Confidence Interval (99%) -24.39 to 23.27 387.57 to 433.47 403.561 to 416.36 Mean Values (ppr) 409.96 and 410.52 410.52 and 0.00 409.96 and 0.00 Evidence For Acceptance

(Similar True Means) STRONG STRONG STRONG

LAN Setup With 20% Packet Loss NC64 vs SYN Cookies NC64 vs No Solutions SYN Cookies vs No Solutions

p-value 0.05 (or 0.05166) <2.2e-16 <2.2e-16

Confidence Interval (99%) -4.97 to 33.85 169.84 to 181.68 171.71 to 208.69 Mean Values (ppr) 190.48 and 176.04 176.04 and 0.28 190.48 and 0.28 Evidence For Acceptance

(Similar True Means)

NEITHER STRONG NOR WEAK

STRONG STRONG

Table 3.1: Significance testing using Welch’s two-sample t-test examinations of normally distributed results for three test cases within the

LAN environment with varying packet loss conditions. Confidence intervals are at 99% level. ppr is packets per run.

The null hypothesis (see §3.3.1) can be rejected for no packet loss condi- tions. It can be accepted for 10% packet loss conditions. And it can neither be rejected nor accepted in 20% loss conditions. In lossy conditions, NC64

performs better because the traffic control daemons are able to serve the control traffic due to manageable capability request rate (a concurrent re- quest handler for MODCMS can help further improve the request/response behaviour).

If we compare the NC64 performance with the performance of a defence- less victim, we see a strong evidence that the NC64 defence is better than having no defence (valid proof of concept). Similarly, if we compare the SYN Cookies performance with that of a defenceless victim, we see a strong evidence that the SYN Cookies defence is better than having no defence.

No Packet Loss Conditions:

The true statistical mean of the SYN Cookies performance is ∼4.74 times more than NC64 for no packet loss conditions. Please see items 1 to 3 in

§3.3.2.1 for reasons of such performance of NC64 defence which show that it is an engineering issue rather than scientific.

By comparing NC64 defence with a defenceless system, it can be said that the scientific nature of NC64 defence against SYN floods is established. Significant p-values of less than 0.05 and a large difference in respective means (i.e, means of the NC64 defence scenario and SYN Cookies only scenario) attribute to the aforementioned scientific contribution. So, an enterprise is encouraged to deploy NC64 defence if SYN Cookies is either not an option (see §§2.3.6.1 and 3.1) or if NC64 defence performance is increased using better engineering.

10% Packet Loss Conditions:

The true statistical mean of the SYN Cookies performance is∼1.0013 times more than NC64 for the 10% packet loss conditions, which implies that NC64 defence performs similar, on average, in lossy environments. But if we consider the p-value, then NC64 defence is better than SYN Cookies. The p-value of 0.95 is far more than 0.05 that implies that the SYN Cookies did not perform better in comparison to NC64.

NC64 defence performs better than having no SYN flood security in 10% packet loss conditions. The p-value is significantly less than 0.05, and the mean ratio is large, contributing to a better confidence interval. The same can be said about SYN Cookies and the defenceless system.

20% Packet Loss Conditions:

True statistical mean of the SYN Cookies performance is∼0.92 times more than the NC64 performance for the 20% packet loss conditions. In terms of mean values, NC64 performs better than SYN Cookies but if we consider p-value (∼0.05) then there is no evidence that one is better than the other.

The NC64 defence performs better than having no SYN flood security in 20% packet loss conditions. The p-value is significantly less than 0.05, and the mean ratio is large contributing to a better confidence interval.

The network packet loss does reduce the magnitude of the attack traffic but the overall impact of the attack traffic has similar affect on the normal traffic, i.e., we still have attack SYN packets which overflow the TCP’s TCB.

3.3.3.2 MAN Analysis

MAN Setup With No Packet Loss NC64 vs SYN Cookies NC64 vs No Solutions SYN Cookies vs No Solutions

p-value <2.2e-16 <2.2e-16 <2.2e-16

Confidence Interval (99%) -1449.08 to -1228.44 1572.65 to 1777.75 2973.29 to 3054.63

Mean Values (ppr) 1675.20 and 3013.96 1675.20 and 0.00 3013.96 and 0.00

Evidence For Acceptance

(Similar True Means) WEAK STRONG STRONG

MAN Setup With 10% Packet Loss NC64 vs SYN Cookies NC64 vs No Solutions SYN Cookies vs No Solutions

p-value <2.2e-16 <2.2e-16 <2.2e-16

Confidence Interval (99%) -134.47 to -96.81 301.99 to 327.06 416.12 to 444.20

Mean Values (ppr) 314.52 and 430.16 314.52 and 0.00 430.16 and 0.00

Evidence For Acceptance

(Similar True Means) WEAK STRONG STRONG

MAN Setup With 20% Packet Loss NC64 vs SYN Cookies NC64 vs No Solutions SYN Cookies vs No Solutions

p-value 0.00 (or 0.0001833) <2.2e-16 <2.2e-16

Confidence Interval (99%) -25.32 to -5.16 158.08 to 171.04 172.08 to 187.52

Mean Values (ppr) 164.56 and 179.80 164.56 and 0.00 179.80 and 0.00

Evidence For Acceptance

(Similar True Means) WEAK STRONG STRONG

Table 3.2: Significance testing using Welch’s two-sample t-test examinations of normally distributed results for three test cases within the

MAN environment with varying packet loss conditions. Confidence intervals are at 99% level. ppr is packets per run.

The null hypothesis (see§3.3.1) can be rejected for 0%, 10%, and 20% packet loss conditions in MAN environments. Even though the null hypothesis is rejected, we can say that NC64 defence is a valid proof of concept in emulated MAN environments, since they mitigated SYN attacks. Such performance of NC64 defence is expected due to latency issues by having extra network elements, and the unoptimized software implementations.

No Packet Loss Conditions:

True statistical mean of the SYN Cookies performance is∼1.80 times more than NC64 for no packet loss conditions, in a MAN environment. If com- pared with LAN, we have a reduction in ∼3 times of the corresponding multiplication factor. According to the p-value, we can say that NC64 is not a better solution than SYN Cookies in terms of the performance of the 200-OK throughput metric. NC64 defence can increase its performance with better engineering.

By comparing the NC64 defence with the defenceless system, the NC64 defence performed better. The SYN Cookies also performed better than the defenceless system.

10% Packet Loss Conditions:

The true statistical mean of the SYN Cookies performance is ∼1.37 times more than NC64 for 10% packet loss conditions. This shows that NC64 performs similar, in terms of means, to SYN Cookies in MAN based lossy

environments. The difference in the two multiplication factors (∼0.37), as compared to a LAN, signifies that packet losses in low data rate networks like MAN contribute more in degradation (although minimal) of NC64 defence rather than SYN Cookies. It is an odd observation for NC64 defence in MAN with 10% packet loss which demands further investigation into NC64 implementation details. If we compare it with a no-loss MAN environment, then it performs better. One reason for this is that during MAN-based delays and lossy conditions, CMS software is less loaded with capability requests than in no loss MAN or LAN environments.

NC64 defence performs better than having no SYN flood security in 10% packet loss conditions in MAN. The p-value is significantly less than 0.05, and the mean ratio is large contributing to a better confidence interval.

20% Packet Loss Conditions:

The true statistical mean of the SYN Cookies performance is ∼1.09 times more than NC64 for 20% packet loss conditions. In terms of mean val- ues NC64 performs similar to SYN Cookies but if we consider the p-value (∼0.00) then SYN Cookies is better than NC64 defence.

NC64 defence performs much better than having no SYN flood security in 20% packet loss conditions in MAN. The p-value is significantly less than 0.05, and the mean ratio is large, contributing to a better confidence interval. The same can be said about the comparison of the SYN Cookies with the defenceless system.

3.3.3.3 WAN Analysis

WAN Setup With No Packet Loss NC64 vs SYN Cookies NC64 vs No Solutions SYN Cookies vs No Solutions

p-value 0.00 (or 0.005458) <2.2e-16 <2.2e-16 Confidence Interval (99%) -29.21 to -1.19 369.80 to 374.85 373.74 to 401.30

Mean Values (ppr) 372.52 and 387.72 372.52 and 0.20 387.72 and 0.20 Evidence For Acceptance

(Similar True Means) WEAK STRONG STRONG

WAN Setup With 10% Packet Loss NC64 vs SYN Cookies NC64 vs No Solutions SYN Cookies vs No Solutions

p-value 4.36e-10 <2.2e-16 <2.2e-16 Confidence Interval (99%) -43.32 to -21.16 172.24 to 192.32 209.78 to 219.25

Mean Values (ppr) 182.88 and 215.12 182.88 and 0.60 215.12 and 0.60 Evidence For Acceptance

(Similar True Means) WEAK STRONG STRONG

WAN Setup With 20% Packet Loss NC64 vs SYN Cookies NC64 vs No Solutions SYN Cookies vs No Solutions

p-value 6.39e-05 <2.2e-16 <2.2e-16 Confidence Interval (99%) -20.51 to -4.93 128.40 to 136.08 138.08 to 151.83

Mean Values (ppr) 132.96 and 145.68 132.96 and 0.72 145.68 and 0.72 Evidence For Acceptance

(Similar True Means) WEAK STRONG STRONG

Table 3.3: Significance testing using Welch’s two-sample t-test examinations of normally distributed results for three test cases within the

WAN environment with varying packet loss conditions. Confidence intervals are at 99% level. ppr is packets per run.

The null hypothesis (see§3.3.1) can be rejected for 0%, 10%, and 20% packet loss conditions. Even though the null hypothesis is rejected, we can say that NC64 defence is a valid proof of concept in a WAN environment as well (as is the case in LAN and MAN), as it mitigated the SYN flood attack.

No Packet Loss Conditions:

The true statistical mean of SYN Cookies performance is∼1.04 times more than NC64 for no packet loss conditions in WAN environment. If compared with the LAN based analysis, we have a reduction of ∼4.8 times of the corresponding multiplication factor. In terms of the p-value, we can say that NC64 is not a better defence than SYN Cookies. The reduction in the multiplication factor is a better result, but the p-value clearly shows that the NC64 implementation requires better engineering and low backend latency.

NC64 did perform similar to SYN Cookies if we look at the width of the confidence interval, and the difference in means.

The NC64 defence and the SYN Cookies defence performed better than the defenceless system.

10% Packet Loss Conditions:

The true statistical mean of the SYN Cookies performance is ∼1.17 times more than NC64 for 10% packet loss conditions. The difference in two multiplication factors, as compared to LAN, is minimal in 10% packet loss case, meaning it performed similar to LAN in such conditions.

NC64 defence performs better than having no SYN flood security in 10% packet loss conditions in WAN. The p-value is significantly less than 0.05, and the mean ratio is large contributing to a better confidence interval. The same can be said about the SYN Cookies versus no defence.

20% Packet Loss Conditions:

The true statistical mean of the SYN Cookies performance is ∼1.10 times more than the NC64 performance for the 20% packet loss conditions in the WAN environment as well in comparison to the MAN environment. Even though from the outset it looks like a similar result, if we consider the p- value (∼6.39e-05), we need to say that the alternative hypothesis of our null hypothesis (see§3.3.1) will take preference.

The NC64 defence performs better than having no SYN flood security in the 20% packet loss conditions in the WAN environment as well. The p-value is significantly less than 0.05, and the mean ratio is large contributing to a better confidence interval. The same can be said about the SYN Cookies only versus no defence.