Startup Performance

One of the first issues discovered in the current version of Snort is extremely slow startup times for large signature-set sizes. Startup performance is generally not relevant due to small signature-set sizes and long run-times. However, it can result in experiment run-time being dominated by startup time. As a result, experiment sizes were kept relatively small (< 50K signatures) in respect to the signature-set available from the randomly generated set of ⇠ 800K signatures (see Chapt. 4.1 on page 91). A functional model for startup time was also required so that experimental measurements could be delayed until the IDS was ready for input.

An experiment was run to determine startup time which would limit experiments. This experiment made use of the Linux “time” program while running Snort against a small file containing a small sample of 10 packets. The packet processing time in this case did not have a measurable impact to the results for the sizes of signature-sets tested. 10 trials were run at increments of 1,000 signatures for each signature-set size between 0 and 45,000

(n = 450). For the randomly generated signature-set, a 5th _{order polynomial provided}

a best-fit (R2 _{= 0.9998) as shown in Fig. 4.2. A single test run of 10 trials using 100K}

signatures corresponded closely to the polynomial model. As a result, the current startup performance hinders the use of extremely large signature-sets for experiments with a large number of signatures. The poor startup performance is likely the result of assumptions made by Snort developers about the largest sizes of signature-sets that would ever be used. Indeed, most performance optimizations of Snort configurations include removing as many signatures as possible without introducing “too many” false negatives. Nonetheless, large signature-sets can still be used, but must be split between multiple Snort instances to achieve reasonable startup times. It is also possible that there are configuration options which may alleviate the issue, though these are not known.

Startup Time vs Ruleset Size

Figure 4.2: Polynomial fit for the startup time of the Snort IDS

Tests with larger sized signature-sets suggests that the startup time is exponential, but the best fit with the experimental data is a polynomial. The best-fit has units of seconds:

0.363 + 0.0026x 1.918⇥ 10 8_{+ x}2_{+ 1.652⇥ 10} 12_{+ x}3 _{3.109⇥ 10} 17_x4_.

4.2.2 Packet Coverage

The expected coverage of the Snort IDS when using a default signature-set was determined by running the IDS against a flat file containing 117GB of Packet Capture (PCAP) data. The data-set represents about 6 hours of traffic between the hours of 9am and 3pm from our corporate network. One caveat of the data-set used is that although collected outside of our corporate firewall, all HTTP and HTTP Secure (HTTPS) traffic was seen through a proxy server. This should not significantly affected the alert output as all individual TCP sessions can still be identified, although we did note Snort error messages regarding upper limits on session tracking. These and other limitations will generally produce an under-estimate of the coverage, so are not a confounding factor regarding measurements.

For coverage measurements, Snort was configured to use the entire official “VRT” signature-set as described in Appendix A.1. All detection signatures except those marked “DELETED” or which failed to compile were enabled in the configuration, representing approximately 4,300 signatures. All of Snort’s built-in preprocessors which were required for this signature-set were left enabled (i.e. stream5, ssl, rpc decode, bo, dcerpc2, ssh, smtp,

frag3, and http inspect). Non-signature-based event generators (those with an generator Identifier (ID) > 1) were disabled or suppressed as it is the coverage of the main signature- based detection engine that was of interest.

Each packet or stream may produce multiple Snort alert events. To eliminate duplicates for purposes of measuring coverage, only events with a unique tuple consisting of (proto, sip, sport, dip, dport, timestamp) were counted. Additional events also occur when re- assembled TCP traffic is processed by a specialized processor. This was ignored, but is not expected to conflate event counts. Many individual packets would result in single events. In many cases if stream-based detection were not used, no event would be detected for any individual packet. As a result, stream-based events should result in an underestimate of Ap.

For a 117GB corporate dataset, the default Snort signature-set results in a packet

coverage of Ap = 0.04%. By our definition of coverage (Sect. 3.1.3, page 64), if every

packet were to produce at least one alert the maximum usable proportion of packets would be 100%. The small coverage of the default signature-set represents a kind of upper bound on any gains via anticipatory bias. This is an upper limit on the proportion of packets which might be used to provide a bias on future detects. As most events are not useful for prediction, the Aq term is needed which describes the ratio of events which are useful for

biasing the detector using a particular predictive engine. For some types of traffic and some attacks, each event may result in multiple predictions, each resulting in a small performance gain. The coverage ratio, Aris thus an underestimate of the actual coverage (which is data

dependent).

As mentioned in Sect. 3.1.3, underestimates in coverage will result in underestimate of performance gains. This is acceptable since we desire an estimate of the least amount of performance gain to expect.

0 10000 20000 30000 40000 4x109 6x109 8x109 1x1010 Rule Count C PU T ime (n s)

Signature set size vs CPU Time (n=400) Linear fit:

y = 103634 x + 4.01621e+09 rms = 0.768 seconds χ2 = 214.47 σ = 1.06 seconds

Figure 4.3: Linear scaling of total CPU-time with signature-set size of the Snort IDS