Despite having the criteria for the evaluation of IDS, the need for attack and realistic background traffic information remains in great demand. There are several approaches for obtaining attack and background traffic data. First, the most common and useful methodology is to build a test bed consisting of a number of connected machines and
60
other elements of a typical network infrastructure. This will simulate the real network despite the use of a limited number of hosts. The minimum number of hosts is three machines: 1) an attacker machine, containing a collection of exploit scripts or attacking tools, 2) a victim machine running vulnerable services, and 3) a monitoring machine running the IDS under testing. After the installation of the test bed, three kinds of traffic are required: normal traffic, background traffic and malicious traffic. The first type can be real traffic obtained from a production environment, or synthetic traffic generated by traffic generators [133-136], whether software-based or hardware-based. Malicious traffic is injected into the background network traffic, and this can be obtained from automated systems such as Metsploit [26] or the manual use of attack scripts. The advantage of this approach is the ease of traffic generation, the ability to repeat the test many times, and the fact that the traffic can be recorded and distributed publicly. On the other hand, this method is expensive and time consuming, particularly if commercial traffic generators [135, 136] are considered, as they perform better than the few, not well documented open-source software ones. Moreover, the use of a limited number of hosts implies less running services, less implemented protocols and the absence of huge number of concurrent connections. Synthetic traffic is generated based on random variables, and some IDSs consider this type of traffic abnormal and may ignore it.
Second, real traffic obtained from a production network infrastructure can be used for IDS evaluation to offer a more realistic approach. The IDS system is connected to a tap or a mirrored port on the edge of a network. This method is less common for reasons of privacy and due to the difficulty in identifying potential unlabeled attacks. Some traffic generators such as Harpoon [137] have been developed as intelligent generators by obtaining real traffic characteristics from live networks.
61
Third, real traffic can be modified to remove all sensitive data and used for testing of IDS systems, which is called sanitised traffic. However, the main task for the IDS is to inspect content and attacks usually residing in packet payload, so this is not the optimal approach for testing IDSs, but can be suitable for other network components.
The first well-documented IDS evaluation methodology to be introduced was the DARPA evaluation 1998 (UNIX dataset) developed in the labs of MIT [55], followed by another dataset in 1999 (Windows NT dataset). Data used in their experiments are labelled and distributed publicly. DARPA datasets have been criticised for not being updated since 1999, for some of the attack types used having become obsolete, and for not covering new emerging attacks [22]. The Lincoln Adaptable Real-time Information Assurance Test-bed (LARIAT) [138] is another evaluation methodology providing a tool for simulation and testing. DEFCON [54] is the worldwide hacker and security expert conference and competition. Malicious traffic can be obtained from DEFCON CTF (Capture The Flag), which contains a huge number of attacking traffic used for IDS testing. NSS Labs [41] is a commercial group that provides a comprehensive methodology for the evaluation of NIDSs. Their approach includes security effectiveness, performance, resistance to evasion techniques, stateful operation, latency, reliability and usability [41]. Background traffic is generated from hardware-based traffic generators such as Spirent SmartBit [136]. Malicious traffic is obtained from automatic tools such as Metasploit [26] and CANVAS [139], or manually defined attacks.
Other efforts have been made to test signature-based IDSs by analysing the collection of attack signatures and then generating mutant patterns to hide the actual attacks. The authors in [25] have developed a cross-testing approach to generate synthetic events to
62
test IDS ability and identify real attacks from modified ones similar to the signatures. [140] addressed the need for publicly well-documented datasets for IDS testing. He presents a set of tools that generate malicious traffic using a virtual network infrastructure. Different platforms were used to create attack traces against various OSs and violating different system services. [141] proposed a framework for offline and online testing to evaluate NIDS resistance to evasion techniques. A comparative evaluation methodology was presented to test Snort and Bro by generating ambiguities in traffic traces.
3.4 Motivation
A typical scenario of employing a NIDS in a network is its implementation on the server with minimum active services. This setup is quite susceptible to insider attacks, especially in high-speed environments. The current NIDSs are also threatened by resource crunch attempts such as DDoS, which has increased from a few megabits in the year 2000 to 40 Gbps in 2008 [142]. The performance criteria of NIDSs demand that every single packet (header, payload) passing through the network needs to be evaluated with the same link speed; however, the massive increase in network speed has generated many concerns. Sending a large amount of traffic or using computationally expensive techniques like fragmentation can compromise a NIDS or make it to start dropping packets.
NIDSs can be implemented as software-based or hardware-based. Software-based NIDSs are more configurable, easy to update and need less maintenance; however, their performance is quite slow. On the other hand, hardware-based NIDSs can handle a larger volume of traffic, but they are expensive, require more maintenance and are hard to update. The choice between the two is a trade-off between cost and performance.
63
This has created the need to evaluate the current software-based systems. This is especially so in current-day high-speed conditions using different implementation scenarios.
We have identified that quite few efforts have been made to measure the performance of NIDSs. Most of the evaluation methodologies are based on testing in moderate traffic conditions. Furthermore, some of these approaches have used previously saved datasets rather than real traffic. These seem unrealistic, as actual system performance was gauged under limited conditions with non-realistic network flow. The results obtained under these conditions could not portray the actual performance output. We have endeavoured to evaluate the system against realistic network conditions, providing the application with different tiers of hardware support in order to analyze its performance more practically. The recent development of multi-core systems has also added a few more opportunities for deploying a software-based system; these shall also be investigated in this chapter.
Our aim in this chapter is to provide answers to the following questions:
Is it possible to deploy a current software-based NIDS such as Snort at a rate above 500 Mbps using commodity hardware? Also to identify the limits of incoming traffic, a system can handle effectively in terms of packet loss.
Does the use of different OSs (normal desktop, server), hardware capabilities (single, multi-core) and configurations (host, virtual) affect NIDS performance?
Identification of mechanisms to improve NIDS performance in high-speed traffic before shifting to hardware solutions.
It is essential that the NIDS is capable to process packets traverse the protected network with speed of the communication link [6, 13, 14]. When the network traffic load
64
becomes higher than the peak processing throughput the NIDS can sustain, the CPU becomes saturated, and the Operating System inevitably starts dropping packets before delivering them to the NIDS, impeding its detection ability [15-17]. Since these packets are not inspected, if they are part of an attack or other malicious activity, then that event will be missed[27, 45].
Assuming a uniform distribution of packets across the network traffic, any packet loss results in a proportional loss in NIDS effectiveness [11]. This relationship has been widely identified in NIDS research [143-146]. Figure 3.1 illustrates the relationship between packet loss and missed alert rate which consequently cause missing attacks and affect the NIDS precision. The scatter plot shows a direct and nearly a linear relationship between the two parameters. The number of missed alerts approaches zero if the packet loss percentage becomes small. The network traffic used in this experiment consists of 530,000 packets containing 521 attacks (1000 packets/alert).
Figure 3.1 Relationship between packet loss & missing alerts
0 5 10 15 20 25 30 0 5 10 15 20 25 30 35 40 45 50 A le rt Loss (% ) Packet Loss (%)
65
Our research has focused on signature-based IDSs with an emphasis on evaluating their performance in high-speed traffic conditions. Snort has been selected as a test platform because of its popularity and status as a de facto IDS standard. We are confident that the results obtained in this research would be equally applicable to other IDSs available on the market. The test environments selected for the research have a significant edge over [40], and our results develop a new understanding of IDS performance limitations.