3.3 Preliminary detection
3.3.2 Current anomaly detectors
A deeper understanding of the MAWI traffic is achieved by analyzing the traffic with three anomaly detectors based on different theoretical backgrounds; one is based on the principal component analysis, one on the gamma modeling and one on the Kullback-Leibler divergence. We selected these three dissimilar detectors to inspect the results from different classes of anomaly detection method.
Principal component analysis
Principal component analysis (PCA) is a mathematical tool that projects a data set onto subspaces in which the variance of the data is maximized. Therefore, these subspaces highlight the main characteristics of the data set and help in classifying it.
The main approach underlying a PCA-based anomaly detector is, first, to monitor the traffic in a matrix, second, uncover the main characteristics of the traffic using PCA (these are considered as the profile of the normal traffic), and finally, report as anomalous the traffic that is featuring separate characteristics. This approach based on PCA is perhaps the most studied technique for anomaly detection, it was first proposed by Lakhina et al. [44] to detect anoma- lous traffic transiting in a network, and it has received much attention in the last few years [48,60,62,37].
For instance, the PCA-based detector employed in our experiments is an improved version proposed by Kanda et al. [37] that overcomes two inherent problems of the original PCA-based detector proposed by Lahkina et al. [44], that are; the restriction of analyzing traffic from several links and the diffi- culty of precisely pinpointing the anomalous traffic due to traffic aggregation [60]. Indeed, the employed detector takes advantage of random projections (or sketches) [42, 48, 16] to analyze only traffic measured at a single link and re- trieve the source IP addresses corresponding to the anomalous traffic. Hereafter we refer to this anomaly detection method as the PCA-based detector.
The result of the PCA-based detector using the MAWI traffic is illustrated by Figure 3.2(a). This detector detects the six prominent anomalies manually reported in the previous section, although, the Blaster worm and the ping flood are partially reported.
Gamma distribution model
The gamma distribution is a model that describes any probability distribution (Gaussian or not) with only two parameters. Dewaele et al. introduced an anomaly detection method based on sketches and multi-resolution gamma mod- eling [16]. Similarly to the PCA-based detector, this detection method uncovers the behavior of the main traffic and reports traffic with different behavior as anomalous. Nevertheless, this detection method relies on histograms and the gamma model that are fundamentally different from the PCA approach.
In a nutshell, this detection method splits the traffic into sketches that are monitored using histograms (contrarily to the PCA-based detector that is rely- ing on traffic matrix). Afterwards, the sketches are modeled using the Gamma distribution and an adaptive reference standing for the normal traffic is com- puted. Thereby, the traffic that is distant from the computed reference is con-
3.3. Preliminary detection
(a) Traffic reported by the PCA-based detector.
(b) Traffic reported by the Gamma-based detector.
(c) Traffic reported by the KL-based detector.
Chapter 3. Preliminary Analysis
sidered as anomalous and the corresponding source and destination IP addresses are retrieved using the sketches.
The result of the Gamma-based anomaly detector using the MAWI traffic is illustrated by Figure3.2(b). This detector successfully identified four prominent anomalies manually reported in Section 3.3.1, however, the Blaster and Sasser worms are not reported using this detector. A careful manual investigation revealed that the traffic corresponding to these worms consists mainly of small flows that are missed by this detector.
Kullback-Leibler divergence
The Kullback-Leibler (KL) divergence is a data differencing metric measur- ing the variance between two probability distributions. It has been applied to anomaly detection by Brauckhoff et al. [13] to detect the prominent changes in the traffic. Similarly to the gamma-based detection method this detector monitors in histograms the probability distribution of the traffic, however, its approach is fundamentally different as the computed reference representing nor- mal traffic is obtained from previous observations.
The approach proposed by Brauckoff et al. [13] is to monitor the traffic in several kinds of histograms that monitor distinct traffic features and apply the Kullback-Leibler (KL) divergence to two consecutive observations. Conse- quently, abnormal variances in the distribution of the monitored traffic features result in high KL divergence values that are detected using an adaptive thresh- old. Traffic features that alter the distribution of the traffic are retrieved using sketches and allows to accurately extract anomalous traffic with an association rule mining algorithm. Thus, the alarms reported by this anomaly detector are association rules, namely 4-tuples (source and destination IP addresses, source and destination port numbers) where at most three elements can be omitted.
The result of the KL-based anomaly detector using the MAWI traffic is illustrated by Figure3.2(c). We observe that this detector reports significantly different traffic compare to the PCA-based and the gamma-based ones. The KL- based detector successfully detected the Blaster and Sasser worms, whereas, it completely missed the Sobig worm. Other prominent anomalies are partially reported by this anomaly detector.
Chapter 4
Anomaly Detection based
on the Hough Transform
4.1
Introduction
Identification of anomalies in Internet backbone traffic is an important task for securing operational networks and maintaining optimal network resources. How- ever, analyzing traffic taken from a high speed Internet backbone — where the payload data is usually inaccessible, the traffic is asymmetric and often sampled — is a challenging issue. A significant difficulty is to accurately characterize anomalous traffic while a wide diversity of threat is constantly emerging. Re- searchers have mainly tried to handle anomaly detection as a statistical issue [9,16,45], but they have faced several common problems; normal traffic is mis- reported when anomalous traffic is dominant, mice flows are usually omitted, and they are in practice difficult to use as the parameter set and output requires advanced knowledge on the underlying statistical analysis.
The main idea of our work is to apply image processing and pattern recog- nition techniques to anomaly detection; traffic is monitored in 2-D scatter plot where each plot represents packets and anomalous traffics appear as “lines”. Anomalies are easily extracted with a line detector and the original data can be retrieved from the identified plots. Thereby, the proposed approach is intuitive to network operators, it also has the advantage of quickly and precisely report- ing anomalies involving mice flows, and it does not assume that the normal traffic is dominant. The method inspects only packet header information at a single point in the network, and it requires no prior information on the traffic or port numbers.
In [24] we proposed the basic idea of this new approach based on pattern recognition of network-related information. Also, the proposed method was partially validated with a single traffic trace. In this chapter, we thoroughly investigate this method; first, we estimate the dependencies of its parameter set. Next, we characterize anomalous behaviors in a large-scale publicly available traffic data set (for 6 years) taken from a trans-Pacific link. We also compare the results of our method with those of different methods based on multiresolution gamma modeling [16] and K-means [63]. Finally, we highlight the different strengths and weaknesses of each method, and emphasize the need for using
Chapter 4. Anomaly Detection based on the Hough Transform
different detection approaches together.