Conclusions

Outlier detection is an important task for data mining applications. Existing algorithms are effective and have been successfully applied in many real-world applications. But these algorithms, especially density-based algorithms, have low efficacy in datasets with different densities. In our work, we have introduced the Rank-based outlier Detection Algorithm (RBDA) that is effective in solving the problem mentioned above.

CHAPTER

3

A

NOMALY

D

ETECTION

A

LGORITHMS

B

ASED ON

C

LUSTERING AND

W

EIGHTED

R

ANKS

Clustering can be used to find anomalous objects in a given data set. This approach has a significant advantage; it reduces the time complexity considerably provided the clustering algorithm is fast.

In this chapter we argue that an object that belongs to a very large cluster is more likely to be declared anomalous even if it is only slightly far away from its neighbors, compared to the case if it had belonged to a small cluster. To account for this imbalance, we propose the concept of modified rank.

The chapter is organized as follows. In Section 3.1 we discuss some clustering approaches for anomaly detection. In Section 3.2 we introduce important notations, and in Section 3.3 we discuss our clustering algorithms, followed by our proposed new algorithm in Section 3.4. In Section 3.5, we compare the performance of the new algorithm with RBDA and other close competitors.

42

3.1

Clustering Approach for Anomaly Detection and a

New Clustering Algorithm

Among existing anomaly detection algorithms, one unsupervised approach is based on clustering using the assumption that normal data objects belong to a cluster whereas anoma- lies do not belong to any clusters. Its essential theme is to find clusters in a given data set and declare any data object that does not belong to any cluster as anomalous. Unlike the classical approaches to clustering in which all objects of the data set must belong to a clus- ter, in this context objects are not required to be in a cluster. K-mean clustering and other well known clustering techniques cannot be applied towards this task; special clustering algorithms have been developed that do not force every data object to belong to a cluster. DBSCAN, ROCK and SNN clustering algorithms proposed by Esteret al. [18], Guhaet al., [28], and Ertozet al.[17], respectively, are among such clustering algorithms.

Unlike previous work, we do not require that all objects that do not belong to a clus- ter are anomalous; instead we assume that data objects that do not belong to a cluster are ‘potential’ anomaly candidates. Further investigation is required to evaluate their anoma- lousness.

In Section 3.3 we describe a new clustering algorithm which exploits “reachability". The main idea is to declare data objectsp andq to be ‘close’ ifq is one of thek nearest neighbor of p and vice versa. We define “reachability" as the transitive closure of the ‘closeness’ relation, e.g; ifp1andp2are ‘close’ andp2andp3are ‘close’ we declare that

p3 is reachable from p1. Finally, reachable data objects are consider to form a cluster. Details in mathematical notation are presented below.

There is one lingering concern related to the size of a cluster. For instance, if a cluster contains only two data objects, is it ‘really a cluster’? If anomalies in the data set form clusters by themselves, then a cluster based approach won’t be able to detect them, leading

to poor performance. Chandolaet al. [11] address such concerns. A subjective judgment has to be made regarding the minimum number of data objects to form a cluster recognizing yet another assumption to detect anomalies: “Normal data objects belong to large and dense clusters, while anomalies either belong to small or sparse clusters."

We proposed a new algorithm by using cluster size and distance (in addition to the rank) directly in evaluating the outlierness of data objects. We present several variations of such algorithms, all of which begin with clustering as a preprocessing step. These algorithms are evaluated on real and simulated test data sets, helping to understand the kinds of problems in which they are most useful. These variations exhibited similar performance to each other, which was considerably better than that of algorithms explored in previous work.

In the next section, after introducing key notations and definitions, we describe Tao and Pi’s [60] DBCOD algorithm. Next we present new outlierness measures and new al- gorithms for outlier detection. These new algorithms are compared with RBDA, DBCOD, and other algorithms, using one synthetic and three real data sets. Brief descriptions of data sets are presented in Section 3.5, followed by our findings. Section 3.6 presents concluding remarks.

3.2

Notation and Definitions

The following definitions are used in the proposed clustering algorithm; all definitions are parameterized with a positive integer parameter`intended to capture the notion of cluster tightness.

• D-reachability(given `): An objectpis directly reachable (D-reachable) fromq, if

p∈ N`(q).

• Reachability: An object p is reachable from q, if there is a chain of objects p ≡ p1, . . . , pn≡q, such thatpi is D-reachable frompi+1for all values ofi.

44

Fig. 3.1: Illustrations of dk(p), Nk(p) andRNk(p)for k = 3. - The large blue circle contains elements of N3(p) = {x, e, y}. Because y is farthest third nearest neighbor of

p, therefore d3(p) = d(p, y), the distance between p and y. The smallest (green) circle

contains elements ofN3(y)and the medium radius circle (red) contains elements ofN3(z).

Note thatry(p) = (rank ofpamong neighbors ofy) = 9. Finally,RN3(p) = ∅, since no other object considerspin their neighborhood.

• Connectedness: Ifpis reachable fromq, andq is reachable fromp, then pandqare connected.

Figure 3.1 illustrates the concepts ofdk(p),Nk(p)andRNk(p).