Dimension Selection Algorithms For Events Space Partitioning

Dimension selection algorithms proposed in [7] are based on the observation that not all dimensions are equally important when it comes to content filtering ie., it not necessary to allocate the same number bits to each dimension when partitioning or rather, greater

granularity can be achieved if more bits are assigned to some dimensions than others and some dimensions need not be allocated with any bits at all.

Event Variance-Based Selection

This algorithm decides the dimensions which are necessary to have better granularity for content filtering than just using all the dimensions based on event distribution in the event space. The algorithm is as follows

• Let event space be represented by ES.

• Let D be the set of all Dimensions in ES. DS be the subset of D.

• Populate ES with all the events and let E represent the set of all events.

• The sum of the variance of the value of each event along each of the dimensions is calculated with the formula Σ(x_i− x_m)²/|E| where xi is the value of the event along i^th dimension. The calculated value is called the selectivity factor of that dimension. This selectivity factor is higher for a dimension if the events are distributed more along the entire length of that dimension.

• Only those dimensions with higher selectivity factor are added to DS.

• Then, Event space partitioning takes place only with DS instead of D which means that more bits are allocated now for the dimensions in DS than they were before. This increases the granularity of content filter because dimensions with higher selectivity factor have more events distributed along its length. Hence, increasing the number of partitions along the length of a dimension by increasing the number of bits allocated to it also increases the granularity of that dimension as depicted in 3.6.

Figure 3.5: Workload-Based Indexing and calculation of MBR

Figure 3.6: A depiction of dimension selection with Event Variance-based Selection algorithm

Both the figures in 3.6 depicts a 2-Dimensional event space with event distribution in which the maximum number of bits that can be appended to an IP address is 2 bits. 3.6 also depicts a subscription sub1 in both the figures. The figure on the left depicts a partitioning strategy in which an equal number of bits are allocated to both dimensions and the one on the right depicts event space partitioned using the Event Variance-Based Selection algorithm. In the latter case, the 2 available bits are allocated to the horizontal dimension which increases selectivity because sub1 now match an event subspace which causes lesser number false positives as the number of unnecessary events is obviously lesser.

Subscription Matching and Event Count-Based Selection

An interesting observation noted in [7] is that, if a large number of subscriptions overlap along one of the dimensions then an event matching a subscription along that dimension has a higher probability of matching several more subscriptions. If this happens to be the case, then that dimension has lesser granularity for content filtering and hence its ability to reduce false positive is low. This concept is depicted in 3.7. The following algorithm decides the dimensions to which bits need to be allocated for calculating DZs(content filter) based subscription overlap and the total number events which are matched to these subscriptions.

The algorithm is as follows

• Let the event space be represented by ES. Let E be the set of all events and S be the set of all subscriptions.

• Then, for each event e in E, the subset of S which matches the event along each dimension d is added as an element to the set S_d^e

• Next, the importance of a dimension d is calculated with the formula Σ(S_d^e)/(|E| ∗ |S).

This formula gives a value between the interval [0,1] which denotes how important a dimension is for granularity of content filters. Greater the value greater is its importance.

This value is called the selectivity factor.

• Then, event space partitioning takes place only with a subset of D whose selectivity factor is high. This means more bits are allocated for the selected dimensions than they were before.

Figure 3.7: A depiction of dimension selection using Subscription matching

Figure 3.8: A depiction of dimension selection using correlation based selection

Correlation-Based Selection

This strategy is based on the fact that often multi-dimensional dataset has several di-mensions which are related to each other. For example, consider a sensor which collects temperature and pressure in a room. It can be shown that the dimensions are positively(in case of sealed room) or negatively (in case of rooms with lots of windows) correlated to each other. When the dimensions are related to each other in this manner, it becomes redundant to allocate the same number of bits to all the dimensions. Allocating the redundant bits to a subset of unrelated dimensions increases the overall granularity of the content filters.[7]

3.8 depicts a 2-Dimensional event space where both the dimensions are positively correlated and event space is allocated with 2 bits.for appending with IP address. The figure on the left depicts an event space where all dimensions are partitioned using the same number of bits and the one on the right with an event space where only one dimension is allocated with both

bits. It can be seen that the subscription in the latter case has more granularity than the one in the former case.