3. Dataset size reduction. 4. BGP-4 patterns. Detection of inter-domain routing problems using BGP-4 protocol patterns P.A.

(1)

Newsletter Inter-domain QoS, Issue 8, March 2004

Online monthly journal of INTERMON consortia

Dynamic information concerning research, standardisation and practical issues of inter-domain QoS

---Special issue on detection of faults in inter-domain

environment

There are different faults in inter-domain

environment which could influence the end-to-end

connection behaviour. The network engineer should be

able to detect different kinds of fault behaviour of QoS

parameter values, routing events (BGP-4 protocol

patterns) and traffic to obtain their patterns and to analyse

their causes in the specific context.

INTERMON toolkit allows to study faults based on

patterns (behaviour structures) by usage of dedicated

tools.

We describe the detection and analysis of different

kinds of patterns using INTERMON tools:

-

BGP-4 patterns which could impact the

behaviour of end-to-end connection

-

QoS outlier , i.e. abnormal delay and packet loss

patterns.

Based on systematical measurement and analysis

strategies to study causes of QoS abnormal behaviour

(outliers) it is possible to relate them to discovered

inter-domain routing events such as BGP-4 patterns and

traffic loads.

---Detection of inter-domain routing problems

using BGP-4 protocol patterns

P.A. Gutiérrez

1. Introduction to BGP-4 database analysis

results

BGP-4 has been the driving force in many

measurement scenarios in the last phase of the

INTERMON project. Due to the lack of access to private

BGP-4 routing data, the project used the incomplete, but

public BGP-4 route repositories.

2. Scenario definition

Several measurement scenarios have been used in the

INTERMON project. BGP-4 data analysis has been

initially conducted for the measurement lag between

Madrid and Salzburg. During the scenario analysis phase,

it was observed that this measurement lag passed through

the London Internet Exchange (LINX), traversing

intermittently routers which donate their routing data to

the RIPE Routing Repositories.

3. Dataset size reduction

The data stored in the 'rrc01' directory structure of the

RIPE Routing Repositories was studied for the period

between the 27

th

of November, 2003 and the 6

th

of

January, 2004.

During this time, the mean size of the Routing Table

Snapshot files (

bview files) was around 510*

5

records.

The mean size of the sum of all daily Routing Activity

files (update files) for one day was around 3*10

5

records.

This huge amount of data is reduced to 11 significant

routing table snapshot and 77 routing updates records for

the whole aforementioned period.

4. BGP-4 patterns

During the period under study , different patterns of

updates could be found, depending on the way they are

classified. INTERMON has classified routing updates

based on the router they arrive at. Based on this first

criterion, following patterns were observed

1. Intermediate AS refresh: An intermediate AS

announces routes to both endpoints in a narrow

time window:

In the previous sample, an AS which is situated

between the endpoints and the Internet Exchange

has refreshed the routing tables and has produced

these advertisements as a result of the refresh.

2. Route flap: A route has become unstable and is

repeatedly being advertised by a peer to the

Internet Exchange during a narrow time

window:

(2)

3. Dampened route flap: In principle, the

dampened route flap is a route flap sequence

terminated with a route withdraw advertisement.

In this case, the route flapping mechanism has

been triggered and the route is put into

quarantine until it is stable again.

4. Traffic Engineering techniques: A route is being

advertised repeatedly with different AS_PATH

attributes. The AS_PATH_PREPEND technique

is used to artificially control the precedence of

the route in the routing table

</Update>

The time window of these pattern is quite broad, in order

to circumvent the route flap dampening mechanism.

Narrow time windows imply a high probability of route

flap dampening, which would imply that the sequence

ends with a route withdraw.

---Detection of outlier patterns for study of

abnormal QoS parameter behaviour in

inter-domain environment

I. Miloucheva

1. Introduction to outliers

Outlier is an abnormal QoS parameter behaviour.

Detection of outlier patterns in QoS parameter

measurement data allows to study the structure of the

abnormal QoS parameter behaviour [MAM 03].

A special tool aimed at outlier detection and analysis

is integrated in INTERMON allowing to analyse

abnormal behaviour of QoS parameter of end-to-end

connections in inter-domain environment.

Using this tool, the structure of outlier values is

analysed and compared, occurrence and timely

relationships of outlier patterns of different QoS

parameter values are discovered.

The interest of this paper is to show the usage of the

outlier detection tool in packet loss and delay outlier

patterns study of real measurement data of end-to-end

connection in inter-domain environment.

2. User interface for study of outliers

Outliers are defined related to an threshold value

d

thr_m

. A sequence of x

1

,...x

i

,...x

n

, belonging to the time

series data sequence {X

t

} build an “outlier”, when for all

x

i

, i = 1..n, hold that x

i

> d

thr_m

.

For instance., threshold for delay outliers could be

selected considering abnormal behaviour in context of

some specific application, for instance threshold of 150

ms in respect to VoIP QoS guarantee. Thresholds could

be also defined by very unexpected values, which occurs

and deviate in multiple times from the normal behaviour.

Packet loss threshold could be defined for some period,

i.e. aggregation interval, as sum of occurred packet

losses, or could be defined in form of packet loss burst,

i.e. threshold for the count of consecutive packet losses.

Outlier patterns are detected using a special tool with

an user interface allowing to set different options for

outlier pattern analysis, as it is shown in the next figure:

(3)

Figure 1: Outlier user interface

The outlier analysis in INTERMON is based the QoS

parameter data collected per flows which are contained in

the cmbase of INTERMON.

There are different functions to study outlier in

INTERMON which are included in the user interface:

-

Outlier structure per flow with optional specification

of start and end time for outlier. Type of outlier

pattern , i.e. decrease, increase, is discovered, which

could support to detect the cause of the outlier, for

instance packets delayed in a router [D20]. The

function includes the discovery of relative length of

outliers duration in the observed interval ?t.L

di

=

? L

di,k

/?t, where L

di,k

is the duration of the outlier

pattern P

di,k.

-

Analysis of the occurrence of the outlier patterns.

Occurrence and frequency of “outlier” sequences in

a given time interval and for a given period (daily,

monthly), i.e. the number of observations of

“outlier” patterns P

di,k

, k = 1..m, related to the time

interval ? t : F

di

= ? P

di, k

/ ?t.

-

Global analysis of outliers in a spatio-temporal

context or in the whole data base with optional

specification of sender and receiver hosts of the

end-to-end connection.

-

Outlier structure dependencies – discover related

outliers found in the specified time intervals

following or preceding a given outlier.

-

Histogram for analysis of the values of the outliers,

and their assignment to appropriate buckets (ranges)

bi, i = 1...k, of QoS threshold values.

Cmbase allows usage of different kind of QoS parameter

measurement data - row and aggregated per specified

time interval. Different options – row and aggregate

could be used in outlier structure study. Usage of

aggregated QoS parameter data allows to increase the

speed of the discovery of outliers, but it has also some

drawbacks:

-

implies reduced accuracy in measurement (the

outlier occurrence is given in the aggregation

interval range and not at exact time it occurs)

-

the sequence of the outliers could not be obtained

from the aggregated data.

The aggregated analysis is therefore useful for study of

global outlier occurrence and dependencies, where the

row data based outlier analysis is aimed at fine grained

outlier study.

3. Scenarios for abnormal QoS behaviour

study based on outliers

The design of cmbase allows to study outliers

considering row and aggregated QoS parameters

measured for end-to-end flows in different

spatio-temporal context.

We discuss detection of end-to-end delay outlier

patterns for fault management using real measurement

data of connection Madrid – Salzburg collected in

cmbase.

We show different scenarios to study outliers:

-

Global analysis of outliers based on selected

spatio-temporal contexts allowing to detect similar outliers

in different end-to-end connections in inter-domain

environment. A particular case is detection of

outliers contained in all measurements collected in

the data base.

-

Outlier structure analysis aimed at end-to-end delay

outlier patterns showing abnormal behaviour found

in the real measurement data as well as packet loss

outlier patterns in real measurement data. This

analysis should also show in exact way dependencies

of end-to-end delay outlier patterns from packet loss

patterns.

The global outlier analysis per aggregation period is

used to study occurrences of outliers in different

connections, which is important for evaluation of the

properties of connections for capacity planning and

forecasting.

Fine grained delay and packet loss outlier structure

analysis based on row QoS parameter data we use for

fault management in order to detect causal behaviour of

outliers and their dependencies.

(4)

3.1 Global end-to-end delay analysis of QoS

parameter data based on aggregated data

Global analysis of QoS parameter data is usually used

to discover similar outlier behaviour in different

connections (flows) measured in some specific time,

which could include a long time period.

The global analysis function allows to detect properties

of all occurrences of outliers of specific kind (end-to-end

delay, packet loss) corresponding to a given threshold in

their specific spatio-temporal context. In cmbase design,

the spatio-temporal context of the end-to-end connection

is described by the flow notion. We use the global

outlier analysis functions, to see if periodical and similar

outliers occur in different flows and their structure.

It is also used to compare the detected outliers and

to relate them to specific events found in this

spatio-temporal context. Appendix 1 shows the maximum delay

outliers exceeding 150 ms as found on different flows in

different times on connection Madrid-Salzburg.

For the end-to-end connection Madrid-Salzburg,

based on QoS monitoring, was detected that an

unexpected abnormal end-to-end delay parameter value

over 200000 ms, ca 4 minutes, occurring 11

th

of

December 2003 at 04:07:00 and 04:09:09. This delay

outlier is extremely unusual, i.e. in multiple times (in

range of 1000 multitudes ) greater value that the normal

end-to-end delay behaviour in the range of 50-100 ms of

this connection.

Using global analysis, it is shown based on

aggregated data (300 sec aggregation interval), that the

abnormal end-to-end delay QoS parameter values in the

range of 3-4 minutes (over 200000 ms) were found also

in other time periods (27

th

of November 2003 at 21:38:00

CET).

The lists of delay outliers over 200000 ms with

description of their occurrence and maximal values,

based on aggregated QoS parameter data describing

maximum delay for all flows of the connection Madrid

Salzburg is shown:

Global Outlier Analysis: maxdelayOutlier value:200000000 [microseconds]

Flow: 141 Conn: 194.179.25.33->212.183.10.170 Aggregate: 300sec

Start/End: Wed Nov 26 16:08:00 CET 2003|Fri Nov 28 16:08:00 CET 2003

Outlier: Maximal outlier: 2.32803424E8 [microseconds] Outlier frequency: 1 Relative length: 0.0034662045

Outlier:Thu Nov 27 21:38:00 CET 2003 Max_QoS_value: 2.32803424E8 [microseconds] Distance: 105600 sec

Start/End: Wed Dec 10 15:17:00 CET 2003|Fri Dec 12 16:02:00 CET 2003

Outlier:Thu Dec 11 04:07:00 CET 2003 Max_QoS_value: 2.4789904E8 [microseconds] Distance: 45900 sec

Flow: 273 Conn: Aggregate: 600sec

Start/End: Wed Dec 10 15:39:09 CET 2003|Sun Dec 14 15:39:09 CET 2003

Outlier:Thu Dec 11 04:09:09 CET 2003 Max_QoS_value: 2.47918688E8 [microseconds] Distance: 44400 sec

The global analysis based on aggregated values allows to

obtain information on spatio-temporal occurrence,

distance (i.e. time interval) between outlier and

maximum outlier value. The accuracy of outlier

frequency and their relative time is restricted by the

aggregation interval (in our case 300 sec) .

Aggregated QoS parameter values are evaluated by

minimum, mean and maximum values per interval. There

is also interesting to check in the aggregation interval

where extreme abnormal maximum delay occurs, what

are the minimum delay values which are normally due to

propagation delay and router overhead (BGP-4).

Minimum delay outlier on the connection Madrid –

Salzburg is found only on 29

th

of July 2003, at 13:09 in

the range of

25629716 microsecond

(see Appendix 2)

which could be due to changes of router and transmission

overhead at this period.

As in Appendix 2 is shown, minimum delay outliers over

150 ms ranging up to

578802 microseconds

are found often

on the connection Salzburg. Although

Brazil-Salzburg connection has longer end-to-end delay in

mean, the extreme maximum and minimum end-to-end

delay outlier are significant lower than on the connection

Madrid-Salzburg. This is shown in appendix 3 where the

maximum end-to-end delay outliers are shown for

Brazil-Madrid which do not exceed 1169183 microsecond in

these measurements.

The observations based on the global analysis for the two

connections have shown at least that there are two kinds

of inter-domain connections concerning end-to-end delay

outliers:

-

connections (like Madrid-Salzburg) with very

extreme end-to-end delay outliers compared to the

normal experienced delay values (in rnge of 1000

times greater outlier values comparable with normal

behaviour)

-

connections (like Brazil-Salzburg) with smaller

extreme outlier values which are greater in range of

10 times than the normal behaviour.

Correspondingly, we could use the global analysis to

detect abnormal (e.g. extreme ) packet loss rate and

(5)

packet loss bursts found in the aggregation intervals of

the different connections (see appendix 4 for packet burst

over 50 packets , i.e. maximum number of sequenced

packet losses found in the different connections in

inter-domain environment)

Packet loss like delay and other parameters depends on

the time scale of measurements. In our measurements we

use interval of 1 sec for sending of packets except in flow

273 where 20 ms was used. Therefore looking at

Appendix 4 we could conclude for the global analysis of

the packet loss bursts that in the connection

Brazil-Salzburg prevailing are bursts in range of 50-62 packets.

Very great packet loss sequences are typical for the two

connections and there are typical more packet loss

sequences reported than at connection Madrid-Salzburg.

However, the two connections experience great packet

loss bursts over 300 sequenced losses ranging to 1000

and more sequenced packet losses at different times

which should be further studied.

3.2 Analysis of outlier patterns using

aggregated data per flow – delay and

packet loss

For the different cases where the extreme abnormal

end-to-end delay outlier over 200000 ms occurred (i.e.

connection 141, 255, 273), we give the corresponding

visualisation of the maximum delay outlier behaviour

considering also outliers which exceed the threshold of

150 ms. The visualisation represents the great difference

of the extreme outlier values to the other outliers, i.e. the

extreme outlier value are given in other measurement

scale, than the rest of outliers which values are close:

Figure 2: Pattern of end-to-end delay outlier based on aggregated data with threshold of 150 ms– flow 141

Figure 3: Pattern of end-to-end delay outliers based on aggregated data with threshold 150 ms – flow 255

Figure 4: Pattern of end-to-end delay outliers based on aggregated data with threshold 150 ms - flow 273

(6)

The figures 3 and 4 show aggregated end-to-end delay

outliers exceeding 150 ms as detected by measurements

on the same connections for same 2 days (10

th

and 11

th

of

December 2003) with different granularity of

measurement intervals i.e. flow 255 interval of 1 sec and

flow 273 – interval of 20 ms. It is obvious that, using

smaller intervals of measurements, much more outliers

are detected for the same period than using greater

intervals. The outlier detection depends on the

measurement interval and this should be considered in

the fault management strategies.

For analysis of the sources of the extreme outlier of 4

minutes end-to-end delay, we checked with the global

outlier analysis interface also other real measured data

for connections Sao Paulo (Brazil) – Salzburg (Austria)

which is much more unreliable than the connection

Madrid-Salzburg and found that such great outliers do

not happened.

Using global analysis, we detected also dependencies of

the extreme and-to-end delay outliers and packet loss

bursts. Very closely to the outliers are significant packet

loss bursts which should be further investigated in the

next scenarios using fine grained outlier structure

analysis.

The packet loss bursts considering 2 and more sequenced

packet losses for the different flows with found extreme

delay outliers are given and the packet loss burst found

near the extreme end-to-end delay outlier is marked.

Figure 6: Occurrence of aggregated packet loss burst outliers exceeding 10 sequenced losses – flow 141

(7)

Figure 7: Occurrence of aggregated packet loss burst outliers exceeding 10 sequenced losses – flow 255

Figure 8: Occurrence of aggregated packet loss burst outliers exceeding 10 sequenced losses - flow 273

It is obvious from the figures 6, 7, and 8, that although in

the three cases, significant packet loss bursts are detected

near the extreme delay outlier, there are also other

occurrences of packet bursts of such magnitude not

directly related in timely manner with the extreme

outliers over 20000 ms.

Similar to end-to-end delay outlier, the packet loss bursts,

i.e. consecutive packets lost , is dependent on the

granularity of measurement, i.e. time intervals in which it

was measured. Therefore, the packet bursts near to the

delay outlier of 11

th

of December measured in the flow

255 are significant smaller, i.e. 248 sequenced packet

losses according 1 second measurement interval,

compared with flow 273 with 24757 packet loss and the

interval of measurement 20 ms. Therefore, packet loss

burst should be considered as measurement metrics for

comparison, when the measurement interval is exactly

given or when instead of consequent losses packets, the

duration of packet loss burst is assumed.

The conclusion of the global outlier analysis of

connection Madrid – Salzburg is that extreme great delay

outliers are specific for this connection and their cause

should be studied related to some events of this

connection.

(8)

The analysis of the outliers using the aggregated QoS

parameter data is based on the maximum QoS parameter

values evaluated in the given aggregation interval.

The accuracy of the results depends on the aggregation

interval. If this interval is in the range acceptable for

application quality study, then the usage of outliers based

on aggregated QoS parameter data is preferred because

of the performance benefits looking in smaller sized

tables of the data base.

Further analysis is aimed to provide fine grained outlier

structure detection using per packet QoS parameter

measurement data, i.e. row data.

3.3 Analysis of outlier values distributions

using histogram

Using the outlier data mining tool, it is possible to do a

histogram of the found maximum end-to-end delay

outlier values considering the aggregated or row QoS

parameter values of all measured flows .

We start with histogram based on aggregated QoS

parameter data considering measurements of all flows.

The histogram based on 5 buckets is shown in figure 9:

Figure 9: Histogram of delay outliers over 150 ms of measured flows using aggregated data

The prevailing outliers of the connection assuming

threshold of 150 ms (selected in respect of VoIP

application) are in the bucket range near 150 ms until 400

ms , than there is decrease. Again, starting with very

extreme values there is some increase.

We could obtain more detailed histogram of delay

outliers over 150 ms by selecting greater number of

buckets, as for instance it is shown in figure 10:

Figure 10: Detailed histogram of delay outliers over 150 ms of measured flows using aggregated data

Here is shown based on more detailed analysis the

decreasing number of outliers starting with 400 ms, very

small number in the range of 600-1000 ms and the

increase of outlier occurrence at extreme values.

Interesting is also the study of outliers based on packet

loss and burst packet loss based on aggregated QoS

parameters and global measurement data analysis:.

Figure 11 shows the histogram of bursty loss:

(9)

Figure 12 shows the packet loss based on aggregated

data:

Figure 12: Histogram of packet loss using aggregated data

There are similarity in the behaviour of the values of

bursty and packet loss. However, there are ranges where

the packet loss is high, but there is no bursty loss, i.e.

there is frequent loss, but not sequenced loss. Here is

again interesting, that extreme values of packet loss and

bursts of losses (i.e. highest burst) are significant.

3.4 End-to-end delay outlier structure analysis

The analysis based on row QoS parameter measurement

data allows to discover:

-

outlier structure : increasing, decreasing , plain

[MHG 03]

-

exact occurrence of outlier

-

exact time interval between outliers.

The outlier analysis based on row QoS parameter

measurement data aggregated data gives insight in exact

structure of the outlier behaviour, but could imply long

data base access times, because of the significant mount

of QoS parameter data per packet.

This analysis is preferred in fault management scenarios

where causes for outlier behaviour should be studied s

well as timely dependencies of outliers.

The next figure gives the end-to-end delay structure

description of the flow 273 with extreme outlier over

200000 ms. Because of the big differences between the

typical outliers ranging from 150 to 600 ms, and the

extreme outlier of 200000 ms, different scales are used

to represent all outliers of the end-to-end connection.

Figure 13: End-to-end delay patterns of outliers of flow 273

The extreme outlier has a string decreasing pattern

over the whole 87 packets building it, i.e. each packet of

this outlier has smaller delay that the preceding packet.

Immediately after the last packet of this outlier there is a

packet loss in time interval comparable with the

end-to-end delay of this outlier with maximum value of

247918684

microseonds.

Decreasing outlier structure with dependent packet

loss could be explained with stopping of transmission in

some device (see, [D20] is given the more detailed

explanation.

We found string decreasing and increasing patterns

typical for this flow.

The next figure shows a preceding outlier starting

Thu Dec 11 02:31:13 CET 2003 with Maximum

end-to-end QoS value: 745317 [Microsec] and Length 65, which

has also a strong decreasing pattern over the whole range

of 65 packets.

(10)

Figure 14: Decreasing pattern of end-to-end delay outlier found at 02:31

It should be noted that the differences between the

sequenced values in the decreasing patterns are in range

of 8-12 ms.

Traffic patterns of 20 ms between sent packets like

flow 273 are typical only for specific kind of real time

applications like VoIP. Applications requiring services in

more coarser traffic patterns will not experience such

patterns. For this purpose we show the patterns obtained

from flow 255 which was measured every second.

Instead of decreasing pattern in the case of the outliers of

247918684

and 745317 [microseconds] maximum value,

the measurement tool has reported only one value in the

two cases. In addition, we could see that a lot of outliers

occurring at the same time and experienced by the fine

grained measurements in flow 273 have not been

experienced in flow 255.

Figure 15: Patterns of end-to-end delay outliers of flow 255

The structure of the outliers of connection 141 is

derived from measurements done in 1 second intervals.

We see therefore small number of outliers. The extreme

end-to-end delay values are experienced with sequence

of decreasing outlier structures (3 outliers).

Figure 16: Outlier end-to-end delay patterns of flow 141

Interesting fact is that besides the extreme outlier there

was only two other small value outliers above 150 ms in

the two day period.

(11)

The conclusion from the discussed outliers in the

particular measurements is:

-

outlier detection and patterns depend on the

measurement scale or granularity

-

there are very extreme outliers which occurred at

different time interesting string decreasing

patterns indicating that some event stopped the

transmission.

3.5 Packet loss analysis

The end-to-end delay outlier behaviour could be

compared with packet loss and burst loss patterns

occurred in the corresponding time.

The detailed analysis of packet loss includes:

-

packet loss length patterns

-

patterns describing occurrences between packet

losses

-

detection of specific time intervals of packet

losses in order to relate them to other QoS

parameters like end-to-end delay.

Using the outlier data mining interface of INTERMON

toolkit we could detect all packet losses, their occurrence

(start, end), length of consecutive lost packets, and

distances between the packet losses.

Appendix 5 show the detailed description of packet loss

patterns for the flow 141. We see there is a significant

amount of packet loss with prevailing packet losses of 1

and 2 packets.

In addition there is a relationships between the detected

packet loss the extreme end-to-end delay outliers of flow

141 over 200000 ms. Immediately after the extreme

end-to-end delay outliers are packet losses:

Packet loss: Thu Nov 27 21:31:52 CET 2003 , Thu Nov 27 21:32:01 CET 2003, Length 8pkts Distance: 11430,105852 Packet loss: Thu Nov 27 21:32:01 CET 2003 , Thu Nov 27 21:32:05 CET 2003, Length 3pkts Distance: 1,105856

Packet loss: Thu Nov 27 21:32:05 CET 2003 , Thu Nov 27 21:35:42 CET 2003, Length 216pkts Distance: 1,106073

In order to see the concrete distribution of the packet

losses for this flow, we obtain the histogram of packet

loss lengths:

Figure 17: Packet loss distribution of flow 141

The distances between packet losses are also of interest

and are studied

The next figure shows the outlier structure analysis for

the aggregated QoS parameter data, measured in

November including similar extreme outlier value.

Figure 18: Packet loss distance distribution of flow 141

For comparison of packet loss distributions we look at

the other flows (i.e. 255 and 273), where extreme

end-to-end delay outliers were encountered.

(12)

The corresponding packet loss analysis for flow 255 is

given in figure 19:

Figure 19: Packet loss distribution of flow 255

It is shown the prevailing occurrences of small packet

losses, especially packet loss of 1 packet length.

However, there are significant packet losses ranging from

40 sequenced lost packets and over.

Figure 20 shows the distribution of packet loss distances

of flow 255:

Figure 20: Packet loss distance distribution of flow 255

The buckets are selected to show the numbers of packet

loss distances starting by 1 and increasing the distances

by 1 until 100 packets distance. The figure shows that

the occurrence of packet loss distances decrease rapidly

with increase of distances to 100 packets, and at very

great packet loss distance, the occurrences again increase

The detailed packet loss analysis has also shown that

immediately after the extreme outlier, there was a

significant packet loss. Similar patterns are also shown in

the analysis for flow 273 measured at the same time like

255 , but in much lower time intervals.

4. Towards efficient analysis of

abnormal QoS behaviour

The analysis of abnormal QoS behaviour by detection

of outliers is a powerful approach which could be used in

different fault management scenarios – detection of QoS

problems caused by link failures, traffic congestion,

inter-domain routing, intrusion and denial of service

attacks. In INTERMON scenarios, we have shown

relationships between end-to-end delay and packet loss

outliers.

Further systematical monitoring and analysis

strategies of end-to-end connections in inter-domain

environment should allow to study dependencies of

discovered QoS outlier patterns and patterns describing

inter-domain routing behaviour, i.e. BGP-4 protocol

patterns.

The important understanding during development of

outlier data mining tools in INTERMON is that more

efficient technologies are needed to store and access the

outlier data. More efficient data base design based on

time series data concepts is required for further reduction

of redundant data storage. For instance, timestamps of

packets sent as time series data in equal intervals should

not be stored. This will reduce stored data and increase

performance.

There is a need for efficient data base for outlier

analysis in spatio-temporal context based on storage of

the abnormal data per end-to-end connections allowing

optimisation of time point and interval queries of outliers.

The spatio-temporal design will reduce the overhead in

the browsing of whole data base to find the flows and

their temporal dependencies belonging to the end-to-end

connection.

Because the outliers are very small portion of the total

measured packets, the data base design should contain

only the outlier packets. The measurement system should

be designed in the way to store the outliers and their

patterns in dedicated data base. Currently, the way to use

real measurement data from cmbase, which is storing all

possible performance information to the sent packets in a

(13)

data base is very slow approach in the case of fault

management scenarios where only outlier values are of

interest.

Further research direction on efficient data base

design is consideration of multiple measurement

granularities (scales) [BJW 00] in order to provide

efficient operations to manipulate and query these data

along the temporal dimensions.

5. Reference

[BJW 00]

C.Bettini, S.Jajodia, S.X.Wang, Time

Granularities in Databases, Data Mining and Temporal

Reasoning, Springer Verlag, ISBN 3540669973, July

2000.

[MAM 03] I. Miloucheva, A. Anzaloni, E. Müller, A

practical approach to forecast Quality of Service

parameters considering outliers, First international

workshop on Inter-domain performance and simulation,

IPS 2003, Salzburg 20-21 February.

[D20] Evaluation of visual data mining, INTERMON

Deliverable D20

Appendix 1: Global end-to-end delay outliers

in cmbase for the flows of Madrid-Salzburg

obtain for aggregated QoS parameter data

Start/End: Tue Nov 18 16:08:00 CET 2003|Thu Nov 20 16:08:00 CET 2003

Outlier: Maximal outlier: 290399.0 [microseconds] Outlier frequency: 6 Relative length: 0.011111111

Outlier: :Tue Nov 18 16:13:00 CET 2003 Max_QoS_value: 290399.0 [microseconds] Distance: 0 sec

Outlier: :Wed Nov 19 08:03:00 CET 2003 Max_QoS_value: 241807.0 [microseconds] Distance: 49200 sec

Outlier: :Thu Nov 20 12:53:00 CET 2003 Max_QoS_value: 290399.0 [microseconds] Distance: 67500 sec

Start/End: Mon Nov 24 16:28:00 CET 2003|Wed Nov 26 16:08:00 CET 2003

Outlier: :Mon Nov 24 19:48:00 CET 2003 Max_QoS_value: 158323.0 [microseconds] Distance: 11700 sec

Start/End: Mon Nov 24 16:20:00 CET 2003|Wed Nov 26 16:20:00 CET 2003

(14)

Start/End: Wed Nov 26 16:08:00 CET 2003|Fri Nov 28 16:08:00 CET 2003

Outlier: :Thu Nov 27 21:38:00 CET 2003 Max_QoS_value: 2.32803424E8 [microseconds] Distance: 34200 sec

Flow: 237 Conn: Aggregate: 1800sec

Start/End: Mon Dec 01 10:00:00 CET 2003|Wed Dec 03 09:59:00 CET 2003

Outlier: :Mon Dec 01 11:30:00 CET 2003 Max_QoS_value: 220143.0 [microseconds] Distance: 3600 sec

Outlier: :Tue Dec 02 05:30:00 CET 2003 Max_QoS_value: 176225.0 [microseconds] Distance: 64800 sec

Flow: 143 Conn: 194.179.25.33->212.183.10.170 Aggregate: 300sec Start/End: Sun Nov 30 16:08:00 CET 2003|Tue Dec 02 16:08:00 CET 2003

Outlier: :Sun Nov 30 19:33:00 CET 2003 Max_QoS_value: 202102.0 [microseconds] Distance: 12000 sec

Flow: 144 Conn: 194.179.25.33->212.183.10.170 Aggregate: 300sec Start/End: Tue Dec 02 16:08:00 CET 2003|Thu Dec 04 16:08:00 CET 2003

Outlier: :Wed Dec 03 06:28:00 CET 2003 Max_QoS_value: 159294.0 [microseconds] Distance: 51300 sec

Flow: 255 Conn: 194.179.25.33->212.183.10.170 Aggregate: 300sec Start/End: Wed Dec 10 15:17:00 CET 2003|Fri Dec 12 16:02:00 CET 2003

Outlier: :Thu Dec 11 02:32:00 CET 2003 Max_QoS_value: 556102.0 [microseconds] Distance: 12900 sec

Outlier: :Thu Dec 11 04:07:00 CET 2003 Max_QoS_value: 2.4789904E8 [microseconds] Distance: 5700 sec

Flow: 256 Conn: 194.179.25.33->212.183.10.170 Aggregate: 300sec Start/End: Fri Dec 12 16:02:00 CET 2003|Sun Dec 14 16:02:00 CET 2003

Outlier: :Sat Dec 13 15:27:00 CET 2003 Max_QoS_value: 196309.0 [microseconds] Distance: 84000 sec

Flow: 257 Conn: 194.179.25.33->212.183.10.170 Aggregate: 300sec Start/End: Sun Dec 14 16:02:00 CET 2003|Tue Dec 16 16:02:00 CET 2003

Flow: 260 Conn: 194.179.25.33->212.183.10.170 Aggregate: 300sec Start/End: Sat Dec 20 16:02:00 CET 2003|Mon Dec 22 16:02:00 CET 2003

Flow: 273 Conn: Aggregate: 600sec Start/End: Wed Dec 10 15:39:09 CET 2003|Sun Dec 14 15:39:09 CET 2003Outlier: 150000 Maximal outlier: 2.47918688E8 [microseconds] Outlier frequency: 28 Relative length: 0.058925476

Outlier:Wed Dec 10 22:59:09 CET 2003 Max_QoS_value: 270618.0 [microseconds] Distance: 25800 sec

Outlier:Thu Dec 11 02:29:09 CET 2003 Max_QoS_value: 745317.0 [microseconds] Distance: 12600 sec

Outlier:Thu Dec 11 04:09:09 CET 2003 Max_QoS_value: 2.47918688E8 [microseconds] Distance: 4200 sec Outlier:Thu Dec 11 05:49:09 CET 2003 Max_QoS_value: 280604.0 [microseconds] Distance: 6000 sec

(15)

Outlier:Fri Dec 12 08:59:09 CET 2003 Max_QoS_value: 193584.0 [microseconds] Distance: 36000 sec

Outlier:Sat Dec 13 22:59:09 CET 2003 Max_QoS_value: 768346.0 [microseconds] Distance: 108000 sec

Outlier:Sat Dec 13 23:59:09 CET 2003 Max_QoS_value: 516283.0 [microseconds] Distance: 3600 sec

Appendix 2 : Global outlie r analysis

Madrid-Salzburg and Brazil-Madrid-Salzburg based on

aggregated data for minimum end-to-end

delay outlier

Global Outlier Analysis: mindelayOutlier value:150000 Flow: 9 Conn: 194.179.25.33->195.3.85.134 Aggregate: 300sec

Start/End: Tue Jul 29 11:54:00 CEST 2003|Thu Jul 31 11:54:00 CEST 2003

Outlier: Maximal outlier: 25629716 [Microsec] Outlier frequency: 1 Relative length: 0.833

Outlier:Tue Jul 29 13:09:00 CEST 2003 Max_val: 25629716 [Microsec] Distance: 0 sec

Flow: 10 Conn: 161.24.3.26->195.3.85.134 Aggregate: 300sec Start/End: Tue Jul 29 11:54:00 CEST 2003|Thu Jul 31 11:54:00 CEST 2003

Flow: 49 Conn: 161.24.3.26->195.3.85.134 Aggregate: 300sec Start/End: Wed Aug 13 16:22:00 CEST 2003|Thu Aug 14 16:22:00 CEST 2003

Outlier:Wed Aug 13 17:12:00 CEST 2003 Max_val: 161004 [Microsec] Distance: 0 sec

Outlier:Thu Aug 14 01:17:00 CEST 2003 Max_val: 161990 [Microsec] Distance: 600 sec

Flow: 43 Conn: 161.24.3.26->195.3.85.134 Aggregate: 300sec Start/End: Thu Aug 14 16:17:00 CEST 2003|Fri Aug 15 16:17:00 CEST 2003

Outlier:Fri Aug 15 02:27:00 CEST 2003 Max_val: 168441 [Microsec] Distance: 2100 sec

Flow: 93 Conn: 161.24.3.26->195.3.85.134 Aggregate: 120sec Start/End: Tue Sep 09 18:13:00 CEST 2003|Thu Sep 11 18:13:00 CEST 2003

Outlier:Thu Sep 11 05:37:00 CEST 2003 Max_val: 207280 [Microsec] Distance: 127320 sec

Flow: 94 Conn: 161.24.3.26->195.3.85.134 Aggregate: 120sec Start/End: Thu Sep 11 18:13:00 CEST 2003|Sat Sep 13 18:13:00 CEST 2003

Outlier:Fri Sep 12 07:29:00 CEST 2003 Max_val: 177957 [Microsec] Distance: 47040 sec

(16)

Appendix 3 : Maximum end-to-end delay

outlier values over 150 ms for connection

Brazil-Salzburg based on aggregated data

Flow: 5 Conn: 161.24.3.26->195.3.85.134 Aggregate: 300sec Start/End: Fri Jul 25 09:31:00 CEST 2003|Fri Jul 25 11:31:00 CEST 2003

Outlier:Fri Jul 25 10:16:00 CEST 2003 Max_val: 242425 [Microsec] Distance: 0 sec

Flow: 6 Conn: 161.24.3.26->195.3.85.134 Aggregate: 300sec Start/End: Fri Jul 25 11:35:00 CEST 2003|Mon Jul 28 11:35:00 CEST 2003

Flow: 10 Conn: 161.24.3.26->195.3.85.134 Aggregate: 300sec Start/End: Tue Jul 29 11:54:00 CEST 2003|Thu Jul 31 11:54:00 CEST 2003

Flow: 22 Conn: 161.24.3.26->195.3.85.134 Aggregate: 300sec Start/End: Thu Aug 07 15:36:00 CEST 2003|Fri Aug 08 15:36:00 CEST 2003

Flow: 41 Conn: 161.24.3.26->195.3.85.134 Aggregate: 300sec Start/End: Mon Aug 11 14:18:00 CEST 2003|Wed Aug 13 14:18:00 CEST 2003

Outlier:Mon Aug 11 17:28:00 CEST 2003 Max_val: 459359 [Microsec] Distance: 0 sec

Flow: 46 Conn: 161.24.3.26->195.3.85.134 Aggregate: 300sec Start/End: Mon Aug 18 16:17:00 CEST 2003|Wed Aug 20 16:17:00 CEST 2003

Outlier:Tue Aug 19 08:47:00 CEST 2003 Max_val: 482491 [Microsec] Distance: 300 sec

Outlier:Tue Aug 19 12:57:00 CEST 2003 Max_val: 275613 [Microsec] Distance: 900 sec

Flow: 93 Conn: 161.24.3.26->195.3.85.134 Aggregate: 120sec Start/End: Tue Sep 09 18:13:00 CEST 2003|Thu Sep 11 18:13:00 CEST 2003

Outlier:Wed Sep 10 00:05:00 CEST 2003 Max_val: 375256 [Microsec] Distance: 0 sec

(17)

(18)

Flow: 94 Conn: 161.24.3.26->195.3.85.134 Aggregate: 120sec Start/End: Thu Sep 11 18:13:00 CEST 2003|Sat Sep 13 18:13:00 CEST 2003

Outlier:Sat Sep 13 02:19:00 CEST 2003 Max_val: 1169183 [Microsec] Distance: 240 sec

3. Dataset size reduction. 4. BGP-4 patterns. Detection of inter-domain routing problems using BGP-4 protocol patterns P.A.

Newsletter Inter-domain QoS, Issue 8, March 2004

Online monthly journal of INTERMON consortia

Dynamic information concerning research, standardisation and practical issues of inter-domain QoS

---Special issue on detection of faults in inter-domain

environment

There are different faults in inter-domain

environment which could influence the end-to-end

connection behaviour. The network engineer should be

able to detect different kinds of fault behaviour of QoS

parameter values, routing events (BGP-4 protocol

patterns) and traffic to obtain their patterns and to analyse

their causes in the specific context.

INTERMON toolkit allows to study faults based on

patterns (behaviour structures) by usage of dedicated

tools.

We describe the detection and analysis of different

kinds of patterns using INTERMON tools:

-

BGP-4 patterns which could impact the

behaviour of end-to-end connection

-

QoS outlier , i.e. abnormal delay and packet loss

patterns.

Based on systematical measurement and analysis

strategies to study causes of QoS abnormal behaviour

(outliers) it is possible to relate them to discovered

inter-domain routing events such as BGP-4 patterns and

traffic loads.

---Detection of inter-domain routing problems

using BGP-4 protocol patterns

P.A. Gutiérrez

1.

Introduction to BGP-4 database analysis

results

BGP-4 has been the driving force in many

measurement scenarios in the last phase of the

INTERMON project. Due to the lack of access to private

BGP-4 routing data, the project used the incomplete, but

public BGP-4 route repositories.

2.

Scenario definition

Several measurement scenarios have been used in the

INTERMON project. BGP-4 data analysis has been

initially conducted for the measurement lag between

Madrid and Salzburg. During the scenario analysis phase,

it was observed that this measurement lag passed through

the London Internet Exchange (LINX), traversing

intermittently routers which donate their routing data to

the RIPE Routing Repositories.

3.

Dataset size reduction

The data stored in the 'rrc01' directory structure of the

RIPE Routing Repositories was studied for the period

between the 27

of November, 2003 and the 6

of

January, 2004.

During this time, the mean size of the Routing Table

Snapshot files (

bview files) was around 5*10

records.

The mean size of the sum of all daily Routing Activity

files (update files) for one day was around 3*10

records.

This huge amount of data is reduced to 11 significant

routing table snapshot and 77 routing updates records for

the whole aforementioned period.

4.

BGP-4 patterns

During the period under study , different patterns of

updates could be found, depending on the way they are

classified. INTERMON has classified routing updates

based on the router they arrive at. Based on this first

criterion, following patterns were observed

1.

Intermediate AS refresh: An intermediate AS

announces routes to both endpoints in a narrow

time window:

In the previous sample, an AS which is situated

bview files) was around 510*