Impact of Aggregation on Scaling Behavior of
Internet Backbone Traffic
Sprint ATL Technical Report TR02-ATL-020157
Zhi-Li Zhang, Vinay Ribeiro
, Sue Moon, Christophe Diot
Sprint ATL 1 Adrian Court Burlingame, CA 94010 zzhang,vribeiro,sbmoon,cdiot @sprintlabs.com
February 1, 2002
Abstract
We study the impact of aggregation on the scaling behavior of Internet backbone traffic, based on traces collected from OC3 and OC12 links in a tier-1 ISP. We make two striking observa-tions regarding the sub-second small time scaling behaviors of Internet backbone traffic: 1) for a majority of these traces, the Hurst parameters at small time scales (1ms - 100ms) are fairly close to 0.5. Hence the traffic at these time scales are nearly uncorrelated or independent; 2) the scaling behaviors at small time scales are link-dependent, and stay fairly invariant over changing utilization and time.
To understand the scaling behavior of network traffic, we develop analytical models and employ them to demonstrate how traffic composition – aggregation of traffic with differ-ent characteristics – affects the small-time scalings of network traffic. The degree of aggregation and burst correlation struc-ture are two major factors in traffic composition. Our trace-based data analysis confirms this. Furthermore, we discover that traffic composition on a backbone link stays fairly consis-tent over time and changing utilization, which we believe is the cause for the invariant small-time scalings we observe in the traces.
Keywords: Internet backbone, scaling behavior, wavelet
analysis, traffic aggregation, traffic composition.
1
Introduction
Scaling behaviors of the Internet traffic have a significant im-pact on network performance and engineering, and thus have been the focus of much research (see [20] and references therein). Self-similar scaling over large time scales (e.g., 1
This work was done while Z. L. Zhang was a visiting scholar at Sprint ATL.
This work was done while V. Ribeiro was an intern at Sprint ATL.
sec and above) has been observed in a variety of network traf-fic from local area network traftraf-fic (LAN) [16] to wide-area Internet and world-wide-web (WWW) traffic [21, 6]. More recently, more complex and richer, perhaps multifractal-like, scaling behaviors below 1 sec time scales have also been re-ported [17, 7, 8, 18, 12, 22]. Since queuing inside routers and network congestion are strongly influenced by traffic fluctua-tions at sub-second small time scales, understanding of small-time scaling behaviors of network traffic is critical to many network design and engineering problems [19, 14, 11] such as router buffer dimensioning, delay-sensitive service provision-ing, and congestion control. The need for such an understand-ing is particularly acute in the backbone Internet with high-capacity links and growing traffic volumes [11].
In this paper we analyze the scaling behavior of the Internet backbone traffic, and study the impact of traffic aggregation on such behaviors. Our analysis is based on day-long packet traces collected from OC3 to OC12 links on a tier-1 ISP (In-ternet Service Provider). The high-precision time stamping as well as the high link capacity allow us to zoom into finer time scales (say, 1 ms time scale) and perform reliable data analy-sis at these time scales. Furthermore, the large set of day-long packet traces collected on several days enable us to make ob-servations over time and changing link utilization, and com-pare scaling behaviors of traffic carried across links of various types (e.g., links connecting to other ISPs of different sizes, or big corporate customers). In contrast, majority of the previous traffic data traces used in the study of traffic scaling behaviors have less than 100 Mb/sec bandwidth, and typically they are relatively short in duration. Our study, we believe, is the first effort to use an extensive amount of data from a commercial tier-1 carrier to study the impact of aggregation on traffic scal-ing behaviors inside the Internet backbone.
We highlight our major observations below. A more de-tailed discussion of these observations will be postponed until Section 2.3, after we introduce some terminology and describe
our data analysis methodology.
Observation 1: As is previously observed on the Internet
WAN traffic, all data traces exhibit a dichotomy of scal-ing behaviors: above 1 sec or so, the traffic has a clear-cut single self-similar scaling, while below it, the scaling be-haviors appear to be more complex. The transition occurs between 100 ms and 1 sec, regardless of link speed, link utilization, link type and time.
This observation is not surprising; it merely confirms that what was previously observed on relatively low-speed links also holds on high-speed links. What is striking, however, are the observations below regarding the small-time (i.e., sub-second) scaling behaviors on these Internet backbone links.
Observation 2: Small-time scalings are in general more
com-plex and are link-dependent. However, we observe that over a range of small time scales (around – for OC 3 links, and – for OC 12 links), a majority of packet traces manifest uncorrelated or nearly uncorrelated scalings with a Hurst parameter less than 0.6 and often close to 0.5. This seems to indicate that traffic on these backbone links appear to be almost independent at these small time scales!
Observation 3: Furthermore, the small-time scaling
behav-iors of all traces seem to be characteristic of a link – they stay fairly invariant over time, regardless of changing uti-lization and traffic volume on the backbone links. Our observations above raise several intriguing and impor-tant questions: Why does the dichotomy of scaling behaviors occur in the wide-area Internet traffic, and occur around 1 sec time scale? Furthermore, what contributes to the rich and com-plex small-time scalings of the Internet backbone traffic that make them link-dependent, but rather invariant of link utiliza-tion and time? In particular, why do traffic on some links ap-pear almost independent, but others not?
This paper is devoted to answering the above questions. We first attempt to understand, from a theoretical perspective, how the dichotomy of scaling behaviors can occur when self-similar traffic is aggregated in the network, and what are the dominant factors that affect the small-time scalings of aggre-gate network traffic. In particular, we use examples to illus-trate how traffic composition – different mixing of traffic with diverse characteristics – can produce rich and complex scaling behaviors at small time scales. Aided with the insights from theory, we then perform a detailed analysis of traffic composi-tion and investigate its impact on small-time scaling behaviors. By mixing traffic with diverse characteristics in various pro-portions, we show how traffic composition plays a key role in determining the small-time scalings of aggregate traffic. Using the traces, we also demonstrate that traffic composition on the backbone links stays surprisingly consistent over time, pro-ducing the time-invariant small-time scalings we reported in Observation 3 above. We postulate two hypotheses as plau-sible causes for the apparent consistent traffic composition on the Internet backbone links over time and support them with
preliminary evidence. Further work is needed to verify and validate these hypotheses.
Our observations and results regarding the small-time scal-ing behaviors of Internet backbone traffic have significant im-plications in network modeling, service provisioning and traf-fic engineering. For example, if the small-time scaling behav-ior is uncorrelated, we might not need to worry about large-time scale long-range-dependent (LRD) behaviors of network traffic at small time scales. This can lead to simpler network models for analyzing network delay performance. That the small-time scalings are link-dependent, but time-invariant can be used to perform link-specific service provisioning and ca-pacity planning [19, 14, 11]. It also has potential applications in traffic engineering to guide routing and load balancing de-cisions based on network prefixes [24].
The remainder of this paper is structured as follows. In Sec-tion 2 we describe packet traces used in the study, present the data analysis methodology we use, and illustrate our observa-tions through representative examples. A short overview of related work is also provided. In Section 3 we develop analyti-cal models and theoretianalyti-cal examples to mathematianalyti-cally explain the scaling behaviors of network traffic. Section 4 contains a detailed analysis of traffic composition based on the traces to illustrate its impact on small-time scaling of network traffic. This paper is concluded in Section 5.
2
Scaling Behaviors of Internet
Back-bone Traffic
We first provide a short description of the packet traces and how they are collected, then present the data analysis method-ology used in our study. We then illustrate our observations us-ing a few representative traces, and conclude with a overview of related work.
2.1
Packet Traces
For this work, we consider traces collected on OC-3 (155 Mbps) and OC-12 (622 Mbps) links within PoPs (Points-of-Presence) from a tier-1 backbone network. The tier-1 net-work covers a wide geographical area and has a variety of link types. Links within a PoP have 3 (155Mpbs) and OC-12 (622 Mbps) speed, and inter-PoP links have OC-48 (2.5 Gbps) or OC-192 (10 Gbps) speed. The customers are tier-2 or lower ISPs (Internet Service Providers), corporations, and international ISPs. Roughly, the links can be classified into four types1: “peering”2links with other tier-1 ISPs; domestic
“peering” links with tier-2 or lower ISPs; international “peer-ing” links with non-U.S.-based ISPs; and “corporate” links to corporate customers, such as Fortune 500 companies, web hosting sites, and web server farms.
1Please note that classification of link type is somewhat subjective and
ambiguous, as it is impossible to precisely pin down the role of each network in these terms.
2By peering links we mean the links carrying traffic between the ISP
net-works in question; they do not reflect the contractual agreements nor BGP policies between the ISPs.
T0 T1 T T 2 3 2k 2k+1 k Finer (Smaller) Coarser (Larger) Time Scales
Figure 1: Time Scales and Dyadic Time Index System.
The packet traces were collected at three different points of presence (PoPs): two in the east coast, and the other in the west coast. Several weeks worth of data has been collected, but in this paper we focus on traces from only two days, namely, Aug. 9th, 2000 (OC-3), and Sep. 5th, 2001 (OC-12). Each trace is a sequence of packet records containing the first 40 bytes of a packet, and a GPS-synchronized timestamp (with an accuracy down to 5 s) which indicates when the packet was observed. For further details about the measurement system, see [11].
Table 1 describes part of the traces used in our study that led to our observations highlighted in the introduction. The listed traces are used in this paper as examples to illustrate our observations and analyses.
2.2
Data Analysis Methodology
Following several other studies, we employ wavelet analysis as our scaling analysis tool. Wavelets provide robust estima-tors of the Hurst parameter while eliminating polynomial nonstationarities in traffic [2]. To introduce the essential ter-minology for our discussion of observed scaling behaviors in Section 2.3, we provide a quick primer on wavelet analysis using the simplest form of wavelets, the Haar wavelets.
We first introduce a dyadic time index system for repre-senting time intervals at different time scales (see Figure 1). Fixing a reference time scale , define time scale as
. At each time scale (scale! for short), let
"
$#%&('*) + ',).- /0 / denote the) -th time unit (of length ). Consider a (stationary) traffic process1 . At time scale ! , 1 2#% denotes the amount of traffic (i.e., the total bytes) ar-riving in time unit"
$#% . Then1 354 1 $#%6+7)98;:<:>= represents the traffic process that is observed at time scale! . The Haar wavelet coefficients?@$#% of the traffic process at time scale! are defined as follows:
? $#%AB 2C2D ' 1 $EFG#D7%IH 1 $EFG#D2%JEF /JK (1)
The energy function of the traffic process at time scale! is then
given by L MONQP? D 2#%R (2) whereNSPUT R denotes expectation.
3Average bandwith is computed over the entire trace.
The energy function captures the second-order statistics of a traffic process, and its scalings as a function of time scale. As an example, consider a (second-order) self-similar process such as fractional Gaussian noise (fGn), which has a single self-similar scaling specified by the so-called Hurst parameter
' VKXW@YZ[Y\/: 1 2#D2%^]_'a` /0bc1 F7#% , )d8e:<: . Here ] denotes equality in distribution.4 In other words, as we move
down from time scalef
F tog g
F` , the traffic process 1V is scaled (in measure) from 1h
F by a factor of ' a` /7b . A consequence of this scaling behavior is that variance of1i scales as follows: varP1j R O D b varP1k R K (3)
The Hurst parameter tells us how correlated the process is – how its value at a future time depends on its value at the current time. For example, fGn with jKW is the so-called white
noise- its values are uncorrelated; while fGn with mlnjKW means that its values are positively correlated.
The relationship of the scaling (i.e., Hurst parameter) of fGn and its energy function is given by the following equation [15]:
L B JopD b F0q L rK (4) Define s t H K (5) Thenupvrw L QxH ! s - upvrw L , i.e.,uyvw L
scales linearly with time scale H ! . Hence the slope of a simple plot of upvrw
L
againstH ! (the negative sign indicates coarser time scales with increasing values on the x-axis), called the energy plot, gives an estimate5of . Clearly if jKW , s (a flat slope) whilezldVKXW yields s ld (a positive slope).
Unlike fGn processes which have a single scaling (a straight line in the energy plot), many processes encountered in prac-tice have different scalings at different time scales. In other words, such processes will not have straight line energy plots. Nonetheless, the slope ofuyvw
L
(against H ! ) at time scale! (denoted by
s
) will tell us how correlated a traffic process is at and near time scale! . Using (5), define{
5' s -a/$` , which is referred to as the (local) Hurst parameter at time scale ! . (We will drop the subscript! when the context is clear.) Hence| captures the scaling of a traffic process at and near time scale! . In the sequel we will refer to ~}nVKXW as
un-correlated scaling,nVK as nearly uncorrelated scaling, jKY(eVK as moderately correlated scaling, andzljKX as strongly correlated scaling (or strong self-similar scaling).
2.3
Illustration of Major Observations
In this section we analyze the scaling behaviors of the Inter-net backbone traffic using wavelets. For all traces we form a time series by counting the number of bytes every s. The traces are partitioned into (overlapping) segments of hour
4Namely, t , if and only if$tQr .r for any .
5From (3), we can also estimate the Hurst parameter
using the variance
plot: plottingXGga
J
againstg . However, it is not as reliable as the
Trace Link Type From/To the Tier-1 ISP Start Time Duration Speed Avg. Bandwidth3
OC3-tier1-dom domestic tier-1 peer To Aug. 8, 2000 15hrs OC-3 42Mbps
OC3-tier2-dom domestic tier-2 peer From Aug. 8, 2000 23hrs OC-3 44Mbps
OC3-corp-dom corporate From Aug. 8, 2000 19hrs OC-3 28Mbps
OC12-tier1-dom domestic tier-1 peer From Sept. 5, 2001 8hrs OC-12 234Mbps
OC12-tier2-int international tier-2 peer From Sept. 5, 2001 6hrs OC-12 228Mbps
OC12-tier2-dom domestic tier 2 peer To Sept. 5, 2001 7hrs OC-12 187Mbps
OC12-corp-dom corporate From Sept. 5, 2001 20hrs OC-12 122Mbps
Table 1: Trace Description
100 101 102 103 104 15 20 25 H=0.5799 time scale (ms) log2(Wavelet Energy) H=0.97113 100 101 102 103 104 15 20 25 H=0.52328 time scale (ms) log2(Wavelet Energy) H=0.99403
(a) OC12-tier1-dom (b) OC12-tier2-int
100 101 102 103 104 15 20 25 H=0.55133 time scale (ms) log2(Wavelet Energy) H=0.82633 100 101 102 103 104 15 20 25 H=0.69341 time scale (ms) log2(Wavelet Energy) H=0.84333 (c) OC12-tier2-dom (d) OC12-corp-dom
Figure 2: Energy plots of OC traces.
101 102 103 104 15 20 25 H=0.56042 time scale (ms) log2(Wavelet Energy) H=0.8758 101 102 103 104 15 20 25 H=0.57185 time scale (ms) log2(Wavelet Energy) H=0.89718 101 102 103 104 15 20 25 H=0.52227 time scale (ms) log2(Wavelet Energy) H=0.84896
(a) OC3-tier1-dom (b) OC3-tier2-dom (c) OC3-corp-dom
Figure 3: Energy plots of OC traces.
each, the beginning of the segments being separated by minutes. We then perform a wavelet analysis of each segment using a Daubechies wavelet with vanishing moments.
6
6The wavelet analysis code is based on the MATLAB programs from [1].
As representative examples to illustrate the observations stated in Section 1, Figure 2 and Figure 3 show the energy plots for a one-hour segment from the backbone packet traces listed in Table 1. We see that all plots show a dichotomy of scaling behaviors, the “knee” point (the transition region of scaling
behaviors) occurring between the 100ms - 1 sec time scales, typically with a (slight) dip of energy in this region. Above 1 sec or so, a single linear self-similar scaling is apparent in all plots. Below 100 ms, the plots exhibit richer and more complex scaling behavior. The energy plots corresponding to majority of links have a rather flat slope in small time scales:7
for three OC-12 peering links, OC12-tier1-dom, OC12-tier2-int, OC12-tier2-dom,YVK over the time scale range of 1 ms - 100 ms (Figure 2(a)–(c)), as well as for the three OC-3 links over the time scales 10 ms - 100 ms (see Figure 3(a) and (c)). In contrast, the OC-12 corporate link (OC12-corp-dom) in Figure 2(d) has a moderately correlated scaling (_}tjKX ). To demonstrate that link utilization and traffic volume are not major factors in determining the small-time scalings of network traffic on these Internet backbone links, in Figure 4 we juxtapose the energy plots of two one-hour segments with different utilization from the same trace, OC3-tier2-dom. Al-though the utilization of one period is about times higher than the other, the local scalings of the two periods are fairly consistent. As an another example, compare the energy plots in Figure 2(a) and Figure 3(a). Although they are from two traces with different link speeds and traffic volumes, the small-time scalings look remarkably similar. Table 2 summarizes the Hurst parameters in small (1-100ms for OC-12 links and 10-100ms for OC-3 links) and large (above 1 sec) time scales measured over five different one-hour segments of the same traces mentioned above.8 Clearly, the small-time scalings are link-dependent, vary from link to link, but are fairly consistent over time on the same link. The small-time scalings of major-ity of the links have a value less than 0.6, implying that traffic on these links appear to be nearly independent. The corporate OC-12 link, however, shows moderately correlated small-time scalings.
The examples above suggest that the scaling is not a func-tion of total traffic volume, link utilizafunc-tion, or speed on an Internet backbone link. It also reveals that the high degree of aggregation on a backbone link in itself does not cause the traffic to appear independent at small times. Rather some other traffic properties and network factors that remain largely time-invariant determine the small-time behaviors we observe.
In an effort to understand the scaling behavior of network traffic we observe, in Section 3 we develop analytical models and theoretical examples to mathematically explain the scal-ing behaviors of network traffic. Guided by the theoretical insights, in Section 4 we perform a detailed analysis of traf-fic composition based on the traces. Our analytical models as well as trace-based analysis uncover traffic composition, i.e., mixing of traffic with diverse characteristics, as a likely cul-prit for the complex small-time scalings of aggregate network
7The smallest time scale we study for 3 links is 10ms and 1ms for
OC-12 links. The reason is that below these time scales the degree of aggregation is fairly low in our traces, thus allowing the variation in packet sizes and other factors to affect the scaling. For example for OC3-tier2-dom the average num-ber of total active flows is only about 10 at the 1ms time scale (see Sections 3.3 and 4 for the definition of active flows). Analyzing the factors affecting scaling below these time scales is part of ongoing work.
8In general the estimates of
at large time scales are not as accurate as
those at fine time scales due to the presence of fewer data points.
traffic: the degree of aggregation (in terms of relative propor-tions of diverse flows) as well as burst correlation structure at small time scales are two major factors. We also observe that traffic composition is fairly consistent over time on a backbone link. We put forth two hypotheses as plausible causes for this phenomenon. Before we delve into the details of our analysis, we conclude this section with a short overview of the related work. 101 102 103 10 15 20 25 H=0.55245 time scale (ms) log2(Wavelet Energy) H=0.58361 10 Mbps 100 Mbps
Figure 4: Scaling plots with changing utilization for OC3-tier2-dom.
2.4
Related Work
The self-similar scaling of LAN traffic was first discovered and reported in the seminal paper by Leland et. al. [16], and subsequently has been found in traffic from a variety of other networks. Description of self-similar network traffic, their modeling and impact on network performance be found in the excellent survey book [20]. More recently, the complex small-time scaling behaviors have been reported in a number of studies [8, 17, 18, ?], most notably [7], where a random multifractal cascade is used for modeling small-time scaling behaviors. Our observations corroborate these earlier studies. In [9], through simulation, the authors prescribe TCP as a pos-sible cause for the diverse small-time scaling behaviors of net-work traffic. In several recent studies, it is shown through ei-ther simulation and/or theoretical analysis that TCP in itself can generate (pseudo-) self-similar behavior (i.e., local Hurst parameter l\jKW ) in a small range of time scales [10, 3]. Although a cascade provides a useful model for describing network behaviors in small time scales, in itself it does not point to the dominant factors that cause such behaviors. The TCP effect in generating self-similar flows also does not pro-vide an answer as to why on peering links traffic appear nearly uncorrelated at small time scales. In another set of recent stud-ies [5, 4], based on analysis of inter-packet arrival times, the authors conclude that packet arrivals and sizes can go locally to independence with increased statistical multiplexing. Our study differs from theirs in two important aspects: (1) we fo-cus on the second-order statistics of traffic volume fluctuations at small time scales (1ms – 100ms), not packet arrivals or sizes; (2) we find that high degree of aggregation in itself does not
Hurst parameter (small time scales) Hurst parameter (large time scales)
Trace Hour 1 Hour2 Hour3 Hour4 Hour5 Hour1 Hour2 Hour 3 Hour4 Hour 5
OC3-tier1-dom 0.56 0.55 0.55 0.55 0.57 0.87 0.88 1.06 0.81 0.85 OC3-tier2-dom 0.57 0.55 0.58 0.57 0.56 0.89 0.89 0.93 0.96 0.86 OC3-corp-dom 0.52 0.52 0.52 0.51 0.46 0.84 0.92 0.91 0.91 0.92 OC12-tier1-dom 0.57 0.56 0.59 0.56 0.57 0.98 0.96 0.98 0.94 0.99 OC12-tier2-int 0.52 0.53 0.52 0.51 0.52 0.99 0.98 0.93 0.95 0.99 OC12-tier2-dom 0.55 0.54 0.53 0.53 0.54 0.82 0.84 0.87 0.84 0.79 OC12-corp-dom 0.69 0.67 0.70 0.69 0.68 0.84 0.81 0.82 0.89 0.90
Table 2: Estimated Hurst parameters for 5 different 1 hour segments.
produce uncorrelated scalings, and that instead the burst cor-relation structures of flows decide the scaling.
3
Understanding Traffic Scaling
Be-haviors: Analytical Models
In this section we develop analytical tools for modeling and studying traffic processes at different time scales. Using these analytical tools and models, we provide mathematical insights into the multi-time-scale behaviors of backbone network traf-fic. In particular, we explain when and why the dichotomy of the large and small time scale behaviors occurs, and how aggregation of traffic with different characteristics affects the scaling behaviors of network traffic at small time scales.
3.1
Time Scales, Scaling Operators, and Local
Hurst Parameters
Using the dyadic time index system described in Section 2.2, we first introduce some useful notation to represent network traffic observed at different time scales. Consider a traffic flow,9 referred to as flow ¡ , that is observed at a backbone link. Let 1
op¢q
denote the traffic process of flow
¡ observed at time scaleg , namely,1
op¢q £4 1
op¢q $#%
+2)@8:<:>= , where1j$#% represents the amount of traffic belonging to flow¡ observed on the backbone link during time unit"
$#% . For conciseness,
we will drop the flow index¡ whenever there is no confusion. Clearly, the traffic processes1k
F and1j observed at two dif-ferent time scales ! H and! are related by the following relationship:
1V
FG#% 1V$#D7% - 1V$#D7%GEF +)¤8&:<: (6) To represent the relationship between 1>
F and1j formally, we introduce the notion of a scaling operator, denoted by¥¦ , which maps1 F to1 : ¥f '1V F/ 1 + where¥ '1V FG#%/ §4 1V$#D7% + 1V$#D7%GEF = K (7) We can intuitively think of the traffic processes of flow¡ at different time scales as follows: 1k describes, via (7), how the
9In Section 4, we will discuss how we define a flow in our empirical study.
traffic of the flow at a given time scale! H “splits” or “perco-lates” into time units of the finer time scale! , as we “zoom in” and examine the traffic flow at finer and finer time scales (see Figure 5). The properties of the scaling operators¥¨ ’s specify the “percolation process” of flow¡ as we move down the time scales, and consequently they determine the scaling behaviors of the observed traffic processes1f ’s of flow¡ . To illustrate that different scaling operators produce different local scalings (i.e., local Hurst parameters), we consider two special exam-ples.
Similarity Scaling Operators and Self-Similar Processes:
For! +2V+ KK K, suppose that the scaling operator¥© satisfies the following property
1V$#D7% ]ª F 1 F7#% + and1j$#D7%GEF ]«ª F 1V FG#%GEF (8) whereª@¬ .
Intuitively, (8) says that as we move down the time scales, the traffic of the flow at time scale! “looks” similar to that at time scale! H : it were as if produced from1
F by, statisti-cally speaking, “shrinking”1
F with a factor ofª . From (8), we write1j ¥> '1V F / ]tª F 1V
F . We refer to this type of scaling operators,¥ £ª
F
, as similarity scaling operators, since it is intimately related to self-similar processes.
To study the scaling behaviors of1> ’s, we compute the
en-ergy function
L
at time scale! using the Haar wavelets. From (1) and (8), we have
?$#% O>®
¯
'1V$EFG#D2% H 1V$EFG#D2%GEF /
]O ® ¯ ª F '1 $#%¦H 1 2#%GEF /°]B± ª F ? F7#% K Hence L _NQP? D 2#% R ª D0 L . Assuming that L ²l³ , we have upvrw L M ! ' H upvrw ª / - uyvw L Hence s´ µ uyvw L µ 'H !¶/
·H -d uyvw ª and uyvw ª K (9) We now relate the similarity operators to self-similar cesses. Consider a (continuous-time) exact self-similar pro-cess ¸ '
"
/ with Hurst parameter S¹lº»` , i.e., ¸ '½¼
" / ] ¼ b¾¿¸ ' " /. Suppose that 1 #%À ¸ ' " #% /, and for !
+7 + KKK, 1 is generated from 1k
F via a similarity opera-tor¥ @Áª F . From 1 F7#D7% ]Áª F 1 #% ]Áª F ¸ ' " #% / and 1hF7#D7%GEF ]ª F 1g #%GEF ]ê F ¸ ' " #%GEF/, we have 1>F7#D7% -1 F7#D7%GEF ]nª F '¸ ' " #% / - ¸ ' " #%GEF //{]nª F ¸ '* / . But 1hF7#D7% -1FG#D7%GEF 1j #% ¸ ' " #%»/ ]£' a` /0b ¾ ¸ ', k/. This implies thatªB b ¾ . Therefore the local Hurst parameter of1j at time scale! equals toupvrw ª Q¹ . This result shows that if1g comes from a self-similar process, and1f ’s are gen-erated by similarity scaling operators, then1 ’s preserve the scaling (i.e., the Hurst parameter) of the original process.
Random Cascade and Multiplicative Processes: For !
+7 + KKK, let Ä be a series of independent random variables (rv’s for short) with NQPÄ
R
a` . Assume also that they are independent of the traffic process1 . Suppose that1 can be generated statistically from1
F via the following relation (we write¥ ¨ Ä ): 1 $#D7%A Ä 1 FG#% and1 2#D2%GEFM§' H Ä /01 FG#% K (10) The resulting processes 1
’s are referred to as a random
cas-cade10 [23]. Expanding the relation (10), we can “trace”
1
back to the traffic processes at coarser time scales, say, up to time scaleh : 1 $#%A Ä $#% ® Ä F7#% ®ÆÅ¶Ç TT T Ä FG#% Ç 1 #%7È (11) whereÉ)rr+2)6F»+ KK K +2)»
F+2)»Ê is the dyadic expansion of) (up to ), i.e.,)3O) -) F F -9T TT$) F -) , andÄdË#%GÌ Ä;Ë if) Ë , H Ä;Ë if) Ë .
Because of (11), the traffic process1 is also said to be
mul-tiplicative, and the rv’sÄ are referred as the multipliers. For each! ,NQPÄ9
R
a` . Using (11) and independence ofÄt ’s, it can be shown (see e.g., [13], or Chapter 20 in [20]) that
L O NQPÄ D F R T TT$NSPÄ D R NQPy', Ä^$EF H / D R NQP1 D R K (12) If Ä ’s are i.i.d rv’s, i.e., Ä m Ä , then
L m NQPÄ D R NQPy', Ä H / D R NQP1 D R. This yields uyvw L c ! ' - uyvw NQPÄ D R / - upvrw NQPy', Ä H / D R - upvrw NQP1 D iR K Therefore s ·H H uyvw NSPÄ D R and H upvrw NQPÄ D R K (13)
This result states that if the traffic processes1k ’s are generated by a random cascade with a series of i.i.d. rv’sÄ , then their local scalings (i.e., local Hurst parameters) are determined by the second moment ofÄ . In particular, assumingÍ ¼¶Î PÄ
R } a`»Ï , we haveNSPÄ D R Í ¼¶Î6PÄ R -eNSPÄ R D }x»` , andÐ} VKXW .
3.2
Mass-Preserving
Time
Scale
and
Di-chotomy of Scaling Behaviors
Using the analytical tools and models introduced in the pre-vious section, we provide a theoretical explanation for when
) Ñ ( Ò n Ó Y time scales
observed traffic processes of flow n
Ô
at different time scales
Õ ) Ö ( × , Ø 0 Ù n Ú k Û Y ) ( 1 2 , 1 n Ü k Ý Y ) Þ ( ß 2 à , á 1 n â k ã Y ) ä ( å , æ 0 ç n è k é X ê ) Ñ ( Ò 1 ë 2 ì , í 1 ë n Ó k î X ) ( 2 , 1 n Ü k Ý X ï backbone link ð ) Ö ( × n Ú X
original traffic processes of flow n
Ô
at different time scales
Õ ) ñ 1 ( ò X . . . . . . . .
Figure 5: Relationship between Original and Observed Traffic Processes at Different Time Scales.
and why the dichotomy of scaling behaviors occurs in network traffic.
For each flow¡ observed on a backbone link, let¸ op¢q 2#% de-note the amount of traffic injected into the network by flow ¡ during each time unit
"
2#% at time scale ! . We refer to
¸ op¢q 4 ¸ op¢q $#%
+2)ó8Z:<:>= as the original traffic processes of flow¡ at time scale! (i.e., before it enters the network), in con-trast to the observed traffic process1
op¢q
at the same time scale (see Figure 5). For ease of exposition, we assume that prop-agation delay is zero11. Under this assumption, if there were
no other cross traffic in the network that delay or disturb the traffic of flow¡ in some manner, then¸
op¢q 2#% 1 op¢q 2#% for all !
and) . Of course, because of cross traffic in the network, this is
not true in general. Especially at fine time scales (say, around 1 ms time scale or less), it is highly likely that¸ $#%õô 1 2#% , because small delays (say, 1 ms) suffered by packets of flow¡ would disturb the original traffic process at these time scales. On the other hand, we expect that at sufficiently coarse time scales (say, 1 minute time scale), ¸Q$#%}ö1 2#% , as the small disturbance caused by the network becomes negligible.
We say that time scale! is mass-preserving (or statistically
mass-preserving) w.r.t flow¡ , if for)S8@:<: ,
¸ op¢q $#% 1 op¢q $#% (or ÷ ζ4 ¸ op¢q $#% 1 op¢q 2#% = }· ). (14)
Note that because (6) holds for both1 oy¢q ’s and
¸
op¢q ’s, if time scale! is mass-preserving for flow¡ , so is any coarser time scale level!6øhY! . Letù
op¢q
be the finest mass-preserving time scale for flow¡ , by which we mean that for any!{§ùh ,
1 op¢q $#% ¸ oy¢q $#% ,
)«8ó:<: . An important consequence of mass-preserving time scales is the following observation. Suppose that the original traffic processes ¸
op¢q
’s of flow ¡ are self-similar with Hurst parameter ¤¹ . Then for!·Âù
oy¢q , the
10To be more precise, it is assumed that the underlying measures of this
series of rv’s converges to a limiting measure.
11Otherwise, in the discussion that follows, we need to shift the time index
for the
scaling operator¥ of the observed traffic process1> must be a similarity scaling operator withªO b¾ .
We now turn our attention to the aggregate traffic of the backbone link. Let 1
oyúGq
denote the aggregate traffic pro-cess on the backbone link at time scale ! . Namely, 1
ú 4 1 oyúGq $#% +2)98d:<:>= , where1 o<úGq
$#% denotes the amount of aggregate traffic observed on the backbone link during the time unit"
$#% . Clearly, for any! and)
1 o<úGq $#% Oû ¢ 1 op¢q 2#% (15)
where the summation is over all flows traversing the backbone link. Define ù oyúGq Zü3ýpþk¢ ù oy¢q , i.e., ù o<úGq
is the finest time scale that is (statistically) mass-preserving w.r.t. all flows traversing the backbone link. Hence for any!.Où
oyúGq and )¤8&:y: ,1 op¢q $#% ¸ op¢q $#% , and thus 1 o<úGq $#% Bû ¢ ¸ op¢q 2#% K (16)
Suppose that the original traffic processes¸ op¢q
of each flow ¡ are self-similar with a Hurst parameter¹£lBjKW , and that the flows are independent of each other. As for any! ¬ ù
oyúJq
,
the scaling operator¥ op¢q
of each flow is a similarity operator withª\ b ¾ , (16) and the independence of the flow traffic processes imply that the scaling operator ¥
oyúJq
of the aggre-gate traffic process 1
oyúGq
at time scale
! is also a similarity operator withªÿ b ¾ . Hence for time scale!;Yÿù
o<úGq , the aggregate traffic process has a local Hurst parameter equal to 3¹ . In other words, above the mass-preserving time scale
ù
oyúGq
w.r.t. all flows, the aggregate traffic exhibits a single lin-ear self-similar scaling with a Hurst parameter determined by the original flow traffic processes.
Consider now the time scales smaller than ù oyúJq
. For each flow¡ , assume that for!{l§ù
op¢q
, the observed traffic process
1
oy¢q
of flow
¡ is generated by a random cascade with a series of i.i.d. random variables Ä
op¢q . Define ù oyúGq F \ü ¢Mù op¢q .
Then it is not too hard to see that for!lZù oyúGq
F , the aggregate traffic process 1
o<úGq
can also be represented as a random cas-cade with a series of appropriately defined random variables. In particular, if Ä
op¢q
Ä . Then the aggregate traffic pro-cess1
o<úGq
is also generated by a random cascade with a series of i.i.d. random variables Ä
o<úGq Ä . If NQP Ä R »` and Í ¼rÎ PÄ R
a`»Ï , then at any time scale !lºù oyúGq
F , the local Hurst parameter of the aggregate traffic process 1
o<úGq is 0.5. More generally, dependent on the properties of the underly-ing random cascades, the resultunderly-ing aggregate traffic can exhibit rich and complex scaling behaviors.
Does Mass-Preserving Time Scale Exist in Practice? The
example above demonstrates in theory why, when observed on a backbone link inside the network, aggregation of originally self-similar traffic processes can exhibit two vastly different scaling behaviors – above a certain time scale the self-similar
scaling is preserved, while zooming in sufficiently small time scale, the traffic can appear purely uncorrelated. It also reveals when such dichotomy of scaling behaviors can occur, and il-lustrates the key role the (finest) mass-preserving time scale
ù
oyúJq
plays in the transition from multiplicative process to
ad-ditive process, as we move from small time scales to large time scales. In practice, we believe that mass-preserving time scale does exist and its existence can be attributed to the feedback control of TCP. It is well known that today’s Internet traffic is predominantly TCP traffic. A TCP flow uses a window-based feedback control to self-pace and regulates the amount of traf-fic it injects into the network. In addition, end-to-end delay jit-ter is significantly smaller than the round-trip-time (RTT), and network packet loss rate is typically also quite small. Hence, over time scales that are a multiple of RTT, the amount of traf-fic a TCP flow injects into the network is (almost) the same as what would be observed on a backbone link. Since RTT of most flows is typically in the order of 10-100 ms, 1 sec is likely a multiple of RTTs for almost all TCP flows. In other words, 1 sec time scale is mass-preserving w.r.t. almost all TCP flows. This may explain why we see a clear-cut single self-similarity scaling above 1 sec time scales. This argument is corroborated by the study in [9] where it is shown that RTT in TCP feedback control affects where the “dip” (or the “knee”) of the energy plot occurs.
3.3
Small-Time Scalings of Aggregate Traffic
In the previous section we explained why dichotomy of scal-ing behaviors in network traffic occurs naturally due to the existence of mass-preserving time scales. In addition, using the notion of random cascade, we demonstrated how aggre-gate network traffic can exhibit more rich and complex scaling behaviors at small time scales (below the massing-preserving time scales). Although random cascade can be used to describe the scaling behaviors of the aggregate traffic, in itself it does not point to the factors that give rise to such behaviors, in par-ticular, when and why apparently uncorrelated scaling occurs in aggregate traffic. As an attempt to answer these questions, in this section we turn to a different model and use it to study the impact of traffic composition – aggregation of traffic with different characteristics – on the scaling behaviors of aggregate traffic at small time scales. For conciseness, in the following we assume thatù
oyúGq
, and consider time scales!lZ as
small time scales.
Suppose that the aggregate traffic is composed of indepen-dent flows with similar characteristics, by which we mean that the traffic processes1
op¢q
of the flows have the same distribu-tion (denoted by1h ) and are independent. We are interested in understanding the dominant factors that affect the local Hurst parameter of the aggregate traffic at different time scales. For any!ól( and ) 8£:<: , define
$#% ¢ 4 1 op¢q $#% lö = , where is the indicator function. In other words,
$#% counts the number of flows that are active, i.e., sending a “burst” of size1 op¢q 2#% lt , in time unit " $#% . Using
aggregate traffic1 oyúGq 2#% observed in " $#% as 1 oyúGq $#% û ¢ 1 oy¢q $#% 4 1 op¢q 2#% le = ] ® û ¢F 1 op¢q $#% (17)
where in the last equity we used the i.i.d. assumption of
1
oy¢q $#% ’s.
Conditioning on
2#% , it is not too hard to show that NQP1 oyúJq 2#% R tNSP $#% R NQP1 $#% R and Í ¼¶Î P1 oyúGq $#% R ONQP 2#% R Í ¼¶Î6P1 $#% R - Í ¼¶Î P $#% R NQP1 2#% R D K (18) Eq. (18) reveals that another way to analyze the scaling be-haviors of the aggregate traffic is to examine how the “flow counting” process
I 4
2#%¶+2){89:<:>= and the “flow burst”
process1 M§4 1 $#%¶+7)S8@:<:>= evolve as a function of time scale ! . Note in particular that evolution of
and1 as a func-tion of time scale! are not independent. To understand this relationship between and1 , define $#% ' / Oû ¢ 4 1 op¢q 2#% T 1 op¢q $#%JE>Ë l =r+!3le ,)+8:<: . (19)
2#% ' / counts the number of common active flows to
"
2#% and"
2#%GE>Ë . From the definition of
$#% , 2#% $EF7#D7% $EF7#D7%GEF H
$EFG#D2% '/. More generally, we can write
2#% as a sum of
$E#%’s minus a sum of various
$E#% ' /’s, where summation is over appropriate time units at time scale ! - that is contained in the time unit
"
$#% . We see that the stronger the correlation among the bursts of each flow at time scale! - is, the smaller
2#% is. In a sense
2#% captures the
(local) degree of aggregation – the number of different flows that contribute bursts in a time unit"
$#% at time scale ! . As we zoom out from a finer time scale to a coarser time scale, the variability in
2#% tells us how “well-mixed” the aggre-gate traffic is: does it comprise of bursts mostly from the same flows or from vastly different flows?
The degree of aggregation at a local time also has direct implication in how 1$#% evolves as we zoom out from finer time scales to coarser time scales. Intuitively, if the aggregate traffic is less well-mixed, i.e., bursts are likely from the same set of flows, then as we zoom out from a finer time scale to a coarser time scale, the flow burst size1
op¢q
2#% grows larger, as neighboring bursts of each flow amass into bigger ones. On the other hand, if the aggregate traffic is well-mixed, then as we zoom out from a finer time scale to a coarser time scale,
1
oy¢q
$#% would grow very slowly since the bursts of each flow is less likely to encounter each other and coalesce. To illustrate how the properties of
and1j affect the local scalings of aggregate traffic, we consider several special examples.
Small Flows with Poisson Arrivals: In this example, we
assume in a time span of
Ë
, flows arrive according to a Poisson process with rate , and that each flow brings a small random burst 1 only in one of the time units of length . Then for ù H !ÿ[ù , 1 1 . By the Pois-son arrival assumption, we see that
is a Poisson process with NQP R V , and Í ¼rÎ P R j . Furthermore, the
aggregate traffic process 1h is a compound Poisson process withNQP1 o<úGq R jg NQP1 R, and Í ¼rÎ P1 oyúGq R Vg NQP1 D R. As g O , we have uyvw Í ¼¶Î P1 oyúGq R §H ! - uyvw¿ ©- upvrw NQP1 D R K (20) From uyvw¿Í ¼¶Î6P 1 oyúGq R HM
!r , we have that the local Hurst parameter of1
oyúJq
at time scale ! is 0.5.
This example intuitively says that if in a given period of time, we have a great number of flows arriving randomly, each carrying a small burst, then the (local) small-time scalings of their aggregate will likely appear to be uncorrelated, i.e., with a Hurst parameter close to 0.5.
Large Flows with Self-Similar Bursts: Assume that in a
time span of
Ë
, Ä flows send a random burst of size 1 #%l during every time unit
"
#% of length . The burst process 4 1 #% +7)·8ÿ:<:>= is self-similar with Hurst parameter "! . Clearly forù H !Qóù , Ä , andÍ ¼¶Î P R . AsÍ ¼¶Î6P1 R D b$# Í ¼¶Î P1 R, upvrw Í ¼rÎ P1 oyúJq R uyvw '½NQP 2#% R Í ¼¶Î P1V$#% R / HM !¶"! K (21) Therefore, the local Hurst parameter of1
oyúGq
at time scale ! is "! . This example shows that if we have a certain number of large flows sending bursts frequently in some period of time, then the scaling behavior of the aggregate of these large flows is determined by the correlation structure of the bursts sent by the flows. If the variance of the bursts scales with a Hurst parameter%!·l§VKXW , then the local the aggregate traffic will exhibit correlated scalings.
As a special case of the above example, suppose that the burst size1V2#% each flow sends during the time unit
"
2#% is
in-dependent of what is sent in other time units, i.e., bursts of each flow at time scale! is uncorrelated. LetÍ
¼¶Î P 1$#% R & D . Then Í ¼¶Î P 1V$#% R Í ¼rÎ P 1 $EF7#D7% R -Í ¼¶Î P 1V$#D7%GEF R Í ¼¶Î6P 1V$EF7#D7% R & D
. In other words, we must have
!Á VKXW . There-fore, the local Hurst parameter of 1
oyúGq
is
VKXW . This special case shows that large flows sending bursts frequently do not in themselves result in a local Hurst parameter(leVKXW , it is the correlation in bursts that matters!
Mixture of Large and Small Flows: We now combine the
two examples above and analyze the local scalings of the re-sulting aggregate traffic processes. Namely, in a time span of
Ë
, we haveÄ large flows that send a random burst of size 1('
#% during each time unit
"
#% of length , and a (possibly infinite) number of small flows that arrive according to a Pois-son process with rate , each of which brings a small random burst1) only in one of the time units of length* . Results from the previous examples yield: forù H+ !.Bù ,
Í ¼¶Î P1 oyúGq R ÄóÍ ¼rÎ P1 ' 2#%aR - jj NSPp'1 ) / D R K (22)
Using the fact thatupvrw ',¼¦--, /
üg4 upvrw ¼k+uyvw ,a= , we have
uyvw Í ¼¶Î P1 oyúGq R ü".0/ uyvw 'ÄóÍ ¼¶Î6P1 ' $#%aR / + upvrw 'Vg NQPy'1 ) / D R /21MK (23)
Define ù6F ' ! H uyvw ü(3 ÄóÍ ¼rÎ P1' R V aNQPp'1 ) / D R + 54 (24) if F D Yd6' ! O , andù F if7' ! jKW .
With some algebra, it can be shown that if! ¬ ùhF, the first term in the right hand side of (23) is smaller than the second term. Therefore we have that forù H8 !.Y ü3ýpþ>4 ùk + ù = ,
uyvw Í ¼¶Î P1 oyúGq R HM !¶ ' ! (25)
and forùjF¦9!QOù ,
uyvw Í ¼¶Î P1 oyúGq
R
H !rK (26)
As this example illustrates, if we have a mixture of flows with different characteristics (e.g., small flows with random bursts vs. large flows with correlated bursts), the scaling behavior of the aggregate traffic changes as the time scale varies, and the local Hurst parameter is primarily determined by what type of flows dominates at that time scale. For instance, in the above example, over the coarser time scales (ù
H £!tY ü3ýpþf4 ù + ù
= ), the local scalings of the aggregate traffic is de-termined by the property of large flows, resulting in a local Hurst parameter
oyúGq
7'
! . However as we zoom in fur-ther into finer time scales, small flows start to dominate in the aggregate traffic behaviors, producing uncorrelated local scal-ings (i.e.,
oyúGq
jKW ) at these time scales. The transition of local scalings occurs at a time scale when the intensity of bursts from small flow overwhelms the “bond” (i.e., correla-tion) among the bursts from large flows. From (24) note that ù F depends only on the ratio of Ä and , not their absolute values. Hence the transition in local scaling behaviors hinges only on the traffic composition, in this case the proportions of large and small flows and their respective characteristics. We can extend this example of mixture of large and small flows to more general cases with more diverse traffic compositions such as flows with different sizes, arrival processes and burst correlation structures. For brevity, we will not get into them here.
In summary, the above examples serve to illustrate that at small time scales, the local scalings of aggregate traffic de-pend heavily on how bursts from different flows are mixed in a local time span. The flow burst characteristics (in lieu of the second-order properties of1 ’s) and the degree of aggregation (in lieu of the second-order properties of
’s) are two dom-inant factors. Traffic composition – how proportions of flows with distinct characteristics (e.g., large and small flows) are mixed – determines the time scales over which different scal-ings manifest themselves. More specifically, the local scaling behaviors of the aggregate traffic depend on how the degree of aggregation
and flow burst1g of each flow type evolve as a function of time scale and which type of flows dominates at a given time scale. As a result, diverse traffic compositions can produce rich and complex local scalings at small time scales.
103 104 105 106 107 0 5 10 15 20 −−−> LARGE SMALL <−−−
flow size (bytes)
% flows with bytes > x
OC12−tier2−dom OC12−corp−dom
Figure 6: CCDF of flow size for a minute time interval.
4
Traffic Composition and Small-Time
Scalings
Based on the theoretical insights obtained from the previous section, we perform a detailed analysis of traffic composition of the traces and investigate the impact of traffic composition on their small-time scaling behaviors. Due to space limita-tion, in this presentation we focus primarily on the two packet traces, OC12-tier2-dom from a peering link and OC12-corp-dom from a corporate link, and use them as examples to il-lustrate our findings. From Figures 2 recall that traffic on the OC12-tier2-dom link has nearly uncorrelated scaling in small time scales 1ms - 100ms, while traffic on the OC12-corp-dom link shows much stronger correlation in the same range of time scales. We are particularly interested in uncovering the causes for this difference in small-time scaling behaviors.
(a) 10 0 101 102 103 4 5 6 7 8 9 H=0.56356 time scale (ms) log2(Wavelet Energy) H=0.73178 OC12−tier2−dom large OC12−corp−dom large (b) 10 0 101 102 103 −3 −2 −1 0 1 2 3 4 5 H=0.54245 time scale (ms) log2(Wavelet Energy) H=0.57208 OC12−tier2−dom small OC12−corp−dom small
Figure 7: Energy plot of (a) large and (b) small flows for OC12-tier2-dom and OC12-corp-dom.
In our analysis, we define a flow as a source-destination /24 network prefix pair (i.e., packets from sources with the same first 24-bit IP addresses and going to destinations with the same first 24-bit IP addresses will be classified as from the same flow). There are several reasons for this definition of flows. First we are interested in the impact of flows with similar characteristics on the aggregate traffic. All packets in /24 flows are likely to take similar (if not the same) network paths before they reach the observed link, and thus experience similar network effects (e.g., RTT, bottleneck links, etc.). The standard 5-tuple flows (e.g., TCP flows) are too small relative to the aggregate traffic on a backbone link to have a significant effect in themselves. They are also too numerous and harder to classify precisely using the collected traces. Furthermore, since /24 network prefixes are the smallest to be honored by the tier-1 ISP, understanding the impact of /24 flows on the aggregate traffic is more meaningful from the stand point of network provisioning and traffic engineering in the backbone network. (a) 10 0 101 102 103 100 101 102 103 104 time scale j (ms) E[N j ] OC12−corp−dom large OC12−tier2−dom large OC12−corp−dom small OC12−tier2−dom small (b) 10 0 101 102 101 102 103 104 105 time scale j (ms) E[Y j ] OC12−corp−dom large OC12−tier2−dom large OC12−corp−dom small OC12−tier2−dom small
Figure 8: (a)uyvw NQP
R vs. H ! and (b)uyvw NSP291 R vs. H ! . To analyze traffic composition, we zoom into -minute seg-ments of the two traces to analyze the characteristics of their flows in detail. A -minute segment contains enough infor-mation to give good estimates of sub-second time scale (1ms-100ms) statistics, while being small enough to enable a fea-sible detailed analysis of local small-time scaling behaviors. Further we have observed little deviation of the energy plot of a -minute segment of backbone traffic from the energy plot obtained from a -hour segment.
Figure 6 shows the complementary cumulative
distribu-(a) 0 1 2 3 4 0 0.2 0.4 0.6 0.8 1 lag autocovariance of N j OC12−corp−dom small OC12−tier2−dom small (b) 0 1 2 3 4 0 0.2 0.4 0.6 0.8 1 lag autocovariance of N j OC12−corp−dom large OC12−tier2−dom large Figure 9: Autocovariance of
$#% for both small and large flows at the ms time scale.
tion function (CCDF) of flow sizes (in bytes) in a given 1-minute time span for both OC12-tier2-dom and OC12-corp-dom. Note that the x-axis is given in logscale, and the distri-bution is heavy-tailed. Clearly on both links, a great majority of flows (more than 80%) send fewer than 10KB of data in a time span of minute, hereafter referred to as small flows. At the other extreme, OC12-corp-dom has about 1% of the flows sending at least 1MB in 1 minute, while OC12-tier2-dom has about 0.5% of the flows sending at least 1MB in 1 minute. These flows will be referred to as large flows.12 The
large flows send 58% of the total bytes of OC12-corp-dom and 35% in the case of OC12-tier2-dom (see Figure 11(b) for plots labeled “minute 1”).
To illustrate the impact of aggregation of flows with “simi-lar” traffic characteristics on small-time scaling behaviors, we extract small and large flows from the traces and study them in isolation. Figure 7 shows the energy plots for both the small and large flows of OC12-tier2-dom and OC12-corp-dom. In the case of OC12-tier2-dom, both the aggregates of small and large flows have nearly uncorrelated scalings in 1ms – 100ms time scales, while in the case of OC12-corp-dom, the aggre-gate of small flows has a nearly correlated scaling (_}OVKXW ) in 1ms – 100ms, while the aggregate of large flows shows a strong correlated scaling in 1ms – 100ms with } VK , close to that of the total aggregate of the OC12-corp-dom trace at these time scales (see Figure 2(d)). These results suggest
12Although our choice of large and small flows is somewhat arbitrary, it
100 101 102 12 14 16 18 20 22 24 26 H=0.54752 time (ms) log2(Wavelet Energy) H=0.58737 H=0.63385 H=0.69106 % large = 0 % large flows=5 % large flows=50 % large flows=100
Figure 10: Energy plots for various mixes of large and small OC12-corp-dom flows.
that the strong correlated scaling we observed in the OC12-corp-dom packet trace is primarily due to the large flows, and the difference in the small-time scaling behaviors of OC12-corp-dom and OC12-tier2-dom links lies in the vastly different characteristics of the large flows in the two traces.
To understand how the traffic characteristics influence the small-time scaling behaviors, it is instructive to look at how the degree of aggregation
and flow burst size 1g (see Section 3.3) change as a function of time scale. For each time scale ! , we compute
$#% , the number of different ac-tive flows in time unit"
2#% . We approximate 1 $#% using 9 1 2#% 1 o<úGq $#% `
$#% , where recall that1 oyúGq
2#% is the total number of bytes observed in"
$#% . Hence
9
1 2#% represents the average burst size sent by each flow in "
2#% . Averaging $#% and 9 1 2#% over all time units" 2#% , we obtainNQP R and NQP:91V
R. Figures 8(a) and (b) plot, respectively,uyvw NQP
R and upvrw NQP:91V R against H ! for both the large and small flows of tier2-dom and OC12-corp-dom (the bars on the plots indicate the mean; one stan-dard deviation). These plots visually depict the different traffic characteristics of these flows.
First, notice that the plots for the small flows of OC12-tier2-dom and OC12-corp-OC12-tier2-dom are almost identical: uyvw
NQP
R
scales linearly with H ! with the same slope (} 1) from 1ms to 100ms, whileupvrw NQP
9
1V
R stays almost constant with
H ! (with the small flows of OC12-corp-dom have a slight positive slope) from 1ms to 100ms. These behaviors are consistent with the theoretical example of small flows with Poisson arrivals in Section 3.3. Not surprisingly, the small flows show a nearly uncorrelated scaling. To demonstrate the flow burst arrivals of the small flows indeed are independent in the small time scales, Figure 9(a) shows the (normalized) autocovariance function of
2#% at the 1ms time scale for the small flows. We see that the autocovariance drops to nearly zero with a lag of , strongly suggesting that the burst arrivals of small flows are indeed in-dependent at 1ms time scale.
Now consider the large flows. We see that in the case of OC12-tier2-dom, upvrw NQP R and upvrw NQP:91V R scale almost
Poisson-like in the time scales of 1ms – 10 ms: uyvw NQP
R in-creases linearly with H ! with a slope } , whileuyvw NQP<91V
R
remains almost flat. Beyond 10 ms time scales, upvrw NQP
R
starts to gradually flatten out, and upvrw NQP1V9
R increases more rapidly. These behaviors indicate that in the range of 1ms -10ms, the bursts of OC12-tier2-dom large flows arrive almost independently – this is further supported by the autocovariance of
$#% at 1 ms time scale, which drops to zero with a lag of 1 (Figure 9(b)). Above 10 ms time scale, the probability of encountering two or more bursts from the same large flow in-creases, thus uyvw NQP:91V
R increases while upvrw NQP
R decreases. Above 100 ms time scale, almost the same set of large flows are found sending bursts in almost every time unit. This is not surprising, as each of the large flows sends at least 1MB in the 1-minute time span.
In contrast, the scaling behaviors of upvrw NQP
R and uyvw NQP91V
R for OC12-corp-dom are quite different from those of OC12-tier2-dom (see Figure 8): uyvw NQP
R scales with H ! almost linearly with a consistent slope (ofVK ) up until 100 ms or larger, then it starts to flatten out; over the same range of time scales,upvrw NQP19
R also scales with
H ! almost linearly with a slope of VK , then it starts to increase more rapidly. The scaling behaviors ofuyvw NQP
R and uyvw NSP19
R in the time scale range of 1ms – 100ms are indicative of the correlation of burst arrivals in OC12-corp-dom: many bursts from neighboring time units are from the same large flows, causingupvrw NQP
R
to grow at a slower rate (than that of OC12-tier2-dom) and uyvw NQP1V9
R at a faster pace (than that of OC12-tier2-dom). This correlation is more evident when we look at the autocovariance of
of OC12-corp-dom at 1ms time scale in Figure 9(b) – the autocovariance is still close to 0.1 with a lag of 4. Our analysis reveals that although both the large flows of OC12-corp-dom and OC12-tier2-dom consist of flows sending at least 1MB during the 1-minute time span, they have vastly different traf-fic characteristics. At relatively coarser time scales (e.g., above 100 ms) large flows from both traces appear to send big bursts frequently, and are not much different. However, when we zoom into the smaller time scales (1ms – 10ms), their differ-ence shows up: big bursts from OC12-corp-dom large flows break up into correlated smaller chunks, while big bursts from OC12-tier2-dom large flows break up into independent small pieces, resulting in very different scaling behaviors in these time scales.
To demonstrate how different traffic composition affects the small-time scalings of aggregate traffic, we mix the small flows of OC12-corp-dom with various portions of large flows from OC12-corp-dom. The energy plots for the resulting traffic ag-gregates are shown in Figure 10. From the figure we see that as we increase the portion of large flows in the traffic aggre-gate, it first affects the slope of the energy plot in the right end (coarser time scales) of 1 ms – 100 ms time scale range, caus-ing it to rise; it gradually extends its impact toward the left (finer time scales), eventually causing the slope to rise over almost entire 1 ms – 100 ms time scale range. This result vali-dates our theory in Section 3.3: increasing the portion of large flows with correlated bursts, their impact can be felt at finer and finer time scales. Diverse traffic compositions helps pro-duce the rich and complex small-time scalings we observed across different links/packet traces.
(a) 10 3 104 105 106 107 0 5 10 15 20
flow size (bytes)
% flows with bytes > x
OC12−tier2−dom minute 1 OC12−tier2−dom minute 2 OC12−corp−dom minute 1 OC12−corp−dom minute 2 (b) 10 3 104 105 106 107 0 20 40 60 80 100
x=flow size (bytes)
% bytes from flows with bytes > x
−−>LARGE
OC12−tier2−dom minute 1 OC12−tier2−dom minute 2 OC12−corp−dom minute 1 OC12−corp−dom minute 2
Figure 11: CCDF of (a) flow size and (b) bytes contributed by flows.
that traffic composition on a link is fairly stable, and is in-variant of link utilization and time. Figure 11(a) shows (for both OC12-corp-dom and OC12-tier2-dom) the flow size dis-tributions over two different 1-minute time spans several hours apart. Figure 11(b) shows the corresponding byte contribution. For the OC12-corp-dom trace the utilization is W>= higher in the second interval than in the first while in the OC12-tier2-dom trace the utilization stays approximately the same. It is clear that there is little variation in these distributions. Fur-thermore, the correlation structures in large flow bursts for both OC12-tier2-dom and OC12-corp-dom also change very little over the time. This is evident from Figure 12, where the scaling behaviors ofuyvw NQP
R and upvrw NQP19
R over the second 1-minute time span are shown: comparing with Figure 8, there is hardly any difference. Given the diversity of the Internet, this consistent traffic composition of a link over time is indeed surprising!
We postulate two hypotheses as plausible causes for this apparent consistent traffic composition on a link over time. Hypothesis I: that the proportionality of flow size distribution stays relatively consistent over time is due to the random na-ture of Internet access and transfer, and the universal heavy-tailed distribution of object sizes. Hence by laws of proba-bility, increased link utilization only changes the total volume of traffic on a link, not the relative proportions of flows with differing sizes. Hypothesis II: that the burst correlation struc-ture (of large flows) also stays relatively consistent over time is due to the invariant network factors such as bottleneck links and propagation delay lurking beyond both sides of an
Inter-(a) 10 0 101 102 103 100 101 102 103 104 time scale j (ms) E[N j ] OC12−corp−dom large OC12−tier2−dom large OC12−corp−dom small OC12−tier2−dom small (b) 10 0 101 102 101 102 103 104 105 time scale j (ms) E[Y j ] OC12−corp−dom large OC12−tier2−dom large OC12−corp−dom small OC12−tier2−dom small
Figure 12: (a)uyvw NSP R vs. H ! and (b)uyvw NQP 9 1 R vs. H ! .
net backbone link as well as the effect of TCP window-based feedback control. These factors together affect how traffic is “perturbed” in the generation and transition processes before they reach the backbone network. Traffic carried on a peering link may come from many diverse access networks with differ-ent bottleneck links, and traverse more hops, thus bursts from large flows are likely broken down into smaller, less correlated pieces; while traffic carried on a corporate link may come from relatively high-speed links, and traverse fewer hops. Hence bursts from large flows have a less chance to be perturbed, and thus correlated when they reach the backbone network. A study of bit rates of TCP connections comprising the large /24 flows (over a 1 hour period) lends credibility to this hypothesis. We divide the total bytes by the duration to obtain the bit rate for each TCP connection. Packets with the same source and destination IP addresses and port numbers and with interarrival time greater thanr seconds are assumed to belong to differ-ent TCP connections. Figure 13 clearly demonstrates that TCP connections in OC12-corp-dom tend to have higher data rates than those in OC12-tier2-dom. The simulation-based study of TCP effect on small-time scaling behaviors in [9] also lends some support to this hypothesis. More work is needed to fur-ther verify and validate these hypotheses.
5
Concluding Discussion
Using packet traces collected from OC3 to OC12 links on a tier-1 ISP we discovered two striking scaling phenomena of Internet backbone traffic at (sub-second) small-time scales:
0 0.2 0.4 0.6 0.8 1 0 20 40 60 80 100 x=rate (Mbps)
% bytes from TCP conn. with rate > x
OC12−tier2−dom OC12−corp−dom
Figure 13: Percentage of bytes vs. TCP connection bit rate for large /24 flows.
1) a majority of the collected traces exhibit uncorrelated or nearly uncorrelated scaling behaviors with a Hurst parameter less than 0.6, often close to 0.5, at small time scales ( 1ms -100ms); 2) the scaling behaviors at small time scales are link-dependent, but stay fairly invariant over changing utilization and time. Motivated by these observations, we developed an-alytical models to understand the multi-time-scale behaviors of network traffic. With theoretical examples we illustrated how traffic composition – aggregation of flows with different characteristics – can affect the small-time scaling behaviors of network traffic. Guided by the theoretical insights, we then performed a detailed analysis of traffic composition and inves-tigated its impact on small-time scaling behaviors. Using the traces, we also demonstrated that traffic composition on the observed Internet backbone links stays surprisingly consistent over time. Two hypotheses were put forth as plausible causes for the apparent consistent traffic composition on a link over time. Further work is still needed to verify and validate these hypotheses.
Our observations and results have significant implications in networking modeling, service provisioning and traffic en-gineering. As pointed out in the introduction, the discovery of uncorrelated small-time scaling behaviors on many Internet backbone links can lead to simpler network models for ana-lyzing network performance at small time scales [19, 11]. The link-dependent small-time scalings also suggest that we need to take traffic composition on a link into consideration, when making relevant service provisioning and traffic engineering decisions. For example, care must be taken when determining how traffic should be aggregated and routed over a backbone network to avoid unnecessarily increasing burst correlation.
Our discovery that small-time scalings are determined by traffic composition on a link also raises many intriguing issues regarding the impact of Internet evolution on traffic behaviors. On the one hand, since the burst correlation of large flows be-comes weaker as time scales become smaller, with increasing speed of Internet backbone links, we would expect traffic on these links are more likely to appear independent at those small time scales that matter to queuing [19]. On the other hand, as broadband access becomes more widely deployed, large files
and objects can be transmitted faster into the Internet, with more correlated bursts. In addition, the changing nature of ap-plications can also alter the picture completely. For example, wide-spread use of distributed file sharing (e.g., exchange of CD-quality music or DVD videos) can increase the proportion of large files transmitted over the Internet, thereby affecting the small-time scalings of the Internet backbone traffic. Fur-thermore, increased use of protocols other than TCP can also effect a change in the small-time scaling behaviors of the Inter-net backbone traffic. These are important questions awaiting to be explored.
References
[1] D. Veitch’s home page. http://www.emulab.ee.mu.oz.au/?
dar-ryl/.
[2] P. Abry and D. Veitch. Wavelet analysis of long-range-dependent traffic. IEEE Trans. on Information Theory, 44(1):2– 15, January 1998.
[3] F. Baccelli and D. Hong. The AIMD model for TCP sessions sharing a common router. to appear in INFOCOM 2002. [4] J. Cao, W. S. Cleveland, D. Lin, and D. X. Sun. The effect of
statistical multiplexing on internet packet traffic: Theory and empirical study. Bell Labs Technical Report, 2001.
[5] J. Cao, W. S. Cleveland, D. Lin, and D. X. Sun. On the non-stationarity of Internet traffic. Proc. ACM SIGMETRICS ‘01, pages 102–112, 2001.
[6] M. Crovella and A. Bestavros. Self-similarity in World Wide Web traffic. Evidence and possible causes. IEEE/ACM Trans. on Networking, 5:835–846, December 1997.
[7] A. Feldmann, A. C. Gilbert, and W. Willinger. Data networks as cascades: Investigating the multifractal nature of Internet WAN traffic. Proc. ACM/SIGCOMM 98, 28:42–55, 1998.
[8] A. Feldmann, A. C. Gilbert, W. Willinger, and T. Kurtz. The changing nature of network traffic: Scaling phenomena. Com-puter Communication Review, 28:5–19, 1998.
[9] A. Feldmann, A. C. Gilbert P. Huang, and W. Willinger. Dy-namics of IP traffic: A study of the role of variability and the impact of control. Proc. ACM/SIGCOMM, 1999.
[10] D. R. Figueiredo, B. Liu, V. Misra, and D. Towsley. On the auto-correlation structure of TCP traffic. UMass CMPSCI Technical Report TR 00-55.
[11] C. Fraleigh, S. Moon, C. Diot, B. Lyles, and F. Tobagi. Packet-level traffic measurements from a tier-1 IP backbone. Sprint ATL Technical Report TR01-ATL-110101, November 2001. [12] R. J. Gibbens and Y. C. Teh. Critical time and space scales for
statistical multiplexing. Proc. of the 16th International Teletraf-fic Congress – ITC 16, pages 87–96, 1999.
[13] A. C. Gilbert, W. Willinger, and A. Feldmann. Scaling analysis of conservative cascades, with applications to network traffic. IEEE Trans. Info. Theory, Special issue on multiscale statistical signal analysis and its applications, 45, April 1999.
[14] M. Grossglauser and J-C. Bolot. On the relevance of long-range dependence in network traffic. Computer-Communication-Review, 26(4):15–24, October 1996.
[15] L.M. Kaplan and C.-C.J. Kuo. Fractal estimation from noisy data via discrete fractional Gaussian noise (DFGN) and the Haar basis. IEEE Trans. Signal Proc., 41(12):3554–3562, Dec. 1993. [16] W. Leland, M. Taqqu, W. Willinger, and D. Wilson. On the self-similar nature of Ethernet traffic (extended version). IEEE/ACM Trans. Networking, pages 1–15, 1994.
[17] J. L´evy V´ehel and R. Riedi. Fractional Brownian motion and data traffic modeling: The other end of the spectrum. Fractals in Engineering, pages 185–202, Springer 1997.
[18] P. Mannersalo and I. Norros. Multifractal analysis of real ATM traffic: a first look. COST257TD, 1997.
[19] A. L. Neidhardt and J. L. Wang. The concept of relevant time scales and its application to queuing analysis of self-similar traf-fic. In Proc. SIGMETRICS ’98/PERFORMANCE ’98, pages 222–232, 1998.
[20] K. Park and W. Willinger, editors. Self-Similar Network Traffic and Performance Evaluation. Wiley Interscience, 2000. [21] V. Paxson and S. Floyd. Wide-area traffic: The failure of
Pois-son Modeling. IEEE/ACM Trans. on Networking, 3:226–244, 1995.
[22] R. Riedi, M. S. Crouse, V. Ribeiro, and R. G. Baraniuk. A mul-tifractal wavelet model with application to TCP network traffic. IEEE Trans. Info. Theory, Special issue on multiscale statistical signal analysis and its applications, 45:992–1018, April 1999. [23] R. H. Riedi. An introduction to multifractals.
http://www.ece.rice.edu/? riedi/.
[24] A. Sridharan, S. Bhattacharyya, C. Diot, R. Guerin, J. Jetcheva, and N. Taft. The impact of aggregation on the performance of traffic aware routing. Proc. of the 17th International Teletraffic Congress Salvador do Bahia, Brazil, December 2001.
