QoS-Aware Routing Schemes
Based on Hierarchical Load-Balancing
for Integrated Services Packet Networks
C. Casetti, R. Lo Cigno, M. Mellia
Dipartimento di Elettronica, Politecnico di Torino, Italy
Abstract—This paper presents a load balancing method to improve
net-work utilization when static routing algorithms are employed. Static rout-ing algorithms can generally be reduced to a path assignment problem with the aim of minimizing a cost function: i.e., maximizing network revenues. Unfortunately the problem, when multiple constraints such as different QoS traffic classes are present, is computationally intractable. Load bal-ancing methods are heuristics that allow the identification of local minima of the cost function that hopefully are close to the real optimum.
The load balancing method we propose is based only on static infor-mation about the average traffic load of each traffic relation: a piece of information which can easily be attained in telecommunication net-works. The path-selection strategy is defined in terms of Set Theory and its performance is evaluated via simulation by comparison against standard Minimum-Hop path selections. Simulation results show that low priority traffic benefits significantly from the more balanced load obtained with our solution.
Integrated-Services Packet Networks (ISPN) are emerging as one of the new challenges in networking research. ISPN is not yet a completely defined term: the assumption in this work is that we deal with a packet-switched network supporting two or more traffic classes differentiated by their required Quality of Service (QoS). Next generation Internet will be (hopefully) an ISPN, but Asynchronous Transfer Mode (ATM) networks are ISPNs as well, with the additional constraint of handling only fixed-size packets named cells.
Routing strategies play an important role in ISPNs, since it is expected that they have wide-area coverage and they are highly meshed. QoS-aware routing takes into account also the charac-teristics of the connection that is being routed through the net-work, in order to exploit network resources to their full extent.
Traditionally, routing strategies have been divided into static and dynamic ones. Dynamic routing strategies , ,  are very appealing for their capacity to deal with unpredictable changes in traffic pattern; however, in very large networks, they suffer from the stale information problem: routing decisions are taken on estimates of the network condition that may be either wrong or out-of-date and this may induce inefficient network exploitation and, ultimately, even network oscillation and col-lapse . Static routing strategies are generally not based on the network load, but on topological properties of the network, such as the number of crossed nodes, the length and the capacity of the links, and so forth. A good example of different static
rout-This work was supported in part by a research contract between CSELT and Politecnico di Torino and in part by the Italian Government through the Min-istry for University and Research (MURST) and through the National Research Council (CNR).
ing strategies can be found in . In our work we focus on static routing strategies, leaving for future research a comparison with dynamic ones.
The performance of static routing algorithms can be enhanced making use of some statistical knowledge of the traffic, such as the long-term average of the traffic offered to the network from nodeito nodej. In any case, the aim of any static rout-ing algorithm is the identification of the set of routes, for any source destination pair(s;d), that optimize (or at least enhance) the network performance. The identification of the optimal set of routes is generally a NP-Complete problem when multiple constraints are present , , hence approximations or heuris-tics are needed to obtain routing algorithms with good perfor-mance in ISPN networks, especially if multiple service classes are present.
Our work introduces a load-balancing heuristics that can be hierarchically applied to different traffic classes for the defini-tion of the route sets to be used by alternate-path (primary-path only can be seen as a special case) routing schemes. Alternate routes are sequentially probed as the CAC (Connection Admis-sion Control) function denies allocation on the chosen path. We point out that routing and CAC functions are not tightly coupled, so that, for instance, a static routing algorithm can be coupled with either a static CAC (e.g., peak or equivalent bandwidth) or with a dynamic CAC based on traffic measures.
The remaining part of the paper is organized as follows: Sec-tion II gives a formal definiSec-tion of the problem, in order to allow a general and theoretical analysis; Section III describes the paths selection algorithms using the formalism defined in Section II; Section IV describes the scenario chosen for results comparison, and Section V presents the numerical results obtained by simu-lating the above scenario; Section VI, finally, closes the paper with hints on the prosecution of the work.
II. SETTHEORYPROBLEMFORMULATION AND
Let us consider a network N( ;), where is the set of nodes andis the set of links, and letM=jj jjandL=jjjj. We denote by the set of all possible
1 loop-free paths, and
(s;d) the set of all possible
1 loop-free paths between the
source node s and the destination node d. Obviously = [
be a generic path, and i
(s;d)be the i-th path from sourcesto destinationd. Let also
be the set of all possible paths sharing the linkl, and
paths fromstodsharing the linkl.
Let()be a generic cost function associated to the path, related to the network topology; for example, () can be a function of the physical length of, or of the number of hops n(), or a function of both. Given two paths
(s;d) connecting the same sourcesand destinationd, they are consid-ered-equivalent if(
). We shall also denote by E
(s;d)the number of-equivalent paths of costfromsto d.
Given a networkN( ;), where a static routing algorithm is employed, let us consider the problem of how to order the set of all possible paths to better exploit the network capacity.
(s;d) 6=1, the sorting algorithm has to choose between two or more equivalent paths: a supplementary metric has to be defined. To this purpose, let us define some new param-eters. Suppose that we know the traffic matrixT =[t
s;d ], where t
s;dis the probability that a source
stries to open a call toward a destinationd. LetB
sbe the offered load of source
s. We shall denote by(s;d)another cost function, derived from the band-width requirement of the connection originated at the sources and routed to the destinationd; we then have(s;d)=t
s. This is called “Static Offered Load”, representing the load that a connection offers to the network, evaluated upon the static pa-rameters of the traffic matrix.
With the above notation, we can compute a “Static Link Cost” function (l)for each linklof the network, defined as follows:
(l)= 1 C l X s;dj(s;d)2 l (s;d) whereC
lis the available bandwidth on the link
l. We deliber-ately did not specify further the nature ofC
l, since this quantity can be used to keep into account higher-priority traffic that effec-tively shrinks the link’s capacity as far as the the lower-priority traffic is concerned (this is the case of best-effort vs. guaran-teed traffic). (l)represents the normalized load that the linkl would carry if we took into account the traffic resulting from all source-destination pairs that route calls over the set of paths
l . Again, the term “static” means that this cost is evaluated on the knowledge of the traffic matrix only, and it is not a real measure of the carried load. However, it should be pointed out that (l) is updated throught the route computation phase as the static-load algorithm keeps adding new paths. Note also that (l)can be greater than one.
By maximizing over all links of a path , we can derive a “Static Cost” ()for that path:
We can thus choose between two (or more)-equivalent paths
(s;d), sorting them according to the cost .
III. ALGORITHM DESCRIPTION
Because the target of the load balancing algorithms is to or-der a set of path, we chose to describe these algorithms in a set theory manner, where ordering functions can be applied.
or a subset of them, as limited by the route selection algorithm
For the sake of simplicity, we first define a set of simple pro-cedures that are later on used to obtain the algorithm we are interested in.
Procedure Hop First: HF(A; ; c). Given a set of pathsA, find a subsetB
ccontaining all the paths with topological cost(2B
c )=c. 1. for all2Ado 2. if()=cthen 3. A=A fg; B c =B c [fg; 4. returnB c;
Procedure Min Hop: MH(A;).
Given a set of pathsA, find the minimum cost c
min for each source-destination pair, and return the subsetB containing all the minimum cost paths.
1. fors=1:::M, ford=1:::M; s6=ddo 2. A (s;d) =A\ (s;d) ; c min = min 2A (s;d) (()); 3. B=B[HF(A (s;d) ;;c min ) 4. returnB;
Procedure Equivalent Path: EP(A; ; e). Given a set of pathsA, find a subsetB
econtaining all the paths with a number of-equivalent paths equal toe.
1. fors=1:::M, ford=1:::M; s6=ddo 2. A (s;d) =A\ (s;d) 3. ifE (s;d)=ethen 4. B e =B e [A (s;d); 5. returnB e
The following procedures rely on vector [L], which is the accumulator for the Static Link Cost. [L]is initialized to zero and is updated whenever the SCU procedure (see below) is exe-cuted.
Procedure Static Cost Update: SCU().
Given a path, update the Static Link Cost (l)for its links. 1. letsanddbe the source and destination nodes linked by; 2. for all linkl2do
3. (l)= (l)+
Procedure Static Load Sorting: SLS(A; ).
Given an unordered path setA, find an ordered subset of paths with the minimum Static Cost for each pair(s;d). Note that this is the only procedure that provides an ordering of the input set of paths. 1. B=;; k=0; 2. fors=1:::M; d=1:::M; s6=ddo 3. A (s;d) =A\ (s;d); 4. find2A (s;d)such that ()=min i2A (s;d) ( ( i )); 2 5. k =; 6. SCU( k); 7. A=A fg; B=B[f k g; k=k+1; 8. returnB;
A. Hierarchical Load Balancing Algorithm
When multiple traffic classes are present, with different QoS requirements as well as different priorities, the path assignment
If there is more than one minimum cost path, one can recursively choose between them again recalculating ()on all the linksl2except for the one
procedure must be aware that connections will be accepted by CAC functions following different criteria. CAC functions and packet schedulers, generally assume hard class separation with acceptance preemption, i.e., regardless of the network being al-ready saturated by low priority calls, a higher priority one will be accepted, and its packets will be served with priority, that is in non FIFO order, once they are in the network. As fas as CAC and scheduling are concerned, a hierarchical path selection tech-nique based on load balancing should be an appropriate solution to help low-priority traffic to avoid routes heavily used by higher priority traffic.
(s;d)be the set of available paths in the net-workN. By dividing it into different subsets and ordering them according to the Static Cost function , we obtain a path as-signment from the sources s to the destinationsd, achieving, hopefully, a better exploitation of the network capacity than by simple Minimum-Hop ordering. Notice that Minimum-Hop or-dering yields exactly the same path selection, regardless of it being run on all traffic classes together or hierarchically class by class.
We describe a single sorting criteria, although more heuristics were studied in , and obviously many more can be defined. The algorithm works in cycles: in every cycle, it assign one and only one path per source-destination pair, if it exists, starting from the paths with the minimum number of hops.
For the sake of clarity the algorithm is described for a sin-gle traffic class, avoiding an additional loop on traffic classes and the consequent complication in the notation. Hierarchically moving from one class to the next, the static link cost (l)is re-set to the static load (l)of higher-QoS-classes sources whose primary path
. We chose not to take into ac-count the load due to secondary paths from higher classes, since secondary paths are less likely to carry newly activated calls than primary paths.
B. Algorithm: Balanced Primary First BPF()
The following algorithm assigns at every loop a single path 2(if it exists) from the sourcesto the destinationd, resort-ing the remainresort-ing path set, in order to find the best path to as-sign in the next cycle. This is obtained by applying, in sequence, the Min Hop algorithm to set aside all the shortest paths for all source-destination pairs, then the Equivalent Path algorithm to identify all the subsets with-equivalent paths. Each subset is processed by the Hop First algorithm which, in turn, yields a sub-subset of shortest paths. Eventually, the Static Load Sort-ing algorithm is used over all sub-subsets identified above to obtain the final order, as shown in Fig. 1. The result is a re-ordered paths set ~
where, for each traffic relation(s;d), there is an ordered paths sub-set ~
(s;d)in which the first one is termed primary and all the others are termed secondary. The routing function probes all the paths in order and the connection is re-fused only if the CAC functions can not allocate it on any paths of the sub-set ~
(s;d), as is done in the Uncontrolled Alternate Routing . 1. while6=;do 2. J= MH(;); 3. e=0; 4. whileJ 6=;do 5. e=e+1; 6. J e= EP( J;;e); 7. J =J J e; 8. e max =e; 9. fore=1:::e maxdo 10. c=1; 11. whileJ e 6=;do 12. J e;c= HF (J e ;;c); 13. ~ J e;c= SLS( J e;c ; ); 14. ~ = ~ [ ~ J e;c ; = ~ J e;c ; c=c+1; 15. return ~ ;
Static Load Sorting
J1,1 Je,1 Je,2 Π J1,2 Jh,k ~ ~ ~ ~ ~ Π Equivalent Path Hop First Je Je,1 Je,2 Je,c J1
Min Hop J
Fig. 1. Algorithm Balanced Primary First (BPF)
The effectiveness of the algorithm presented in the previous sections was studied by means of a call-level simulation tool for high-speed networks, called ANCLES , assuming two traffic classes: a high priority, guaranteed-QoS class and a low priority, best-effort class. The network topology of choice is shown in Fig. 2; it combines a rather tight mesh with hundreds of loop-free paths and a localized bottleneck. The link speed, which is the same for all links, is equal to 500 Mb/s. A mix of constant- and variable-bit rate sources, located in every node, are assumed to generate traffic in the highest QoS class: the constant-bit rate sources try to open calls atB
=2Mb/s rate toward each other, (call holding times are determined by i.i.d. exponential random variables averaging at 500 seconds); the variable-bit rate sources activate calls atB
=10Mb/s which, once activated and admitted to the network, hold for an exponen-tially distributed period, with average 1000 seconds; variable-bit rate calls exhibit a burstiness degree (peak-to-mean rate ratio) equal to 5, thus resulting in an average rateB
=2Mb/s. The call admission control scheme to which the above calls are subject is rather simple; for each link, new calls are accepted if: X C1 B p + X C2 B m C (1) whereC 1 and C
2 are the sets of constant- and variable-bit rate calls, respectively,Cis the link capacity, andis a protection coefficient (1) that can be set so as to avoid that the whole
N1 N2 N3 N4 N5 N6 N9 N7 N8 N10 N11 N12
Fig. 2. The examined topology
link capacity is used up.B
mindicates a parameter characteriz-ing connections with a cell rate value midway betweenB
pand B a, according to a factor ; that is: B m =B a +(B p B a ) (2)
A discussion of the CAC scheme just described and its relation to some equivalent bandwidth expressions can be found in . In our simulations, we chose = 0:9and = 0:4(and, for some results,=1, yielding peak allocation).
For what concerns best-effort, lower-priority users, nodes N1, N8 and N12 are requested to activate connections with a peak rate B
= 5Mb=s (but we also report results with B
=10Mb=s) and average holding time equal to 60 seconds. Each best-effort source equally directs its traffic toward nodes N4, N6 and N10. The actual bandwidth, possibly smaller than the peak, exploited by users of this class is determined by a max-min fairness allocation scheme that, for each call, accounts for the number of calls on the bottleneck link and for the residual capacity available to the best-effort class1. No CAC is
imple-mented for best-effort calls.
The network traffic was split among guaranteed (40% of the overall traffic) and best-effort (60%) calls. Also, calls are routed according to the Hierarchical Load Balancing Algorithm de-tailed in Subsection III-A; note that, since there is no CAC for best-effort traffic, all calls in this class are routed over their re-spective primary paths.
So as to provide an insight into the complexity of the load bal-ancing algorithms we are proposing, we estimated that sorting all possible paths of the network in Figure 2 (approx. 40,000 paths) required just 2 seconds on a Pentium II-266MHz work-station.
Simulations were run to obtain an estimate of how much bandwidth can be spared when the proposed load-balancing al-gorithm is used, with respect to a situation where the same
Of course, this is only an estimate of what the best-effort connections get, depending on the actual policy used by the network to allocate resources to connections; however regardless of the policies and the performance they attain, the relative merits of routing schemes should not be affected.
1e-05 1e-04 1e-03 1e-02 1e-01 1e+00 1.5 1.6 1.7 1.8 1.9 2
Call Blocking Probability
Load no lb
γ = 0.4
γ = 1
Fig. 3. Call blocking probabilities as a function of the overall guaranteed-class network load in Gb/s
ing and CAC algorithms were deployed, without preventively balancing the load (i.e. sorting the set of the available paths only by topological cost). Figure 3 shows the call blocking prob-abilities for guaranteed traffic which were estimated as a func-tion of the overall guaranteed-class load offered to the network (in Gb/s), when different values of, ranging from a peak-rate characterization of variable-bit rate sources ( = 1) to a more relaxed one (=0:4). Solid lines refer to curves obtained with-out load balancing, while dashed lines refer to the load balancing algorithm.
It can be observed that balancing guaranteed traffic does not improve calls’ chances of being admitted to the network, nor does it lead to higher blocking probabilities.
If we now turn our attention onto the lower QoS traffic class, we can try and gauge how the combination of load balancing applied to both traffic classes can, on the one hand, rationalize the distribution of guaranteed traffic, thus freeing up resources, and, on the other, efficiently allocate the leftover bandwidth to best-effort traffic. To this purpose, we defined a users’ “reward” as the ratio between the peak bandwidth that a best-effort call requests and the actual average bandwidth that the call has been able to exploit throughout its life. Figures 4 and 5 report such rewards, averaged over all calls that were activated during the simulation, for different values of the admission parameter. It can be seen that a reward increment as high as 5% can be achieved when the load balancing algorithm is applied.
A closer inspection of the results allows us to link the rewards to the actual paths choice as determined by the load balancing algorithm. Table I details the effectiveness of the load balancing algorithm at limiting the number of times non-primary (and thus more resources consuming) paths are checked for admission of the higher-QoS calls before a suitable path is found and the call is routed over it. Expressed through percentages, these quanti-ties also provide us with a general indication of the call setup time and of how it can be considerably shortened when using a load-balancing algorithm.
0.6 0.65 0.7 0.75 0.8 0.85 0.9 1.5 1.6 1.7 1.8 1.9 2 Users reward Load no lb with lb 5 Mb/s 10 Mb/s
Fig. 4. Average best-effort users’ reward as a function of guaranteed-class net-work load in Gb/s -= 0.4 0.6 0.65 0.7 0.75 0.8 0.85 0.9 1.5 1.6 1.7 1.8 1.9 2 Users reward Load no lb with lb 5 Mb/s 10 Mb/s
Fig. 5. Average best-effort users’ reward as a function of guaranteed-class net-work load in Gb/s -= 1
Path Load = 1.5 Load = 1.6
# Unbal. Bal. Unbal. Bal.
1 98.65 99.98 97.30 99.91
2 0.63 0.01 1.26 0.04
3 0.32 0.003 0.64 0.01
>3 0.40 0.03 0.8 0.04
Path Load = 1.8 Load = 2.0
# Unbal. Bal. Unbal. Bal.
1 94.00 99.09 90.83 96.46
2 2.75 0.52 4.23 2.00
3 1.37 0.12 2.01 0.47
>3 1.88 0.27 2.93 1.07
PERCENTAGE OF TIMES A GUARANTEED-CLASS CALL IS ROUTED OVER FIRST-,SECOND-,THIRD-AND GREATER-THAN-THIRD-CHOICE PATHS,
This paper has presented a load balancing path selection algo-rithm for static routing strategies, that is suited for hierarchical implementation in networks that support multiple QoS traffic.
The load balancing metric was based only on the minimum available bandwidth on the path; however metrics that are based also on the expected delay and/or delay variations are not diffi-cult to be defined and will be studied in the future.
The proposed sorting algorithm was implemented and a sim-ulation study was carried out on a complex, meshed topol-ogy, with two traffic classes, in order to compare the proposed scheme with a standard Minimum-Hop path selection algorithm, applying an uncontrolled alternate routing strategy to the paths selected. Results show that the proposed strategy allows the lower traffic class to obtain an increased share of resources, while obviously not affecting the higher class traffic.
The total traffic admitted within the network was not observed to change significantly; however it must be noted that only a single topology with uniform traffic distribution was studied in this early stage of the research and the situation might change significantly if the traffic load is not uniform. Future research will cover this point.
The authors wish to thank Gabriele Favalessa and Luca Medico for their contributions to the simulator code.
 E. Crawley, R. Nair, B. Rajagopalan, H. Sandick, “A Framework for QoS-based Routing in the Internet”, Internet-Draft, July 1998, work in progress  Q. Ma, P. Steenkiste. “Routing Traffic with Quality-of-Service Guarantees in Integrated Services Networks”, In Proceedings of the 8th IEEE/ACM
International Workshop on Network and Operating Systems Support for Digital Audio and Video (NOSSDAV’98), pages 115–126, England, July
 R. Guerin, S. Kamat, A. Orda, and T. Przygienda, “QoS Routing Mecha-nisms and OSPF Extensions”, Internet-Draft, Mar. 1998, work in progress  C. Huitema, Routing in the Internet, Prentice Hall, Englewood Cliffs, NJ,
 S. Sibal, A. DeSimone, Controlling Alternate Routing in General-Mesh
Packet Flow Networks, SIGCOMM’94, London, England, Aug. 1994
 Z. Wang, J. Crowcroft, “QoS Routing for Supporting Resource Reserva-tion”, IEEE Journal on Selected Areas in Communication, Sept. 1996  M. Marathe, R. Sundaram, S. Ravi, D. Rosenkrantz, H. Hunt, “Bicriteria
Network Design Problems”, Proceedings of the 22nd International
Col-loquium on Automata Languages and Programming, LNCS 944, pp.
 C. Casetti, G. Favalessa, R. Lo Cigno, L. Medico, M. Mellia, “Load-Balancing Solutions for Static Routing Schemes in ATM Networks”, Sixth
IFIP Workshop on Performance Modelling and Evaluation of ATM Net-works, Ilkley, U.K., 20-22 July, 1998
 M. Ajmone Marsan, A. Bianco, C. Casetti, C. F. Chiasserini, A. Francini, R. Lo Cigno, M. Mellia, M. Munaf`o, An Integrated Software Environment
for the Simulation of ATM Networks, SCSC’97, Arlington, VA, USA, July
 M. Ajmone Marsan, A. Bianco, C. Casetti, C. F. Chiasserini, A. Francini, R. Lo cigno, M. Munaf`o, “An Integrated Simulation Environment for the Analysis of ATM Networks at Multiple Time Scales”, Computer Networks
and ISDN Systems, Special Issue on Modeling of Wired and Wireless
-ATM Networks, Vol. 29, no. 17–18, pp. 2165–2185, Feb. 1998  ANCLES – ATM Networks Call-LEvel Simulator –
 M. Ajmone Marsan, A. Bianco, C. Casetti, P. Castelli, R. Lo Cigno, M. Mellia, M. Munaf `o, “A CAC Algorithm Supporting Different QoS Classes”, D. Kouvatsos (editor), Performance Modelling and Evaluation