Classification of Load Balancing in a Distributed System

(1)

IJEAM www.ijeam.com 100

Classification

Classification of Load Balancing in a Distributed System

of Load Balancing in a Distributed System

Divya Aaggarwal1, Vikas Siwach2 1

M. Tech. Student (CSE) U.I.E.T., MDU, Rohtak (Haryana) 2

Assistant Professor, U.I.E.T., MDU, Rohtak (Haryana)

Abstract

Load balancing is the process of improving the performance of a parallel and distributed system through a redistribution of load among the processors. The purpose of load balancing in Distributed System is to minimize the response time by transferring the load from overloaded nodes to under loaded-nodes. In this paper, various types of load balancing techniques are discussed that are currently available. The main purpose of this paper is to classify various load balancing algorithms/techniques which helps in future for designing new techniques.

Keywords: Distributed System, Load Balancing, Load Balancing Algorithms.

1. Introduction

Distributed computing systems have become a natural setting in many environments for business and academia. This is due to the rapid increase in processor and/or memory hungry applications coupled with the advent of low-cost powerful workstations [1].The cost/performance ratio of networks of workstations has been constantly improving. This trend is expected to continue in the near future. The aggregate peak rate of such systems often matches or exceeds the peak rate offered by the fastest parallel computers. Thus, distributed computing systems are a viable and less expensive alternative to parallel computers. However, a serious difficulty in concurrent

programming of a distributed system is how to deal with scheduling and load balancing of such a system which may consist of heterogeneous computers. A distributed system can be viewed as a collection of computing and communication resources shared by active users. When the demand for computing power increases the load balancing problem becomes important.

Figure 1: Distributed System

In a typical distributed system setting, tasks arrive at the different nodes in a random fashion. This causes a situation of non-uniform loading across the system nodes to occur. Loading imbalance is observed by the existence of nodes that are highly loaded while others are lightly loaded or even idle. Such situations are harmful to the system performance in terms of the mean response time of tasks and resource utilization [1].

(2)

A system [2-3] of distributed computers with tens or hundreds of computers connected by high-speed networks has many advantages over a system that has the same standalone computers. A distributed system provide the resource sharing as one of its major advantages, which provide the better performance and reliability than any other traditional system in the same conditions[4].

In distributed systems more than one processor processing parallel programs. The amount of processing time needed to execute all processes assigned to a processor is called workload of a processor. A system [5] [6] of distributed computers with tens or hundreds of computers connected by high speed networks has many advantages over a system that has the same standalone computers. A distributed system provide the resource sharing as one of its major advantages, which provide the better performance and reliability than any other traditional system in the same conditions. One of the research issues in distributed systems is the development of effective techniques for distributing workload on multiple processors. The main goal is to distribute the jobs among processors to maximize throughput, maintain stability, resource utilization and should be fault tolerant in nature. In section 2 we discussed load balancing and its classification in brief. Categorizations of different load balancing algorithms are enlisted in section3.

2. Load Balancing

In computer networking, load balancing is a technique to spread work between two or more computers, network links, CPUs, hard drives, or other resources, in order to get

optimal resource utilization, throughput, or response time. Using multiple components with load balancing, instead of a single component, may increase reliability through redundancy [29]. Load balancing attempts to maximize system throughput by keeping all processors busy Load balancing is done by migrating tasks from the overloaded nodes to other lightly loaded nodes to improve the overall system performance.

Load Balancing based on the idea of migration of excess load from heavily loaded node to lightly loaded ones. The problem starts with to determine when to migrate a process or task [30].This solution is typically based on local load situation: for example, a simple procedure may be the comparison of the load between various nodes with those of the neighboring node and a determination of the node to which the task is to be migrated.

Jobs in a distributed system can be divided into different classes based on their resource usage characteristics and ownership. For example the jobs that belong to a single user can form a class. Alternatively, we can distinguish different classes of jobs by their execution times [31]. There are three typical approaches to load balancing problem in single class job distributed systems:

a) Global approach: In this case there is only one

decision maker that optimizes the response time of the entire system over all jobs and the operating point is called social optimum. This is the classical approach and has been studied extensively using different techniques such as

(3)

nonlinear optimization [7, 8] and polymatroid optimization [9].

b) Cooperative approach: In this case there are

several decision makers (e.g. jobs, computers) that cooperate in making the decisions such that each of them will operate at its optimum. Decision makers have complete freedom of preplay communication to make joint agreements about their operating points. This situation can be modeled as a cooperative game and game theory offers a suitable modeling framework [10].

c) Non-cooperative approach: In this case each of

infinitely many jobs optimizes its own response time independently of the others and they all eventually reach equilibrium. This situation can be viewed as a non-cooperative game among jobs. The equilibrium is called Wardrop equilibrium [11]. At the Wardrop equilibrium a job cannot receive any further benefit by changing its own decision. If the number of jobs are finite the Wardrop equilibrium reduces to the well-known Nash equilibrium [10].

2.1 Types of Load Balancing

There are two types of load balancing algorithms:

A. Static Load-Balancing

In this method, the performance of the nodes is determined at the beginning of execution. Then depending upon their performance the workload is distributed in the start by the master node. The slave processors calculate their allocated work and submit their result to the master. A task is

always executed on the node to which it is assigned that is static load balancing methods are non-preemptive. A general disadvantage of all static schemes is that the final selection of a host for process allocation is made when the process is created and cannot be changed during process execution to make changes in the system load [12]. Major load balancing algorithms are Round Robin [13] and Randomized Algorithms[14], Central Manager [15]Algorithm and Threshold[12,16] Algorithm.

B. Dynamic Load-Balancing

It differs from static algorithms in that the workload is distributed among the nodes at runtime. The master assigns new processes to the slaves based on the new information collected [17,18]. Unlike static algorithms, dynamic algorithms allocate processes dynamically when one of the processors becomes under loaded. Instead, they are buffered in the queue on the main host and allocated dynamically upon requests from remote hosts[12]. This method is consisted of Central Queue Algorithm and Local Queue Algorithm [19].

Load balancing algorithms work on the principle that in which situation workload is assigned, during compile time or at runtime. Comparison shows that static load balancing algorithms are more stable compare to dynamic. It is also ease to predict the behavior of static, but at the same time, dynamic distributed algorithms are always considered better than static algorithms [12].

(4)

Processing power does not mean only the processing speed of Processor; it includes the overall configuration of node. In applications with constant workloads, static load balancing can be used as a pre-processor to the computation. Other applications, such as adaptive finite element methods, have workloads that are unpredictable or change during the computation; such applications require dynamic load balancers that adjust the decomposition as the computation proceeds. Load balancing algorithms vary in their complexity where complexity is measured by the amount of communication used to approximate the least loaded node. Static algorithms collect no information and make probabilistic balancing decisions, while dynamic algorithms collect varying amounts of state information to make their decisions. The most significant parameter of the system was found to be the cost of transferring a job from one node to another. It is this cost that limits the dynamic algorithms, but at the high end of complexity are the dynamic algorithms which do collect varying amounts of information. Potentially the more information an algorithm can collect the better decision it will make. The problem with the complex balancing algorithms is that they cannot keep up with the rapidly changing state information of the system. The best they can do is to make their decisions based on information passed between two nodes i.e. the node itself and the node that sent it the information concerning its queue length.

Load balancing is done basically to do following benefits.

• Load balancing reduces mean job response time under job transfer overhead.

• Load balancing increase the performance of each host.

• Small jobs will not suffer from starvation. Since load imbalance leads directly to processor idle times, high efficiency can only be achieved if the computational load is evenly balanced among the processors.

Generally a load balancing scheme consists of three phases-1.Information collection, 2.Decision-Making and 3.Data migration.

1. Information collection: During this phase the load

balancer gathers the information of workload distribution and the state of computing environment and detects whether there is a load imbalance.

2. Decision-Making: This phase focuses on calculating an

optimal data distribution.

3. Data migration: This phase transfer the excess amount

of workload from overloaded processor to under loaded ones.

3. Categorization of Different Load

Balancing Algorithms

A number of Dynamic load balancing algorithms are existing for different systems; a small description is presented for each algorithm, followed by a detailed classification.

3.1 Decision

and

Migration

based

algorithms

[25],[26],[8] These algorithms are classified as fallows.

3.1.1. Local Decision and Local Migration

(LDLM):

In this strategy a processor distributes some of his load to its neighbors after each fixed time interval. This is a LDLM because the

(5)

decision to migrate a load unit is done purely local. The receiver of a load is also a direct neighbor. The balancing is initiated by a processing unit which sends a load unit. We implemented this strategy after x iterations of the simulator y load units are sent to random neighbors.

3. 1.2. Direct neighborhood (d-N):

If the

local load increased by more than Up percent or decrease by more than Down percent, actual value is broadcasted to direct neighbors. if the load of a processing element exceeds that of its least neighbor load by more than d percent, then it sends one unit to that neighbor.

3.1.3. Local

Decision

and

Global

Migration (LDGM):

In this strategy the load units are migrated over the whole network to a randomly chosen processor.

3.2 The Random Algorithm [16]:

Each

time an element (task) is created on a processor, it is sent on a randomly selected node anywhere in the system. For each node, the expectation to receive a part of the load is the same regardless of its place in the system.

3.3 The Tiling (Direct Neighborhood DN)

Algorithm [13]:

It divides the system in small and disjointed sub-domains of processors called windows. A perfect load balancing is realized in each window using

regular communications. In order to propagate the work over the entire system, the window is shifted (slightly moved so that they overlap only a part of the old domain) for the next balancing phase.

3.4 The X-Tiling Algorithm [13]: Similar to

Tiling algorithm but extra links are added to the current topology of the processor to form a hypercube topology.

3.5 The Rake Algorithm [13]: It uses only

regular communications with processors in the neighborhood set. Firstly, the average load is processed and broadcasted. In the first transfer phase, during p iterations, each processor sends to its right neighbor the data over the average workload It uses only regular communications with processors in the neighborhood set. Firstly, the average load is processed. In the second transfer phase, during the extra workloads, each processor sends to its right the work over the average workload + 1.

3.6 The Pre-Computed Sliding Algorithm

[13]: It is an improvement of the Rake

algorithm. Instead of transferring data over the average workload of the system, it computes the minimal number of data exchanges needed to balance the load of the system. Unlike the Rake algorithm, it may send data in two directions.

3.7 The Average Neighbor Algorithm

(6)

islands. An island is made of a center processor and all the processors in its neighborhood. It works on the load balancing every node in the island. The partial overlapping allows the load to propagate.

3.8 The Direct Neighborhood Repeated

Algorithm [21]:

Once a sender-receiver

couple is established, the migrating load can move from the sender to the receiver. In its turn, the receiver can have an even less neighborhood. The receiver is allowed to directly forward the migrating load to the less loaded nodes. Load migration stops when there are no more useful transfers.

3.9 The Central Algorithm [11], [28]:

Firstly, the average workload is computed and broadcasted to every processor in the system. Then, the processors are classified into 3 classes: idle, overloaded, and the others. The algorithm tries to match each overloaded node with an idle peer.

Table Table Table

Table 1111: Various Load Ba: Various Load Ba: Various Load Ba: Various Load Balancing Algorithmslancing Algorithmslancing Algorithms lancing Algorithms

S. No.

Algorithm

Description

1. Decision and Migration based algorithm

There are three types: a) Local decision and

local migration b) Direct

neighborhood c) Local decision and

global migration 2. Random In this, each time an

Algorithm element is created on processor and sent randomly selected node anywhere in the system. 3. Tiling Directed

neighborhood Algorithm

A perfect load balancing is realized in each window using regular communications.

4. X-Tiling Algorithm

In this extra links are added to the current topology of the processor to form a hypercube topology. 5. Rake

Algorithm

It uses only regular communications with processors in the neighborhood set. 6. Pre-Computed Sliding Algorithm

It computes the minimum no. of data exchanges needed to balance load of the system.

7. Average Neighbor Algorithm

It works on the load balancing every node in the island. The partial overlapping allows the load to propagate.

8. Direct Neighborhood Repeated Algorithm

In this receiver is allowed to directly forward the migrating load to the less loaded nodes.

9. Central Algorithm

In this average workload is computed and broadcasted to every processor in the system.

4. Conclusion

The main aim in distributed system is to execute the process at minimum cost i.e. time is most important factor can be considered in cost calculation. Load balancing is the task of distribution of application tasks to different processors in an efficient manner to minimize program execution time. It involves assigning work to each processor proportional to its computing power, hence minimizing the idle time and at the same time overload of

(7)

work to the processor. The Classification is considered helpful for designers to compare different load-balancing algorithms and design new ones tailored for their needs. In the future, we intend to develop a framework for applications with load balancing that utilizes this Classification and helps the designer to make his own algorithm.

References

[1] A. E. El-Abd, "Load balancing in distributed computing systems using fuzzy expert systems," presented at International Conference on Modern Problems of Radio Engineering, Telecommunications and Computer Science (TCSET 2002), Lviv-Slavsko, Ukraine, 2002.

[2] S. Malik, "Dynamic Load Balancing in a Network of Workstation," 19 November 2000 2000.

[3] D. L. Eager, E. D. Lazowska, and J. Zahorjan, "Adaptive load sharing in homogeneous distributed systems," IEEE Trans. Softw. Eng., vol. 12, pp. 662-675, 1986.

[4] S. Sharma, S. Singh, and M. Sharma, "Performance Analysis of Load Balancing Algorithms," World Academy of Science, Engineering and Technology, vol. 38, 2008.

[5] S. Malik, “Dynamic Load Balancing in a Network of Workstation”, 95.515 Research Report, 19 November, 2000.

[6] Derek L. Eager, Edward D. Lazowska , John Zahorjan, “Adaptive load sharing in homogeneous distributed systems”, IEEE Transactions on Software Engineering, v.12 n.5, p.662-675, May 1986.

[7] X. Tang and S. T. Chanson. Optimizing static job scheduling in a network of heterogeneous computers.

In Proc. of the Intl. Conf. on Parallel Processing, pages 373–382, August 2000.

[8] A. N. Tantawi and D. Towsley. Optimal static load balancing in distributed computer systems. Journal of the ACM, 32(2):445–465, April 1985.

[9] K. W. Ross and D. D. Yao. Optimal load balancing and scheduling in a distributed computer system.

Journal of the ACM, 38(3):676–690, July 1991.

[10] D. Fudenberg and J. Tirole. Game Theory. The MIT Press, 1994.

[11] H. Kameda, J. Li, C. Kim, and Y. Zhang. Optimal

Load Balancing in Distributed Computer Systems.

Springer Verlag, London, 1997.

[12] S. Sharma, S. Singh, and M. Sharma, "Performance Analysis of Load Balancing Algorithms," World Academy of Science, Engineering and Technology, vol. 38, 2008.

[13] Z. Xu and R. Huang, "Performance Study of Load Balancing Algorithms in Distributed Web Server Systems " CS213 Parallel and Distributed Processing Project Report.

[14] R. Motwani and P. Raghavan, "Randomized algorithms," ACM Comput. Surv., vol. 28, pp. 33-37, 1996.

[15] P. L. McEntire, J. G. O'Reilly, and R. E. Larson, Distributed Computing: Concepts and Implementations. New York: IEEE Press, 1984. [16] W. I. Kim and C. S. Kang, "An adaptive soft handover

algorithm for traffic-load shedding in the WCDMA mobile communication system," presented at WCNC'2003, 2003.

[17] S. Malik, "Dynamic Load Balancing in a Network of Workstation," 19 November 2000 2000.

[18] Y.-T. Wang and A.-R. J. T. Morris, "Load Sharing in Distributed Systems," IEEE Transactions on Computers, vol. 34, pp. 204- 217, 1985.

(8)

[19] W. Leinberger, G. Karypis, and V. Kumar, "Load Balancing Across Near-Homogeneous Multi-Resource Servers," presented at s,Cancun, Mexico, 2000.

[20] S.Zhou, A Trace Driven Study Of Load balancing ,

IEEE Trans.On Software

Engineering,Vol.14,no.9,1988,pp,1327-1341.

[21] F. Ramme, Lastausgleichsverfahren in Vertileilten Systemen, Master Thesis,University of Paderborn.1990.

[22] Powley, C., Ferguson, C. and Korf, R. E.: Depth-First Heuristic Search on a SIMD Machine. Artificial Intelligence, vol. 60 (1993) 199-242.

[23] Subramanian, R. and Scherson, I.: An Analysis of Diffusive Load Balancing. Proceedings of Sixt h Annual ACM Symposium on Parallel Algorithms and Architectures, (June 1994) 220-225 .

[24] Dekeyser, J. L., Fonlupt, C. and Marquet, P.: Analysis of Synchronous Dynamic Load Balancing algorithms”, Parallel Computing: State-of-the Art Perspective (ParCo'95), volume 11 of Advances in Parallel Computing, pages 455--462, Gent, Belgium ( September 1995)

[25] Saletore, V. A.: A distributive and adaptive dynamic load balancing scheme for parallel processing of medium- grain tasks. Proceedings of the 5th Distributed Memory Conference (April 1990) 995-999.

[26] Willebeek- LeMair, M.H. and Reeves, A.P.: Strategies for dynamic load balancing on highly parallel computers. IEEE Trans. on parallel and distributed systems, vol. 4, No. 9 (Sept. 1993).

[27] Corradi, A., Leonardi, L. and Zambonelli, F.: Diffusive load-balancing policies for dynamic applications. IEEE Concurrency Parallel, Distributed and Mobile Computing (January-March 1999) 22-31. [28] Hillis, W.D.: The Connection Machine. MIT press,

1985.

[29] Abbas Karimi, Faraneh Zarafshan , Adznan b. Jantan, “A New Fuzzy Approach for Dynamic Load Balancing Algorithm”, (IJCSIS) International Journal of Computer Science and Information Security, Vol. 6, No. 1, 2009

[30] Parveen Jain, Daya Gupta, “An Algorithm for Dynamic Load Balancing in Distributed Systems with Multiple Supporting Nodes by Exploiting the Interrupt Service”, International Journal of Recent Trends in Engineering, Vol 1, No. 1, May 2009. [31] Daniel Grosu, Anthony T. Chronopoulos Ming-Ying

Leung, “Load Balancing in Distributed Systems: An Approach Using Cooperative Games.”, NASA NAG 2-1383 (1999-2001)