A Microcell Oriented Load Balancing Model for Collaborative Virtual Environments

A Microcell Oriented Load Balancing Model for Collaborative Virtual

Environments

Dewan Tanvir Ahmed, Shervin Shirmohammadi

School of Information Technology and Engineering University of Ottawa, Canada

E–mail: {dahmed, shervin}@discover.uottawa.ca.

Abstract – In this paper, we present a microcell oriented load bal-ancing mechanism in the context of a hybrid MMOG (massively mul-tiplayer online game) architecture. The objective of the proposed approach is to define a fair load balancing model for peer-to-peer MMOGs and virtual environments that minimizes peer-to-peer distor-tions in the overlays. In the case of an overloaded server, the pro-posed approach identifies appropriate microcell(s) that reduces over-lay maintenance cost and minimizes inter-server communications. The load is relieved by devolving these microcells to the other less loaded servers. The model is justified through extensive simulation.

Keywords – Area of interest, hotspot, load Balancing, MMOG

I. INTRODUCTION

Since the experience of Networked Virtual Environments (NVE) in 1980s for military simulation, many other simi-lar applications have been introduced in this domain over the decades. Massively Multiplayer Online Game (MMOG) is a kind of NVE where many players interact simultaneously. Cur-rently, Second Life and World of Warcraft are the two most popular NVEs. Traditionally, commercial NVEs practice the client-server architecture which is expensive to deploy and maintain. For example, Second Life has approximately 5000 servers to support its virtual world (i.e. game world). Such huge deployment cost as well as regular maintenance demand suggests a need for an alternative solution. The P2P architec-ture can be a good solution in terms of scalability, but consid-ering the business prospect and the quality of gaming experi-ence, it reveals that the pure P2P architecture is an infeasible solution. Recently, the hybrid P2P architecture is considered as an ideal solution both for the vendors and end-users [1][2][3].

Crowding, a large group of people gathered in one place, is a natural phenomenon in MMOGs. The load-balancing al-gorithms partition the players’ data and workload among the servers, i.e. the masters. The effective load division greatly in-fluences the performance of the game. The decompositions that balance the masters’ loads and have a low inter-server communication costs are preferred. Although a wide variety of load-balancing algorithms have been developed, their effec-tiveness depends on the characteristics of the application using

them. We discuss current efforts toward improving partitioning algorithms for network game applications and architectures.

Numerous partitioning strategies have been developed. The strategies are distinguished by different trade-offs like partition quality, amount of data movement, etc. The characteristics of an application (e.g., computation-to-communication ratio, cost of data movement, and frequency of repartitioning) determine which strategies are most appropriate. For example, the geo-metric algorithms like recursive bisection partitioning provide high-speed, medium-quality decompositions that depend only on geometric information. The graph-based algorithms pro-vide higher quality decompositions based on communication cost among the partitions, but at a higher cost.

At its simplest, a load-balancing algorithm attempts to dis-tribute equal numbers of players to partitions while minimiz-ing inter-partition communication costs. A partition’s sub-zone consists of the data exclusively assigned to the sub-zone. The union of sub-zones is equal to the entire problem domain. The workloads in a continuously changing system evolve in time, so a partitioning policy that works well for a static system or for a slowly-changing environment may not be effective in a highly dynamic application. In finite element methods, for ex-ample, with adaptive mesh refinement, the process workloads can vary singnificantly as elements are added and/or removed from the mesh. The dynamic repartitioning of mesh data often called dynamic load balancing becomes necessary.

The zoning decomposes the game world into disjoint re-gions, i.e. zones (here we call microcell), that gives a provi-sion of easier state management. Each zone is coordinated by a server that performs the statement management with the active participation of the players; hence, it is a hybrid peer-to-peer system: there is a server that communicates to clients, and also clients communicate among each other. A zonal MMOG ar-chitecture must cover many aspects: zone organization, zonal communication strategy, interest management, seamless zone crossing, load balancing and many others [2]-[4]. In this re-search, we are only covering load balancing for collaborative virtual environments.

This paper presents a microcell oriented load balancing mechanism for the servers while minimizing peer-to-peer dis-tortions in the overlays. The road map of the paper is as

fol-lows. We study the nature of load balancing and related works in Section II. The microcell oriented load balancing approach is presented in Section III. The simulation results and corre-sponding analysis are covered in Section IV. Finally, we con-clude the paper in Section V with future research directions.

II. LITERATURE REVIEW

Generally, each user in a NVE is only interested in a small portion of the world where most of the interactions take place. This interaction space is called Area of Interest (AOI). In MMOG, it is possible that many players move into a zone caus-ing the degradation of the game’s quality and affectcaus-ing play-ers’ gaming experience. This problem is known as the hotspot problem. Chen et al. mention that flocking (i.e. hotspot or too many players moving into a zone administered by a server) is a natural phenomenon in MMOGs that cannot be avoided [5]. This is logical as some areas are more interesting than others in the game world. Moreover, this hotspot is unpredictable as it depends on game events like players chasing each other or performing a mission. Chen et al. also mention that in most cases, the static partitioning handles it poorly [5]. A preven-tive mechanism for migration is to impose boundaries between zones: cities are separated by forests or wilderness, where par-ties do not tend to stay for long [6]. Thus, a migrating player does not experience others’ interactions at the boundaries. But such restrictions cannot always be imposed, and even if ap-plied, they do not avoid hotpots in ”battle” situations where everyone must be in the same zone facing the enemy.

Takuji et al. propose a zoned federation model to adjust multiplayer online game to peer-to-peer network [10]. This ap-proach inserts a zoning layer between the game and the peer-to-peer network. A zone is a small space of a game world, and a zone owner is an authoritative server of a specific zone. Here each node changes its role to a zone owner and works like a centralized authoritative server. The workloads of the authoritative centralized game server are distributed to a feder-ation of nodes by dividing the whole game world into zones. A player keeps connection to the authoritative node in order to receive latest updates directly and to request for modified data. The authoritative node for a particular global state is discov-ered by DHT (distributed hash table). In other words, a zon-ing layer confines the update request and notification within a single zone, and the zoning layer depends on DHT layer for data storage and rendezvous. The big question is the feasi-bility of carrying out complete server functionalities through a peer node. One of the important design issues is that sub-spaces overlapped with each player’s view must be switched quickly and seamlessly as the view moves around the game space. The zone federation approach does not propose tech-niques for achieving this in detail, and the concept of a data holder causes high latencies. It also does not handle the sit-uation if the zone owner dies. Overall, the complexity of the proposed architecture makes it difficult to deploy.

Lu et al. present some techniques required for load balanc-ing [3]. An ideal solution ensures equal load distribution to prevent exhausting one server while keeping spare resources at other servers. Theoretically, it is a good solution as it is ideal and does not disrupt players’ gaming experience. But assess-ing the load of the servers, frequently migratassess-ing players from one sever to another, and also tracking the area of interest seem to be a big problem in this regard. This would be even more critical for hybrid systems where the players have routing re-sponsibility.

According to Duong et al., the dynamic load distribution methods can be categorized into load sharing algorithms and load balancing algorithms [7]. The load sharing algorithms prevent unbalanced load among servers. The load balancing algorithms address the situation by equalizing the workload among all servers in a distributed manner. The computational overhead associated with load balancing reduces algorithms’ effectiveness [8], and subsequently the gaming experiences.

In this paper, we present a load balancing model for peer-to-peer MMOGs. In the case of an overloaded server, i.e. the master, the proposed approach determines one or more suitable microcells (a small region in the virtual space) of that loaded server. The microcells are chosen to reduce overlay mainte-nance cost and to minimize inter-server communications. The load is reduced by devolving those microcells to the other less loaded servers.

III. THE MICROCELL ORIENTED LOAD BALANCING MODEL FOR HYBRID MMOG

In this model, a set of microcells defines the virtual game world that is maintained and coordinated by a set of servers called server pool. The server pool having n servers is rep-resented by a set S = {S1, S2, ..., Sn}. Each server, e.g.

Si, serves one or more microcells depending on load

distribu-tion and avatars’ populadistribu-tion in the corresponding microcells. The microcells are given unique identification numbers: left to right, top to bottom in order.

In zonal MMOGs, the game world is usually divided into several manageable regions like zones. The zone concept is not entirely new and has been considered over the last few years. The hexagonal shape has a couple of advantages over others like circular [4], hence considered in this design to de-fine the shape of a microcell. Each microcell is run by a server called master: a special node that coordinates the interactions in a collaborative manner with the active participation of the microcell members, i.e. avatars. So, a set of master nodes, i.e. the server pool, regulates the operation of the MMOG and provides overlay services. In that sense, the system is hybrid as it combines the benefits of both centralized and distributed systems. The layout of microcells and a random microcell dis-tribution to servers are shown in Figure 1.

Fig. 1. The layout of microcells and a random microcell distribution to servers

A. Load definition

In this paper, we are dealing with hotspots. So, the implied resolution in this regard is that the server pool can handle the total system load. As described before, a hotspot refers to a crowded region in a zonal MMOG, leading to the need for load balancing. But, the key question is how to figure out a hotspot. Most of the approaches like [2] consider the total number of players in a zone/microcell as an identifying attribute. If the total number of players in a microcell exceeds the maximum affordable limit, it is a hotspot. Logically, it is correct. But, here we look at it in a different way. In MMOG, there are many players like soldiers, robots, aircrafts, etc with various features. In addition, the actions that actually take place in a game have an influence on message generation rate. For example, in spite of the regular updates from a player, there may be an imme-diate message due to a trigger in a pistol by an avatar. Every message requires reaching to every other interested player in the interested region which eventually increases the load of the master. So, the load is not completely related to the number of players, actually, it is purely linked with the message gener-ation rate. Thus, the message genergener-ation rate, Rm_{, is used to}

estimate the load of the masters.

B. Marking of a loaded server

Say, the server, Si, is in charge of a set of microcells, Mi =

{mi1, mi2, ..., mik}. Let R

m

i = {ri1, ri2, ..., rik} are the

mes-sage generation rate of the microcells {mi1, mi2, ..., mik}

re-spectively where all of them are governed by server Si. Thus,

the load of the server Siaccording to message generation rate

is:

Rm i =

Pk

j=1rij

Let us define a message threshold term Tm_{. Thus, a server is}

loaded or there is a hotspot if Rm_i > Tm.

C. An approach to load balance

When there is a hotspot, the load of the concerning server is relieved by moving one or more microcells to other less loaded

Fig. 2. The buffer region and microcell covering

servers. As the movement of microcell introduces complexi-ties, some intelligence is required to identify the best possible set of microcells that need to be moved.

The reduced deployment and maintenance overhead of the hybrid MMOGs (peers or avatars perform some of the message relaying tasks) come at the cost of performance due to peer dy-namics. Many techniques have been proposed in literatures such as entity typing and the smart interest driven zone cross-ing with buffer region [9] to keep the desired level of perfor-mance. As it is difficult to predict avatars’ movement around a cell boundary (possible frequent in and out movements), it could be effective to share the relevant state information be-tween two adjacent microcells. Instead of sharing the complete state information, a buffer region is defined between two adja-cent microcells, as shown in Figure 2. This greatly reduces ex-changing of unimportant messages. But, if two adjacent micro-cells are handled by a single master, there is no buffer region in between, as shown in Figure 2. According to that figure, there is a buffer region between Microcell 2-4 and Microcell 3-3 as they are administered by two different servers but Microcell 2-4 and Microcell 2-3 have a common master and hence no buffer region in between.

The objective of changing a microcell ownership from one master to another is to approach hotspot. But, the key point that must be taken care of is to minimize inter-server commu-nication cost while carrying out such microcell ownership al-teration. Let, Ii is the number of interested links completely

internal to the Microcell i among the avatars and Bi,j is the

number of interested links involved between Microcell i and Microcell j in the buffer region.

Let, the Server Siis overloaded and currently serving k

mi-crocells: Mi= {mi1, mi2, ..., mik}. Our target is to determine

an ordered list of microcells so that the microcell ownership alteration according to that order will introduce less perfor-mance penalty in the overlays of hybrid MMOGs. The algo-rithm initially forms group of microcells based on the micro-cell’s neighborhood properties. A microcell can be a member of a nonempty group if and only if it has a common edge with any microcells of that group. Let, after the application of this grouping policy, Server Si has l groups Gi1, G

i 2, ..., G

i l in

or-der of their cardinality, i.e. |Gi1| ≤ |Gi2| ≤ ... ≤ |Gil|. The

general strategy is to find a microcell within a group with a minimum number of interested links among the microcells of that group. Due to the change of ownership, these links be-come foreign links that will cause inter-server communication. Thus, a group with more microcells may have more common edges than a small group, at least heuristically. Thus, it can be effective to keep those groups intact. So, a group with a fewer microcells is chosen for load balancing and that is why we start with the group Gi1, then Gi2 and so on, if necessary.

Finally within a group, a microcell with a minimum number of potential foreign links is considered for ownership alteration.

IV. SIMULATION AND ANALYSIS

We have carried out detailed simulation to verify the pro-posed model and to evaluate the performance. The focused MMOG genres are MMORPG (Massively Multi-player On-line Role Playing Game) and FPS (First Person Shooter). We have chosen the realistic traffic for these two types of game due to their varying characteristics, as explained below. In some sense, MMORPG and FPS games are similar because of their low bandwidth requirement. Compared to FPS, the bandwidth requirement of MMORPG is even lower due to its strategic na-ture. In addition, the game traffic of an MMORPG has strong periodicity but usually follows temporal locality.

The traffic pattern of a client is a composite function of dif-ferent factors for the chosen game. Most of the cases, the pat-tern follows the exponential distribution, e.g. ShenZhu online [10]. The packet interarrival time is in between 0ms to 600ms. In our simulation, each player’s traffic is modeled using expo-nential distribution and the packet interarrival time was set to 550ms as measured in other works like in [10]. It should be noted that, the client packet interarrival time is independent on the number of gamers [11]. The most computationally expen-sive part of the cycle in FPS like Quake is rendering. It causes the slower hosts to have significantly higher and more variable interarrival times, while the fastest hosts transmit most packets at 14ms intervals [12][13]. Thus, the client traffic can be mod-eled as one extreme [13] or two extreme distributions [14]. We considered one extreme distribution function. The client traffic

Fig. 3. MMORPG client traffic from ShenZhu: exponential distribution, mean packet inter-arrival time 550ms

Fig. 4. Quake client traffic: extreme distribution, a = 50, b = 10

of ShenZhu and Quake are shown in Figure 3 and Figure 4, respectively, for an hour long simulation in two scales: w.r.t. second (the top one) and w.r.t. minute (the bottom one). From this setup it reveals that on an average each client generates more traffic in FPS compared to MMORPG.

The inbound traffic of a server depends on the number of microcells administered and the number of players in each mi-crocell. The summarized simulation parameters for MMORPG are given in Table I. As explained earlier, we have considered packet counts to define load. The traffic volume of a microcell is shown in Figure 5 for 20 players according the distribution mentioned earlier. The aggregate server traffic from eight mi-crocells are shown in Figure 6 which has an average of 293.52 packet/sec with a variance of 329.37.

TABLE I. The simulation setup parameters

Parameter Values

Simulation platform Microsoft Windows XP Professional (SP2)

Tools used Matlab

Run time 3600 sec

Number of servers 100

Number of microcells 1000

Averger number of players in each microcell 15 to 30

Server’s average capacity 8 microcells

Fig. 5. The traffic volume of a microcell in MMORPG (The size of a

microcell: 20) Fig. 6. The Aggregate server traffic of eight microcells in MMORPG

managing model is justified thorough Figure 7 and Figure 8. For an hour long simulation with identical setup, we observed system load at every minute. Figure 7 shows the standard devi-ation of packets being processed by all servers of an MMORPG system. It clearly shows the load imbalance of the servers. The significance of the microcell based MMOG model is the balanced load of the deployed servers. From Figure 8, it re-veals that the standard deviation of packets being handled by the servers has dropped to a large extend.

V. CONCLUSION

In this paper, we have demonstrated a microcell oriented load balancing mechanism for peer-to-peer MMOGs. The ob-ject of the proposed approach is to discover one or more mi-crocells of a loaded server in terms of low movement cost. In the case of an overloaded server, the proposed approach identi-fies appropriate microcell(s) that reduces overlay maintenance cost and minimizes inter-server communications. The load is reduced by devolving these microcells to the other less loaded

Fig. 7. The standard deviation of the processed packets by the servers overtime

Fig. 8. The standard deviation of the processed packets by the servers overtime using microcell model

servers.

VI. REFERENCES

[1] A. Yu and S. T. Vuong, “MOPAR: a mobile peer-to-peer overlay rrchi-tecture for interest management of massively multiplayer online games,” in NOSSDAV ’05: Proceedings of the international workshop on Net-work and operating systems support for digital audio and video, New York, NY, USA, 2005, pp. 99–104, ACM Press.

[2] I. Kazem, D.T. Ahmed, and S. Shirmohammadi, “A visibility-driven ap-proach to managing interest in collaborative virtual environments with dynamic load balancing,” in Proceedings of IEEE International Sym-posium on Distributed Simulation and Real Time Applications (DS-RT, 2007.

[3] T. Iimura, H. Hazeyama, and Y. Kadobayashi, “Zoned federation of game servers: a peer-to-peer approach to scalable multi-player online games,” in NetGames ’04: Proceedings of 3rd ACM SIGCOMM work-shop on Network and system support for games, New York, NY, USA, 2004, pp. 116–120, ACM.

[4] D.T. Ahmed, S. Shirmohammadi, and I. Kazem, “Zone based messaging in collaborative virtual environments,” in IEEE International Workshop on Haptic Audio Visual Environments and their Applications (HAVE), 2006, pp. 165–170.

[5] J. Chen, B. Wu, M. Delap, B. Knutsson, H. Lu, and C. Amza, “Local-ity aware dynamic load management for massively multiplayer games,” in PPoPP ’05: Proceedings of the tenth ACM SIGPLAN symposium on Principles and practice of parallel programming, New York, NY, USA, 2005, pp. 289–300, ACM Press.

[6] B. BDe Vleeschauwer, B. Van Den Bossche, T. Verdickt, F. De Turck, B. Dhoedt, and P. Demeester, “Dynamic microcell assignment for mas-sively multiplayer online gaming,” in NetGames ’05: Proceedings of 4th ACM SIGCOMM workshop on Network and system support for games, New York, NY, USA, 2005, pp. 1–7, ACM.

[7] T.N. Binh Duong and S. Zhou, “A dynamic load sharing algorithm for massively multiplayer online games,” in ICON ’03: Proceedings of 11th IEEE International Conference on Networking, 2003, pp. 131–136. [8] F. Lu, S. Parkin, and G. Morgan, “Load balancing for massively

multi-player online games,” in NetGames ’06: Proceedings of 5th ACM SIG-COMM workshop on Network and system support for games, New York, NY, USA, 2006, p. 1, ACM Press.

[9] D.T. Ahmed, S. Shirmohammadi, and J. Oliveira, “Improving gaming experience in zonal MMOGs,” in MULTIMEDIA ’07: Proceedings of the 15th international conference on Multimedia, New York, NY, USA, 2007, pp. 581–584, ACM.

[10] K. Chen, P. Huang, C. Huang, and C. Lei, “Game traffic analysis: an mmorpg perspective,” in NOSSDAV ’05: Proceedings of the interna-tional workshop on Network and operating systems support for digital audio and video, New York, NY, USA, 2005, pp. 19–24, ACM. [11] R.A. Bangun, E. Dutkiewicz, and G.J. Anido, “An analysis of

multi-player network games traffic,” Multimedia Signal Processing, 1999 IEEE 3rd Workshop on, pp. 3–8, 1999.

[12] T. Lang, P. Branch, and G. Armitage, “A synthetic traffic model for quake3,” in ACE ’04: Proceedings of the 2004 ACM SIGCHI Interna-tional Conference on Advances in computer entertainment technology, New York, NY, USA, 2004, pp. 233–238, ACM.

[13] M.S. Borella, “Source models of network game traffic,” 2000. [14] C. Chambers, W. Feng, S. Sahu, and D. Saha, “Measurement-based

char-acterization of a collection of on-line games,” in IMC ’05: Proceedings of the 5th ACM SIGCOMM conference on Internet measurement, New York, NY, USA, 2005, pp. 1–14, ACM.