Transparent Componentisation:
A hybrid approach to support
the development of contemporary
distributed systems
Shen Lin
School of Computing and Communications
Lancaster University
A thesis submitted for the degree of
Doctor of Philosophy
I wish to thank the many people who have helped and encouraged me to reach the completion of my PhD. First of all, my supervisors Dr. Fran¸cois Ta¨ıani and Professor Gordon Blair, who has provided advice, encouragement, insightful criticism, and proofreading throughout my PhD and in the preparation of this thesis.
Secondly, the members of the Middleware Research Group, who of-fered me invaluable support and a friendly work environment during my time at Lancaster. I am grateful to Geoff Coulson, Paul Grace, Bholanathsingh Surajbali, Carlos Flores Cortes, Barry Porter, Rajiv Ramdhany, Vatsala Nundloll, and Nelly Bencomo for their helpful advices and guidance.
Finally, I would like to thank my parents Jianyong and Hua, my uncle Qiang, and my fianc´ee Xiaolu, for the love and supports that I received from them.
Abstract
Distributed computing systems are increasingly pervading all aspects of daily life. This rapid growth is characterised by the growing com-plexity of these systems, which unfolds in three dimensions. First, contemporary distributed systems must often cater for computation nodes with heterogeneous computing and networking capacities; sec-ond, they must deal with dynamic changes such as network churns and mobile nodes; and finally, they are often large scale and must be able to grow elastically to meet evolving expectations.
This thesis investigates how the above complexity dimensions can be made easier to control by using novel software development ap-proaches and frameworks. In particular, the proposed work seeks to develop approaches that promote three key properties in contempo-rary distributed systems: 1) configurability to construct customised systems that target heterogeneous operating environments; 2) dy-namic adaptability to adapt to dynamic changes; and, 3) understand-ability and simplicity to facilitate software reuse and to hide low-level
To address these issues, this thesis proposes a hybrid software devel-opment approach that combines the advantages of component frame-works with that of high-level protocol specification languages. This hybrid approach, termed Transparent Componentisation, automati-cally maps a high-level protocol specification onto an underlying com-ponent framework. It thus allows developers to focus on the program-matic description of a distributed system’s behaviour in simple and high-level terms. Meanwhile, it transparently retains the benefits of a component architecture such as component reuse, configurability, and runtime adaptability. As a proof of concept, this thesis presents the Whispers/GossipKit framework for gossip-based distributed sys-tems, a representative subclass of contemporary distributed systems. Whispers/GossipKit is evaluated to demonstrate that it success-fully retains the simplicity and understandability of a high-level pro-tocol specification language while encouraging component reuse and supporting transparent (re)configuration thanks to its component un-derpinnings.
Contents
1 Introduction 1
1.1 Gossip-based Systems . . . 3
1.2 Software Development Approaches . . . 7
1.3 Hypothesis and Approach . . . 10
1.4 Thesis Outline . . . 14
1.5 Associated Publications . . . 15
2 Gossip Protocols 17 2.1 Introduction to Gossip Protocols . . . 18
2.1.1 Gossip for Disseminating Information . . . 19
2.1.2 Peer Sampling Service . . . 22
2.1.3 Convergence of Properties . . . 24
2.1.4 Gossip on Mobile Adhoc Networks . . . 25
2.2 Gossip Patterns . . . 27
2.2.1 Underlying Elements . . . 27
2.2.2 Key Patterns . . . 30
2.2.3 Other Contextual Issues . . . 37
2.3 Coexisting Gossip Protocols . . . 44
2.3.1 Types of Coexistence . . . 44
2.3.2 Example of Complex Gossip Composite . . . 44
2.3.3 Discussion . . . 47
3 Software Development Approaches 51
3.1 Component-based Technology . . . 52
3.1.1 Overview of Component Approaches . . . 53
3.1.2 Lightweight Component Framework . . . 55
3.1.3 Component-based Middleware Platform . . . 57
3.1.4 Event-driven Systems . . . 62
3.2 Domain Specific Languages . . . 64
3.2.1 Domain Specific Languages for Components . . . 68
3.2.2 High-level Protocol Specification Languages . . . 74
3.2.3 Discussion . . . 78
3.3 Summary . . . 80
4 The Whispers/GossipKit Programming Framework 82 4.1 General Principle . . . 83
4.2 Overview of The Whispers/GossipKit Programming Framework 85 4.3 The GossipKit Component Framework . . . 88
4.3.1 Overview of the GossipKit Framework . . . 89
4.3.2 Common Architecture of Gossip Protocols . . . 91
4.3.3 Architecture Generalisation . . . 96
4.3.4 GossipKit’s Event Engine . . . 98
4.3.4.1 Event Sources . . . 101
4.3.4.2 Event Data . . . 104
4.3.4.3 The Event Engine . . . 109
4.3.5 Configuration . . . 114
4.3.6 Runtime Reconfiguration . . . 118
4.4 The WhispersProtocol Specification Language . . . 126
4.4.1 Macro-level Programming . . . 127
4.4.2 Language Primitives . . . 131
4.4.3 Key Features . . . 135
4.5 Mapping Between Whispers Expressions and GossipKit Com-ponentisation . . . 141
4.5.1 Mapping Decisions . . . 142
CONTENTS 4.6 Implementation Concerns . . . 148 4.7 Summary . . . 152 5 Evaluation 153 5.1 Experimental Approach . . . 155 5.2 Primary Criteria . . . 157 5.2.1 Simplicity . . . 157 5.2.2 Reusability . . . 160 5.2.3 Configurability . . . 162 5.2.4 Reconfigurability . . . 166
5.2.5 Discussion of the Primary Evaluation . . . 170
5.3 Performance and Overhead . . . 175
5.3.1 Component Invocation Overhead . . . 175
5.3.2 Reconfiguration Process Overhead . . . 177
5.3.3 Memory Usage . . . 179
5.3.4 Discussion of the Performance Evaluation . . . 182
5.4 Summary . . . 184 6 Conclusion 185 6.1 Summary . . . 185 6.2 Major Results . . . 188 6.3 Future Work . . . 191 6.4 Concluding Remarks . . . 194
A The Structure of GossipKit’s XML Configuration 196
B GossipKit’s XML Configuration File for the RPS Protocol 198
C Whispers Grammar 201
D Whispers Programs 206
E Java Implementation of the Random Peer Sampling Protocol 211
2.1 Compositional gossip example . . . 45
3.1 OpenCom Component [Grace (2004)] . . . 56
3.2 Control-Forward-State Architectural Pattern [Grace et al. (2004)] 60 3.3 Event-driven Architecture [Bhatti et al. (1998)] . . . 63
4.1 Development process . . . 87
4.2 Overview of the GossipKit framework . . . 90
4.3 GossipKit Common Architectural Pattern . . . 92
4.4 Compose two different components to realise a more complexPeer Selection service . . . 99
4.5 Use nested events to realise push-pull gossip in GossipKit . . . . 107
4.6 GossipKit’s Event Engine . . . 109
4.7 Use case study: configuring the RPS protocol . . . 118
4.8 Per-node program of RPS . . . 148
4.9 Component realisation of RPS . . . 148
4.10 A node running the Averaging application initially holds value 0 . 150 4.11 The node successfully estimated the system size is 10 . . . 151
5.1 SCAMP uses the reactive push pattern
163
5.2 Configuration of RPS that uses periodic push-pull
163
5.3 Customised configurations based on periodic push-pull to realised T-Man, Averaging, and Ordered Slicing
LIST OF FIGURES
5.4 Anti Entropy uses the lazy push gossip pattern for data consis-tency
163
5.5 Probabilistic broadcasting protocols used in mobile adhoc networks 163
5.6 Initial random graph maintained by RPS . . . 168
5.7 5th rounds since 1st reconfiguration
168
5.8 Ring constructed at the 11th round . . . 168
5.9 Topology at the 20th round . . . 168
5.10 Grid constructed at the 23rd round . . . 168
5.11 Number of gossip rounds used to achieve ring topology. Gossip period is every 5 seconds. . . 169
5.12 Time length (measured in seconds) used to achieve ring topology. 169
5.13 Number of gossip rounds used to achieve grid topology. Gossip period is every 5 seconds. . . 170
5.14 Time length (measured in seconds) used to achieve grid topology 170
2.1 Reactive Push Pattern . . . 31
2.2 Periodic Push Pattern . . . 33
2.3 Periodic Pull Pattern . . . 33
2.4 Periodic Push-pull Pattern . . . 34
2.5 Lazy Push Pattern . . . 35
2.6 Decision Based Broadcast Pattern . . . 35
2.7 Sleep Based Broadcast Pattern . . . 36
2.8 A categorisation of gossip protocols . . . 38
2.9 Three main types of interactions . . . 40
3.1 The Control-Forward-State Pattern captures distributed algorithms in three different application domains . . . 61
4.1 Fields in a GossipKitevent . . . 104
4.2 The structure of the Data Content field in GossipKit events . . 105
4.3 Summary of Whispers’ language primitives . . . 132
5.1 The implemented gossip overlays overs all the gossip patterns . . 156
5.2 The numbers of lines of code to implement the eight gossip proto-cols by using transparent componentisation, Java, and component configuration . . . 158
5.3 The complexity to implement the eight gossip protocols by using transparent componentisation, Java, and component configuration 159 5.4 Components reused in the development of 8 gossip systems . . . . 160
5.5 The number of lines of code in the reusable components and the specific components of each gossip system . . . 161
LIST OF TABLES
5.6 The time used for reconfiguration . . . 178
5.7 Byte code size of GossipKit components . . . 178
5.8 Byte code size of the eight gossip protocols. The byte code size of the four composite protocols include the size of RPS . . . 179
5.9 GossipKitcomponent size measured as the byte code size of the compiled Java class files . . . 180
5.10 Dynamic memory usage of the eight gossip protocols. The mea-surements of the two gossip protocols that run on wireless adhoc networks, Gossip1 and Gossip2, do not include the Jist/SWANS simulator . . . 181
Introduction
Distributed computing is increasingly pervading all aspects of daily life, signifi-cantly affecting human communication, business processes, and the way scientific experiments are carried out. For instance, Skype allows millions of users to make video calls and share files with other Skype users anywhere in the world; Amazon and Google infrastructures provide remote computing services and capacities (e.g virtual servers, data processing, storage) to individuals and businesses; and sci-entific testbeds such as PlanetLab [Peterson et Roscoe(2006)] and Weevil [Wang
et al. (2005)] support research experiments that require large or distributed com-putation resources.
As a result of this continuous increase of distributed applications, the complex-ity of contemporary distributed systems is rapidly growing in three dimensions Feiler et al.(2006). These systems are oftenheterogenous, involving computation nodes with varying capacities and operating over an increasing range of
network-ing technologies (e.g. fixed networks, mobile ad hoc networks, satellite links, etc.) that differ in their bandwidth and latency. This complexity is further com-pounded by highly dynamic network environments: nodes may be mobile; new nodes may join the network at any time; and existing nodes may leave, either voluntarily or by failures. Finally, contemporary distributed systems often grow dynamically to alarge scale, involving very large numbers of participating nodes that are deployed over large areas (e.g. Internet-based applications such as Skype and BitTorrent, large sensor systems for environmental monitoring [Sheldonet al.
(2005); Popaet al. (2005); Howard et Flikkema (2008)]).
Gossip-based systems have been proposed as one of a number of key tech-niques in handling this complexity of contemporary distributed systems. These gossip-based systems are adaptive to node and network heterogeneity, possess self-organising properties to cope with dynamic environments, and offer scalable communication in potentially large-scale networks. However, in spite of these advantages, this thesis argues that the lack of development support is one of the main reasons that prevent gossip-based systems from being widely adopted in practice. As section 1.1 will argue, existing gossip-based systems lack develop-ment support in four main aspects: 1) the developdevelop-ment process of these systems can involve complex issues such as design choices, networking idiosyncrasies, and protocol composition; 2) most gossip-based protocols/systems have been devel-oped in a one-off and per-protocol fashion, making them hard to reuse; 3) they
cannot be flexibly configured to provide customised systems for specific require-ments or operating environrequire-ments; and finally, 4) they provide limited support on runtime reconfiguration of system behaviours, which is an essential property for long running or adaptive systems. To facilitate the development of gossip-based systems, this thesis investigates new design patterns and architectural principles that can be integrated into a common platform to simplify the programming of gossip systems, promote code reuse, and ease systemconfiguration and reconfig-uration. Towards this aim, this thesis describes the design, implementation, and evaluation of a novel programming framework that combines the strengths of a high-level protocol specification language and a component framework, based on the survey of a variety of gossip-based protocols and software mechanisms that are potentially beneficial for developing gossip-based systems. In addition to its main objective, this thesis hopes that the lessons learnt from the building of the programming framework of gossip-based systems shed light on the development support of a wider range of contemporary distributed systems in general.
1.1
Gossip-based Systems
Gossip protocols (also known as epidemic protocols) [Kermarrec et van Steen (2007);Friedman et al. (2007)] have emerged as a promising approach to address some of the problems of contemporary distributed systems. In a typical gossip
1.1 Gossip-based Systems
protocol each node of a network randomly communicates with a small number of their peers, causing information to rapidly spread over the network in the way a rumour is gossiped amongst a group of people or a disease epidemic spreads over a population. Compared with more traditional systems, gossip-based approaches offer several advantages: i) many gossip algorithms [Frey et al. (2009); Haas
et al.(2002); Nedoset al. (2007)] are aware of node and network heterogeneities, and are able to adapt the behaviours of individual nodes to cope with these heterogeneities; ii) gossip protocols areself-organising in dynamic networks since their randomised communication allows multiple routes to be explored in case of failures and they do not require centralised coordination; and finally, iii) because each node performs a limited set of operations at a fixed rate, they provide scalable communication in large distributed systems. Because of these benefits, gossip protocols have been considered as a promising approach for a wide range of services such as ad hoc routing [Haaset al. (2002)], multimedia streaming [Liu et Zhou (2006)], replicated database consistency [Demers et al. (1987); Holliday
et al. (2003)], information dissemination [Birman et al. (1999); Chandra et al.
(2001); Luo et al. (2003)], data aggregation [Kempe et al. (2003); Jelasity et al.
(2005);Guptaet al.(2001)], topology construction [Jelasity et Babaoglu(2005)], and peer sampling [Voulgaris et al. (2005); Jelasityet al. (2007)].
This thesis argues that developing gossip-based applications can be a complex task because of insufficient software development support and, as a result, most
implementations so far have been focused on prototypes and preliminary deploy-ments. The following discusses four main aspects where software development support is required to facilitate the development of gossip-based systems.
Simplicity The existing development process should be simplified, providing a better understanding of the design space of gossip protocols and hiding low-level programming details from developers. The need of software development support arises from the wide range of gossip protocols that target a diverse set of services that operate on different networking environments that range from fixed networks to mobile adhoc networks (see examples above). These protocols differ in many aspects (communication patterns, state, network requirements and usage), making a large design space that needs to be explored by developers. Furthermore, individual gossip protocols must offer distinct APIs to a variety of applications that use them, and must grapple with the idiosyncrasies of the underlying network they rely on (APIs, churn, resilience, costs, performance, meta-information). Advanced gossip protocols are also often composite, and rely on simpler gossip protocols for part of their functionality.
Reusability Most existing gossip protocols/systems have been developed in an adhoc and per-protocol fashion that does not help to capture their similarities for software reuse. As the number of gossip protocols and the community that research them have grown considerably, the lack of a unifying framework means much development effort is wasted in the implementation of gossip-based systems
1.1 Gossip-based Systems
that involve duplicated communication patterns and local processing algorithms. This repeated effort also hampers the use of standardised implementation, and increases the risk to introduce programming errors into systems.
Configurability Gossip-based systems provide many different services in a diverse range of environments. Thus they often need to be customised for their targeting applications and operating environments. For instance, gossip pro-tocols that run on mobile devices often require a minimal configuration to fit the memory constraints while those running on fixed networks can include rich non-functional elements (e.g. quality of service, fault tolerance, real time require-ments). This fact strongly requires flexible configuration support that allows de-velopers of gossip-based systems to easily compose different customising features into some base gossip algorithms. However, the research community of gossip-based mechanisms has been focused on prototyping novel gossip-gossip-based algorithms and exploring a wider range of possible application domains. These prototype systems are often implemented in a single programming language and then com-piled and linked to a static application, hence resulting in monolithic code that is not configurable. The lack of configurability means developers have to code different customising features in low-level programming languages for individual gossip systems, and hence wastes the implementation effort.
Reconfigurability Most existing gossip-based systems are statically encap-sulated in a self-contained environment, which does not allow system behaviours
to be flexibly reconfigured at runtime apart from a fixed number of parametric changes. This means that these systems are unable to modify their behaviours [Freyet al.(2009);Haaset al.(2002);Nedoset al.(2007)], for instance by actions such as adding new protocols, discarding existing protocols, or changing part of a protocol’s elements. Thus, existing gossip-based systems provide limited capac-ities to adapt to changing conditions such as dynamic networking environments and new application requirements. In practice, this poor support of reconfig-urability prevents gossip-based systems from being adopted for certain types of services such as long running systems with evolving requirements and infrastruc-tures that are difficult or expensive to install new programs on (e.g. a sensor network with nodes scattered in a wild area).
1.2
Software Development Approaches
Several existing software mechanisms are potentially exploitable to support the development of gossip-based systems and distributed systems in general (e.g. component-based technology [Heineman et Councill (2001)], model-driven archi-tecture [Voelter et Schmidt (2006)], aspect-oriented development [Kiczales et al.
(1997)], domain specific language [Mernik et al. (2005)]). These mechanisms are often integrated into a middleware framework [Emmerich et al.(2007)], providing general guidelines and tools for building specific middlewares and distributed
ap-1.2 Software Development Approaches
plications. Amongst these mechanisms, component-based technology and domain specific language are particularly relevant in the context of this thesis.
A component-based technology specifies a system as a set of components and their interactions. It allows these components to be developed independently and then assembled together to form a concrete software system according to some domain specific rules. A component architecture is often described in a general purpose or a domain specific architectural description language, which provides declarativeexpressions to formally represent component-based systems in terms of components, connectors, and composition rules. Component-based technologies provide several benefits for developing distributed systems, including those based on gossip: i) components are reusable in the composition of different systems; ii) component mechanisms ease system (re)configuration, because components can be flexibly composed to provide customised systems and advanced component frameworks often allow their architectures to be modified at runtime to adapt to environment/requirement changes [Coulsonet al. (2004);Brunetonet al.(2006)]; and iii) component configurations provide high-level abstractions of software sys-tem architectures in terms of components and connections, hence facilitating exploratory design and architectural analysis. Because of these advantages, com-ponents have been successfully applied in the industry with Enterprise JavaBeans (EJB), the CORBA Component Model (CCM), and Microsoft DCOM, and have found a strong following in the research community, giving rise to a number of
lightweight component frameworks (e.g. OpenCom [Coulson et al.(2004)], Frac-tal [Bruneton et al. (2006)]) and their associated middleware frameworks (e.g. GridKit [Graceet al. (2004)], RAPIDWare [McKinley et al. (2001)]).
In practice, coarse-grained components are relatively less flexible and reusable in the construction of component-based systems [Mehta et Heineman (2002)], comparing with fine-grained components. On the other hand, fine-grained compo-nent systems often involve a large number of compocompo-nents and connectors [Steppe
et al.(2004);Edwardset al.(2004)], making the architectural description of com-ponent systems a tedious and error-prone task. Furthermore, because compo-nent architectural descriptions are by nature declarative and focus on structures rather than algorithms [Clements (1996)], the low-level implementation details of individual components still need to be described in a general purpose program-ming language such as Java or C. This in turn hampers the understandability of component systems, preventing developers who are unfamiliar with a particular framework from understanding component configurations if they do not dive into the specifics of each component.
In contrast to component frameworks, protocol specification language is a spe-cific class of domain spespe-cific languages that focuses explicitly on the algorithmic logic of distributed systems (e.g. Lotos [van Eijk et Diaz(1989)], Estelle [Amer et C¸ e¸celi(1990)], PLAN-P [Thibaultet al. (1998)], Promela++ [Basuet al.(1997)], Mace [Killianet al.(2007)]), rather than on their compositional structures. These
1.3 Hypothesis and Approach
protocol specification languages provide simple and high-level expressions to cap-ture the logical behaviours of distributed and parallel algorithms. Compared to components, protocol specification languages are less able to support dynamic adaptive systems. Because they do not explicitly expose fine-grained dependen-cies between different program parts, they make it more difficult to reason about and implement dynamic changes. As a result, they often require adaptive be-haviours to be statically hard-wired prior to system deployment, or a complete new version to be installed to replace an existing one.
1.3
Hypothesis and Approach
Contemporary distributed systems such as those based on gossiping mechanisms involve an increasing complexity in terms of heterogeneity, dynamism, and large scale. Following the rapid emergence of these complex systems, it is increasingly important to provide new software mechanisms to support their development. Based on this observation, this thesis aims to offer new design patterns and ar-chitectural principles to facilitate the development of contemporary distributed systems. To narrow down this broad scope of research problem, this work fo-cuses on the development support for gossip-based systems — a representative subclass of contemporary distributed systems. More specifically, it aims to ad-dress the existing four problems that are associated with developing gossip-based
systems (see section 1.1): simplifying the programming effort, promoting code reuse, supporting configurable system development, and runtime reconfiguration. Considering these four objectives and the potential software development ap-proaches to achieve them, both component frameworks and high-level protocol specification languages seem to be highly complementary for developing contem-porary distributed systems such as gossip-based systems. Component-based tech-nology seems a promising solution to improve reusability and (re)configurability while high-level protocol specification language provides abstractions of system behaviours to simplify the programming and the understanding of distributed algorithms (see section1.2 above). However, these two approaches are difficult to combine: bringing components to high-level protocol specification languages tend to undermine their programmatic simplicity by forcing developers to navigate back and forth between structural and behavioural concerns. This tension be-comes worse with finer structural decomposition which jeopardises this approach for fine-grained adaptation. Symmetrically, adding a behavioural dimension to component assembly makes composition harder to grasp and manipulate by de-velopers, canceling the very benefits it should bring. As a result, existing systems often choose the best one of these technologies for their development, prevent-ing themselves from offerprevent-ing reusability, (re)configurability, and simplicity at the same time.
program-1.3 Hypothesis and Approach
ming framework that combines the strengths of both techniques. More precisely, this work advocates a hybrid framework that consists of two layers: 1) a high-level protocol specification language that allows developers to focus solely on describing the logical behaviours of gossip algorithms, and supports automat-ical transformation of these algorithmic descriptions into software components and their composition rules, and 2) a underlying component framework that pro-motes component reuse and supports (re)configuration based on the components and composition rules generated by the higher-level language layer.
The main hypothesis of this thesis is that the application of this hybrid ap-proach can facilitate the development of gossip-based systems in terms of sim-plicity, reusability, configurability, and reconfigurability. More specifically and in order to test this hypothesis, this thesis aims to achieve the following goals.
• Identify an overall architectural pattern that captures the key behaviours of gossip protocols as components and their interactions. This process is based on the study of a variety of existing gossip algorithms in the literature.
• Capture the commonalities and variabilities of gossip protocols in a compo-nent framework. Implement these common and variable parts as software components that can be reused in the development of different gossip pro-tocols/systems.
as-sembled from individual software components according to those component interactions identified in the overall architectural pattern. Furthermore, this thesis looks for a reconfiguration mechanism that can dynamically modify the behaviours of gossiping nodes across a network. This reconfiguration mechanism should be scalable and be tolerant to dynamic networking en-vironments that involve transient node and network failures.
• Investigate the relationship between component configuration and high-level protocol specification language. More precisely, the components and inter-actions in the common architectural pattern of gossip protocols need to be mapped onto the language constructs of a high-level protocol specifica-tion language. In addispecifica-tion, this language should offer simple abstracspecifica-tions to describe programming details such as message transmissions, network interfaces, protocol composites and data structure and processes.
This thesis does not address a number of important problems that are also related to gossip-based technology. First, this thesis does not aim to provide any new gossip algorithm or discover new application domains that gossip systems can be used. Second, this thesis adopts an experimental approach to system research, therefore the evaluation is based on experiments and quantitative measurements rather than formal proofs. Finally, the security issues of gossip-based systems are not investigated in this thesis.
1.4 Thesis Outline
1.4
Thesis Outline
The following two chapters investigate the state of the art that is in the related areas of research to this thesis. The aim of chapter 2 is two-fold: first, itsurveys gossip protocols to a wide extent, looking at their fundamental communication patterns, their key behaviours, and how they interface with their upper layer applications; and second, based on this detailed survey, it analyses the problem space that is related to software support of gossip-based systems and explains why this thesis decides to focus on the reusability, (re)configurability, and sim-plicity. Chapter3discusses thesoftware engineering approaches and architectures that are potentially applicable for gossip programming, examining closely their strengths as well as weaknesses.
Chapter 4 first discusses the general principle that is applicable to the broad research question of this work — how best to provide software development sup-port for a wide variety of contemporary distributed systems. It then presents the proposed solution of this work to achieve the specific research goals — a hybrid programming framework that combines the strengths of both high-level proto-col specification languages and componentisation to support the development of gossip-based systems.
Chapter 5 focuses on the evaluation of the implemented framework. It de-scribes the experimental setup and results, and analyses in detail how each of the
research problems identified in chapter 2 is addressed with these results.
Finally, chapter6summaries the major results and contributions of this thesis, and suggests areas with potential for future work.
1.5
Associated Publications
Subsets of my work towards this thesis have been published in international con-ferences and workshops as follows:
• Shen Lin, Fran¸cois Ta¨ıani, Marin Bertier, Gordon S. Blair, Anne-Marie Ker-marrec,Transparent Componentisation: High-level (Re)configurable Programming for Evolving Distributed Systems, accepted by the Dependable and Adaptive Distributed Systems track of the 26th Sympo-sium on Applied Computing, March 2011, Taichung, Taiwan.
• Shen Lin, Fran¸cois Ta¨ıani, Gordon S. Blair, Facilitating Gossip Pro-gramming with the GossipKit Framework, In Proceedings of the 8th IFIP International Conference on Distributed Applications and Interoper-able Systems, pp. 238-252, June 2008, Oslo, Norway.
• Shen Lin, Fran¸cois Ta¨ıani, Gordon S. Blair, GossipKit: A Framework of Gossip Protocol Family, In Proceedings of the 5th International Workshop on Middleware for Network Eccentric and Mobile Applications,
1.5 Associated Publications
11-12 September 2007, Magdeburg, Germany.
Finally, although not central to this thesis, the following publication demon-strates an interesting application of the middleware framework reported in the thesis. The complete publication is included in Appendix A.
• Shen Lin, Fran¸cois Ta¨ıani, Gordon S. Blair, Exploiting Synergies Be-tween Coexisting Overlays, In Proceedings of the 9th IFIP Interna-tional Conference on Distributed Applications and Interoperable Systems, pp. 1-15, 9-11 June 2009, Lisbon, Portugal.
Gossip Protocols
Domain analysis is often the first phase in the software development process of a specific domain [Hjørland et Albrechtsen (1995)]. A domain analysis provides several benefits for the subsequent design and implementation of a software de-velopment framework: i) it helps to better understand and model the domain; ii) it encourages systematic software reuse by identifying the recurring elements in a software domain; and iii) it captures the key elements of a software sys-tem and identifies the interactions between these elements , hence facilitating the development of (re)configuration mechanisms.
In the case of this thesis, domain analysis is applied to the gossip protocols to support systematic design and implementation of a programming framework for gossip-based systems. More specifically, this chapter explores the design space of gossip-based protocols, identifying the common and the variable parts of a typical system in the gossip family. It starts with an introduction to gossip
2.1 Introduction to Gossip Protocols
protocols with reference to typical examples (section 2.1). This introduction is followed by a detailed survey of gossip protocols, capturing a finite set of recurring elements, architectural patterns, and application contexts that are involved in the design of a gossip protocol (section 2.2). This survey involves the study of over 30 gossip protocols from the literature, which makes a representative sample of the gossip family. During the survey of gossip protocols, this work also notices that multiple gossip protocols can coexist on a single node in practice, running independently to provide distinct services or collaborating with each other. These coexisting gossip protocols are presented in section2.3, together with a discussion on the potential challenges that are associated with developing gossip systems that comprise multiple coexisting gossip protocols. Finally, section 2.4 discusses the particular issues of gossip-based development that this thesis focuses on.
2.1
Introduction to Gossip Protocols
Gossip-based algorithms have been applied to develop distributed systems for more than 20 years. In particular, these gossip approaches have become extremely popular in the past decade and given rise to a number of gossip protocols. In order to explore the design space of gossip protocols, the work described in this thesis has studied over 30 gossip protocols from the literature. The following dis-cusses these gossip protocols from a historical perspective with reference to some
representative examples. First, it discusses the early gossip protocols that were applied to propagate information, and associated the scalability of gossip com-munication with the underlying epidemic theory (section 2.1.1). Because there lacked a scalable membership service for these early gossip protocols to select random peers from the network group, gossip-based peer sampling protocols are developed (section 2.1.2). These peer sampling protocols illustrated the conver-gence property of gossiping, which opened a broader scope of distributed services that gossip protocols can be applied to (section 2.1.3). More recently, gossip-based approaches have been used to operate on mobile adhoc networks (section
2.1.4).
2.1.1
Gossip for Disseminating Information
The underlying idea of gossip dates back to the work at Xerox Research Cen-ter, which applied gossip-based communication to maintain wide-area database systems in the Clearinghouse project [Demers et al. (1987)]. In this project, a database was replicated at several hundred or thousand nodes in a fully-connected, heterogeneous, slightly unreliable, and slowly changing network. This large and complex environment made it difficult to maintain consistent data amongst all the database replicas. To address this issue, this project required algorithms that can propagate database updates at some nodes to all the other
2.1 Introduction to Gossip Protocols
nodes in an efficient and robust way as well as to scale gracefully as the number of nodes increases.
The algorithm proposed by Xerox Research Center was a two-phase protocol: an efficient unreliable broadcast (e.g. UDP broadcast) is used to propagate the updates to as many nodes in the network as possible; and running in the back-ground, an anti-entropy algorithm is used to repair the nodes that failed to receive the unreliable broadcasting messages. In this anti-entropy algorithm, every node randomly chooses another node at each fixed time interval t, and then sends its database contents to this random node to resolve the differences between the two nodes. Demers et al. (1987) showed this anti-entropy algorithm is essentially re-lated to the mathematic theory that studies the propagation of epidemics [Pittel (1987)]. The nodes that hold new database contents are similar to the population that are infected by an epidemic disease, and anti-entropy is one of the simple epidemic behaviours that allows the epidemic to eventually spread over the en-tire population. The epidemic theory [Pittel (1987)] also provides the worst case scenario of anti-entropy’s infection speed in the case of fully-connected topology: when starting with a single infected node, the average number of time intervals (N(t)) that are required to infect the entire population is:
for large node numbern. This logarithmic infection speed makes the anti-entropy algorithm a scalable approach to disseminate information over large networks. Apart from this scalable propagation speed, the anti-entropy protocol illustrates the two key benefits of gossip protocols. First, it involves repeated communication with nodes that are randomly selected with a uniform probability, and hence allows multiple routes to be explored to avoid network failures. Second, because it consumes a fixed amount of local and network overheads on each node, this algorithm is scalable to large networks.
However, Xerox’s anti-entropy protocol is only tolerant to network failures, but not to node failures. Furthermore, because the Xerox work assumes database contents are updated at a low frequency (at most a few per second), it did not consider real-time applications that require continuous delivery of multimedia up-dates to all nodes (e.g. Internet radio, TV or conference). In spite of these limi-tations, Xerox’s gossip-based approach opened new possibilities for implementing scalable applications, and it motivated several subsequent research works that im-proved on Xerox’s original anti-entropy algorithm for database consistency (e.g. [Hollidayet al.(2003)]) and reliable multicast (e.g. pbcast [Birmanet al.(1999)], lpcast [Eugster et al. (2003)]). Amongst these works, Hollidayet al. (2003) used gossip to propagate transaction records to prevent concurrent database transac-tions from accessing uncommitted data; pbcast [Birman et al. (1999)] proposed additional controls to tolerate node failures and higher throughput of updates
2.1 Introduction to Gossip Protocols
than Xerox considered, making gossip a possible approach for multicast services that have real-time requirements; and lpcast [Eugster et al. (2003)] suggested scalable message buffering for gossip-based multicast.
2.1.2
Peer Sampling Service
The above mentioned gossip algorithms assume that each node knows all other participating nodes in order to select a random subset of nodes to gossip with. This model requires a synchronised view of membership on all nodes and in-creasing memory capacity as the system grows, which is inherently unscalable. This problem motivated the development of scalable membership services such as SCAMP [Ganesh et al. (2001)] and RPS (random peer sampling) [Jelasity et al.
(2007)] that allow gossip-based systems to operate on dynamic networks that involve churns. SCAMP allows each newly joined node to propagate a gossip message that contains its contact information to existing members. On receipt of the gossip message, every node either adds the contact information of the new node to its view or forward the message to members in its view based on some probabilistic decisions. As a result, SCAMP maintains a static graph where each node has a view that contains the contact information of log2(n) random nodes
for group size n, and it reactively re-balances the graph every time a new node joins the group.
However, SCAMP is less robust in dynamic systems where nodes join and leave frequently [Jelasity et al. (2007)]. To address this issue, RPS uses periodic gossip to ensure that each node maintains a fresh view of random members. In RPS, each node maintains a random sample of the node population. This sample contains the contact information of C random peers in the system, where C is a small constant number. At each gossip round, each node n selects one random peer i from its sample and sends a copy of its sample to i. On receipt of the sample from node n, i immediately replies with a copy of its sample and carries out an update process on its local sample. In this process, Node i first merges
n’s sample with its own to form a sample with size 2C, and then discards C
random peers in the sample to obtain an updated sample with size C. Finally, on receipt of i’s sample, node n also carries out the same update process on its sample. By executing the RPS algorithm, each node obtains a fresh sample that provides the contact information ofC different random nodes for selection at each gossip rounds. To join the network, a node adds the contact information of itself and of an existing random node in the system to start the periodic execution of the main RPS algorithm. The periodic behaviour makes RPS more robust to churns as the experimental result in [Jelasity et al. (2007)] demonstrates, but it repeatedly generates a potentially large amount of network messages.
2.1 Introduction to Gossip Protocols
2.1.3
Convergence of Properties
The RPS algorithm illustrates theconvergencefeature of gossip-based approaches: unlike gossip-based dissemination algorithms (e.g. Xerox’s work, pbcast) that use gossip to spread the same information to all nodes, RPS requires individ-ual nodes to periodically exchange their local properties (in this case local views of the global membership) with some random peers, and as a result eventually brings these local views to converge to a particular global property (in this case a balanced random graph where each node contains a uniformly random sample of the membership). Jelasity et al. (2007) also provided experimental results to demonstrate that this convergence feature scales to distributed applications that involve very large number of nodes, demonstrating an average speed ofO(log2(n))
gossip rounds to reach global convergence for large node number n. This scalable convergence feature, together with the scalable membership service provided by RPS, significantly broadens the scope of application domains that gossip-based mechanisms can be employed for. To name a few examples, T-Man [Jelasity et Babaoglu (2005)] creates and maintains various structured overlay topologies (e.g. ring, grid, cluster) from an initially random graph. The convergence fea-ture of gossip-based communication has also been exploited to aggregate various converged global knowledge such as the maximum, minimum or average of local data [Kempe et al. (2003); Jelasity et al. (2005); Gupta et al. (2001)]. Finally,
ordered slicing [Jelasity et Kermarrec(2006)] allows nodes in a distributed system to converge to partitioned groups that are ordered with respect to a particular measurable property (e.g. bandwidth, workload).
2.1.4
Gossip on Mobile Adhoc Networks
More recently, gossip-based approaches have also been adopted to provide scalable and efficient communication in mobile adhoc networks [Haas et al. (2002); Luo
et al.(2003);Sassonet al.(2003);Houet al.(2006,2005)]. Mobile adhoc networks significantly differ from IP-based networks on several aspects: 1) communication in mobile adhoc networks often relies on radio broadcast, which allows a broadcast message to be received by all the nodes within the radio broadcast range (i.e. physical neighbours); 2) nodes are less robust, and often have limited resources such as memory, processing capacity, and energy power; 3) links are less reliable because of dynamic factors such as mobility, lack of collision detection, and faulty nodes; and 4) it is more expensive to establish point-to-point routes with distant nodes because of dynamic networking environments and unreliable links. Because of these differences, it is not practical to use gossip algorithms that rely on random peer selection in mobile adhoc networks [Friedman et al. (2007)].
Instead, most gossip algorithms in mobile adhoc networks focus on efficient dissemination of broadcasting message to all or almost all nodes. Traditionally,
2.1 Introduction to Gossip Protocols
message dissemination in mobile adhoc networks is based on certain forms of flooding, whereby every node forwards the received broadcast message to its physical neighbours. Because a single radio broadcast can be heard by all the physical neighbours, flooding often involves many unnecessary broadcasts, and hence wastes energy and generates heavy network overhead. Instead of forwarding all the received messages, gossip-based algorithms in mobile adhoc networks only forward broadcast messages with a certain probability. For instance, one of the gossip protocols proposed in [Haaset al.(2002)] suggested that gossip probability of each node should be proportional to the number of physical neighbours; Smart Gossip [Kyasanur et al. (2006)] detects nodes that are on the critical paths of the network, and then informs these nodes to broadcast with higher probability; and the gossip-based sleep protocol (GSP) [Houet al.(2006)] instructs individual nodes to enter sleep mode with certain probability so that they do not correspond to the incoming messages for a short period.
The experiments of these probabilistic gossip algorithms have shown that they effectively reduce the number of broadcasts in a dissemination while achieving almost the same message delivery as flooding. This is because the probabilis-tic gossip exhibits a bimodal behaviour as predicted by the percolation theory [Meester et Roy (1996)]: as soon as the gossip probability p meets some mini-mum threshold, a broadcast message will be received by all or almost all nodes with a very high probability.
2.2
Gossip Patterns
This section explores the design space of gossip protocols. First, it analyses the recurring elements that are involved in gossip protocols, discussing the possible design choices for each element and their potential benefits and limitations (sec-tion 2.2.1). Based on this analysis, section 2.2.2 presents five recurring architec-tural patterns that are formed by these elements. Finally and in addition to the analysis of gossip protocols’ internal behaviours such as their key elements and architectural patterns, section 2.2.3 examines the contextual issues that might affect the gossip design.
2.2.1
Underlying Elements
Based on the study of over 30 gossip protocols/systems, this work observes several recurring elements that are used in all the gossip algorithms: 1) a gossip message is either triggered reactively or periodically; 2) gossip communication involve three basic styles of data flows (i.e. pull, push, push-pull); and 3) all the gos-sip algorithms use certain forms of randomised communication, as the following discusses.
Gossip TriggerGossip protocols can executeperiodically (e.g. RPS) orreact to external events (e.g. SCAMP). Periodic gossip effectively avoids possible traffic congestion. This is because gossip rounds are not synchronised and many periodic
2.2 Gossip Patterns
gossip algorithms allow individual nodes to send gossip messages at a random time within the time interval of a gossip round, thus preventing nodes from generating outgoing messages at the same time. But periodic gossip propagates information more slowly because it does not forward messages immediately on reception. In contrast, reactive gossip allows new information to quickly spread over a network at a cost of generating a large amount of network traffic within a short period.
Direction of Data FlowsThis survey has observed three basic styles of data flows that can either be used along or in combination with each other to form various types of gossip-based communication. First of all,push-pull is a pair-wise exchange of data. This exchange diffuses the data on both nodes at a single gossip round, and hence foster rapid convergence of local data to a converged global state.
In a push style gossip, each node sends its data to some random peers but does not require replies from these peers. As a result, push style gossip only uses half of the number of messages compared with pair-wise exchange, but requires longer time when it is used to achieve convergence or aggregation. Push style gossip is more suitable for disseminating information because nodes do not need to reply to senders, and are only required to forward the information to other nodes.
In apull style gossip, a node queries some random peers for their data. This gossip style ensures that the gossip data is only transferred if there is an actual
demand. It helps to effectively reduce the network overhead if the size of the gossip data is considerably large.
Style of Randomisation Gossip-based algorithms involve certain forms of randomised communication. The following presents three common styles of ran-domisation that are often used in the existing gossip algorithms.
1. Uniform Randomness In fixed networks, the randomness is achieved through point-to-point communication with randomly selected peers. In some gossip algorithms that operate on fixed networks, communication targets are randomly selected with a uniform probability. This uniform randomness offers equal opportunities for each node to propagate its local information and receive information from others, and hence optimises the average speed to global convergence and data aggregation.
2. Biased Randomness Some other gossip algorithms on fixed networks do not use a uniformly random selection of peers, but select peers based on biased randomness. For instance, directional gossip [Lin et Marzullo(1999)] takes account of the topology of wide area networks that are formed by several local networks. It assigns a higher probability to peers that are not well connected (e.g. nodes in different local networks) than those that are better connected (i.e. neighbours in the same local network).
2.2 Gossip Patterns
is a common form of randomised gossip communication. More specifically, the probabilities that individual nodes decide to broadcast are often differ-ent: the gossip probability might depend on the number of physical neigh-bours, the number of hops that the gossip message has been forwarded [Haas
et al. (2002)], the energy consumption [Hou et al. (2006)] or the network traffic pattern [Hou et al. (2005)]. Node mobility also introduces random-ness for gossip communication in mobile adhoc networks. Using nodes that carry information while moving randomly at different physical locations, mobility can help to spread information and possibly reconnect partitioned networks [Friedman et al. (2007)].
2.2.2
Key Patterns
The previous section analysed the design choices of the key elements that are often involved in developing gossip protocols. Furthermore, the author records the design choice of each individual key element for the 30 gossip protocols reviewed in this thesis. As a result, the author identifies five common combinations of these key elements — termed gossip patterns — into which these 30 gossip protocols can be allocated. For each gossip pattern, the following (i) describes its general behaviour in terms of its design choices of key elements, (ii) lists the gossip protocols that are allocated into this gossip pattern, and (iii) illustrates a concrete
example.
Pattern 1. Reactive Push As shown in table 2.1, this gossip pattern is triggered on receipt of a gossip message from other nodes (i.e. reactive trigger), and it uses “push” style communication to send its data or forward a message to some “randomly selected peers”.
More precisely, a source node s disseminates a message to a set of nodes N
that it knows. For each node n ∈ N, as soon as n receives the message from s, it immediately sends the message to some randomly selected nodes if it has not seen the message before. Reactive push has been proposed in several research works to propagate information on fixed networks, either being used along or in combination with other types of gossip communication (see the protocol list in table 2.1).
Trigger Direction of Data Flow Randomness
Elements Reactive Periodic Push Pull Peer Selection Prob. Decision
X X X
SCAMP [Ganeshet al.(2001)], Anonymous Gossip [Chandraet al.(2001)],
Protocols Directional Gossip [Lin et Marzullo(1999)], PlumTree [Leitaoet al.(2007a)],
Unstructured Epidemic Multicast [Carvalhoet al.(2007)],
Spatial Gossip [Kempeet al.(2001)]
Table 2.1: Reactive Push Pattern
Example: As introduced in section 2.1.2, SCAMP [Ganesh et al. (2001)] is a gossip protocol that uses reactive push to balance the network connectivity so that each node maintains a view of logN evenly distributed random nodes. In this protocol, each node n listens to requests from new nodes that want to join
2.2 Gossip Patterns
the gossiping group. On receipt of a join request from node i, node n forwards the request to all the nodes that it is connected to. On receipt of a forwarded request, a node j adds nodei to its view with probability p. If iis not added to the view, nodej forwardsi’s join request to a random node in its view. Thus, the SCAMP protocol propagates the notification of a newly joined node and allows the new node to be added to the views of some random nodes, which in turn helps to balance the random graph every time a node joins the network.
Pattern 2. Periodic Dissemination This gossip pattern includes three sub-patterns (see period push, period pull, and period push-pull in table refPe-riodicPushPattern , table refPeriodicPullPattern, and table refPeriodicPushpull-Pattern, respectively). All these sub-patterns involve periodic message dissemi-nation and random peer selection. They only differ in the direction of data flows in the system. During each gossip round, each node selects a fixed number of random peers, and then disseminates its information to these peers (i.e. push), requests information from them (i.e. pull), or both (i.e push-pull). Periodic data dissemination allows nodes to obtain the data of others. Gossip algorithms based on this pattern are often used to achieve convergence of some global properties or to aggregate data on fixed networks.
Example: The averaging [Jelasityet al.(2005)] protocol illustrates the use of periodic exchange to aggregate data. At each gossip round, each noden selects a random peer i to exchange their values. On receipt of a valuevn from a random
Trigger Direction of Data Flow Randomness
Elements Reactive Periodic Push Pull Peer Selection Prob. Decision
X X X
G-SDP [Nedoset al.(2007)], G-FDS [Renesseet al.(1998)],
Protocols RDG [Luoet al.(2003)], GSGC [Guoet al.(1997)],
Neem [Pereiraet al.(2003)], Gravitational Gossip [Jenkinset al.(2001)] Table 2.2: Periodic Push Pattern
Trigger Direction of Data Flow Randomness
Elements Reactive Periodic Push Pull Peer Selection Prob. Decision
X X X
Protocols Neem [Pereiraet al.(2003)], K-Walker [Simontonet al.(2006)] Table 2.3: Periodic Pull Pattern
node n, each node i updates its value vi to be the average of vn and vi. This
protocol gradually reduces the variance of the values stored on individual nodes, and eventually allows the value stored on every node to converge to an global av-erage. Furthermore, the averaging protocols requires coordinated communication to ensure that nodes participating in one pair-wise exchange do not simultane-ously exchange data with another node, thus preventing the converged global average drifts away from the actual average. The averaging protocol can be used to estimate the average workload on individual nodes, and hence helps to balance the workload. Another interesting application of this protocol is to estimate the network size. When it is used for size estimation, each node initially holds value 0 except for one node that holds value 1. Eventually, the value v stored on each node will converge to the global average 1
N for network size N. Thus, each local
node can obtain the network size by evaluating the formula 1
v =
1
1
N
2.2 Gossip Patterns
Trigger Direction of Data Flow Randomness
Elements Reactive Periodic Push Pull Peer Selection Prob. Decision
X X X X
Cyclon [Voulgariset al.(2005)], TAG [Liu et Zhou(2006)],
HyParView [Leitaoet al.(2007b)], RPS [Jelasityet al.(2007)],
Protocols T-Man [Jelasity et Babaoglu(2005)], T-Chord [Montresoret al.(2005)],
Averaging [Jelasityet al.(2005)], Ordered Slicing [Jelasity et Kermarrec(2006)],
Hierarchical Gossip [Guptaet al.(2006)], Newscast [Jelasity et al.(2003)] Table 2.4: Periodic Push-pull Pattern
Pattern 3. Periodic Lazy Push Although “periodic lazy push” looks similar to the “periodic push-pull” pattern (see table 2.5), they are different in the way they use the push and pull dissemination. In a lazy push pattern, nodes do not send their data contents directly to other nodes. Instead, each node n
advertises the data it has to some randomly selected peers during each gossip round (i.e. push the advertisements first instead of the actual data content). On receipt of the advertisement from n, node iqueries n for the actual data content if this advertisement is interesting to i, and finally, node n sends its data to i
on receipt of the request (i.e. pull the data content). Lazy push is often used to achieve data consistency amongst a large number of nodes. Several variations of lazy push have been applied to maintain the consistency of replicated database systems and to repair message losses for multicast services on fixed networks.
Example: Pbcast [Birman et al.(1999)] is often used to repair message losses in unreliable multicast systems. Apart from the generic behaviour of lazy push, this anti-entropy protocol also sets limits on the number of requests and transmissions
Trigger Direction of Data Flow Randomness
Elements Reactive Periodic Push Pull Peer Selection Prob. Decision
X X X X
PlumTree [Leitaoet al.(2007a)], lpcast [Eugsteret al.(2003)],
Protocols pbcast [Birmanet al.(1999)],
Xerox database maintenance [Demerset al.(1987)] Table 2.5: Lazy Push Pattern
that a node can send at a gossip round to reduce possible traffic congestion.
Pattern 4. Decision Based BroadcastThis pattern is similar to “reactive push”, but it uses probabilistic broadcast to achieve randomised communication in a mobile wireless network (see table 2.6). To initiate a message dissemination, a node n broadcasts this message to its neighbours. On receipt of this message, each neighbouring node i remains silent if it has seen the message before. Oth-erwise, node i either broadcasts the message or remains silent based on certain probabilistic decisions. This pattern is often observed in gossip algorithms that aim to achieve efficient multicast in mobile adhoc networks (see section 2.1.4).
Trigger Direction of Data Flow Randomness
Elements Reactive Periodic Push Pull Peer Selection Prob. Decision
X X X
Protocols Smart Gossip [Kyasanuret al.(2006)], Polarized Gossip [Beraldi(2008)], GossipN (N= 1,2,3) [Haaset al.(2002)]
Table 2.6: Decision Based Broadcast Pattern
Example: Gossip2 is used to broadcast routing requests over a adhoc network to propagate routing destinations for AODV routing [Perkins et Royer (1997)]. The gossip decision of Gossip2 is based on four parameters: p1, k, p2, and
2.2 Gossip Patterns
n. The parameter p1 indicates the usual probability that Gossip2 forwards a request message. In additional to this basic parameter,Gossip2 also provides two optimisations. First, to prevent a routing request from dying too early, Gossip2 informs that each node forwards a routing request message with probability 1 in the firstk hops. Second, to improve the message delivery rate in spatial networks, each node gossips with probability p2 > p1 if it has less than n neighbours. The simulation results in [Haas et al. (2002)] shows that Gossip2 can help to achieve 99% message delivery and reduce 26% of messages on average when the parameters are set to p1 = 0.6, k= 4, p2 = 1, and n = 6.
Pattern 5. Sleep Based BroadcastInstead of broadcasting with a certain probability, sleep-based broadcast randomly enters the sleep mode to save the battery energy of mobile devices (see table 2.7). More precisely, active nodes forward the received messages to their neighbours by using wireless broadcast in a mobile adhoc network. Each node then randomly enters the sleep mode for a certain length of time with gossip probability p. Nodes in the sleep mode do not react to incoming messages and hence saves energy consumption.
Trigger Direction of Data Flow Randomness
Elements Reactive Periodic Push Pull Peer Selection Prob. Decision
X X X
Protocols GSP [Houet al.(2005)], T-GSP [Houet al.(2006)]
Table 2.7: Sleep Based Broadcast Pattern
are used for energy conservation in mobile adhoc networks. In the GSP protocol, each node generates a random time interval T, and then decides to enter the sleep mode for a certain length of time with probability p or to stay awake for the time interval T. The T-GSP protocol is similar to the GSP protocol except that it requires nodes to stay awake when they are carrying frequent network traffic. This improvement of T-GSP helps to reduce the probability of breaking active communications. Houet al.(2006) has shown that these gossip based sleep protocols can help battery power on nodes to last 25% longer while remains the same delivery rate and transmission latency, compared to those non-gossip based ones such as DSR [Johnson et Maltz (1996)].
Summary The five gossip patterns identified in this section are summarised in Table 2.8. Table 2.8 also allocates the 30 gossip protocols that have been studied in this thesis onto each pattern. Note that some gossip systems involve more than one gossip patterns, hence they appear in the protocol lists of different gossip patterns in Table 2.8.
2.2.3
Other Contextual Issues
In addition to the architectural patterns that capture the gossiping behaviours, the development of gossip systems in practice is affected by their application contexts. The actual application context decides the underlying transport
mech-2.2 Gossip Patterns
Gossip Patterns Example Protocols
SCAMP, Anonymous Gossip, PlumTree,
Reactive Push Unstructured Epidemic Multicast,
Spatial Gossip, Directional Gossip G-SDP, G-FDS, RDG,
Push GSGC, Neem,
Gravitational Gossip Cyclon, TAG, HyParView, Periodic Dissemination Push-pull RPS, T-Man, T-Chord,
Averaging, Ordered Slicing, Hierarchical Gossip, Newscast
Pull K-Walker, Neem
Periodic Lazy Push PlumTree, lpcast, pbcast,
Xerox database maintenance Decision Based Broadcast Smart Gossip, Polarized Gossip,
GossipN (N = 1,2,3)
Sleep Based Broadcast GSP, T-GSP
Table 2.8: A categorisation of gossip protocols
anisms that are available to gossip applications, the style of services that gossip applications provide to higher-level applications, and the non-functional features that are required to customise protocol behaviours for the targeted application requirements or operating environments. This subsection discusses the typical contextual issues that are involved in the development of a gossip system as fol-lows.
Underlying Transport Mechanism Gossip protocols have been proposed to operate on a variety of networking environments to support different appli-cations. These different networks offer many possible transport mechanisms for gossip-based applications: gossip systems that operate on IP-based networks rely
on transport protocols such as TCP and UDP while those operate mobile ad-hoc networks can use one-hop radio broadcast or routing protocols (e.g. AODV routing [Perkins et al. (2003)]) to communicate. Each of these transport proto-cols has its advantages and weaknesses, and should thus be selected according to the actual application. TCP provides reliable and point-to-point communication between gossiping peers and can be used to detect node failures, but comparing with UDP-based communication, it incurs relatively longer latency because it waits for out-of-order messages and retransmits lost messages. UDP is the op-posite of TCP: it is more suitable for timely communication than for accurate delivery of messages, and can be used to broadcast within a local network.
In mobile adhoc networks, as introduced in section 2.1.4, radio broadcast is preferred over AODV routing since point-to-point communication is expensive and unreliable in adhoc networks. This fact makes gossip algorithms based on random peer selection generally unsuitable for real applications in adhoc net-works [Friedman et al. (2007); Chandra et al. (2001)]. Although very few works [Chandra et al. (2001); Luo et al. (2003)] proposed gossip algorithms that use point-to-point communication in adhoc networks, they admit the limitations of the underlying AODV routing mechanism.
Style of Service Many gossip protocols do not operate in isolation with ex-ternal systems and applications. Gossip protocols can either provide services to high-level applications (e.g. membership service by RPS and SCAMP, multicast
2.2 Gossip Patterns
Interaction Type Interaction Details
Gossip Data Access
Read (e.g. Averaging, T-Man)
Write (e.g. Xerox database maintenance, pbcast) Process (e.g. RPS, Averaging, T-Man)
Message Dissemination Control Message (e.g. SCAMP) Reactive Gossip
Parameter Setting
Probability of Broadcast Transmission Limit
Number of Communication Targets ...
Table 2.9: Three main types of interactions
by Gossip2, aggregation of global average by the averaging protocol) or
collabo-rate with each other to form more sophisticated composite systems (see section
2.3 below). Therefore, in addition to supporting the development of gossip sys-tems, it is important to study how the implemented gossip systems can be easily composed or integrated into existing applications. Towards this aim, the follow-ing categorises the types of interaction that gossip protocols are often involved, based on the survey of a number of gossip protocols.
As Table2.9 illustrates, there exists three main types of interactions. First of all, each node that runs gossip protocols often maintains a state of certain forms of data that are communicated amongst peers, and the gossip data in the state often needs to be accessed by other applications. This survey observed three types of operations for accessing gossip data: 1)read operation that is used to retrieve the converged properties (e.g. the global average of individual nodes’ local attributes [Kempeet al. (2003)]) or to deliver messages that are received by gossiping nodes
(e.g. pbcast, lpcast); 2) write operation that adds or updates gossip data that are obtained by external processes (e.g. local temperature that is detected by a heat sensor); and 3) data processing operation processes the data in the state of a gossip protocol, and returns the resulting data to external applications (e.g. RPS returns a fixed number of peers that are randomly selected from its ’peer sample’ state).
Second, despite a number of periodic gossip protocols that repeatedly dissem-inate gossip messages at each gossip round, there exist two type of messages that are disseminated in reaction to external invocations. First, reactive gossip pro-tocols are often used as the underlying broadcasting mechanism of higher-level services such as adhoc routing [Haaset al.(2002)] and distributed sensoring sys-tems [Houet al.(2006,2005)], and they only disseminate gossip-based broadcast messages when they receive requests from the higher-level applications that use them. Second, some gossip protocols are triggered by external applications or end users to send control messages. For instance, nodes in gossip-based member-ship service such as SCAMP need to send messages to indicate their joining or leaving. As a result, a gossip application that involves these two types of message dissemination must provide interfaces for external users and applications to send these messages.
Third, many gossip protocols are parameterised, and they allow these param-eters to be configured statically or at runtime to adapt to the specific application
2.2 Gossip Patterns
requirements and network environments [Sassonet al.(2003)]. This fact requires gossip-based applications to make their parameters accessible by external users or applications via pre-defined interfaces. Because these parameters are application dependent, they vary widely and make it particularly
