Lazy approaches in replicated databases

In [59] it was found that blocking protocols have serious performance degradation when implemented on wide area replicated databases; deadlocks increase as the cube of the number of sites and as the fourth power of transaction size. Lazy approaches try to offer better performance removing the atomic commitment; once a transaction has committed at the originating site, it eventually commits independently at repli- cated sites. In [88], a single-master (or primary site) approach is described; it is able to tolerate site failures but not network partitions. Another lazy protocol that is not based on a single-master approach, is described in [60]. Its approach is what is gen- erally called write-anywhere, since a client can perform write operations on all the replicated databases. Vector clocks [64] are used to order operations and each site stores enough information to have a complete picture of the events in the overall sys- tem. This protocol avoids global deadlocks and reduces delays caused by locking. In order to commit an operation, all sites must be eventually available, but the protocol can be extended using a quorum system to resolve commit decision among a subset of available sites.

Chapter 3

Replication and consistency in

wide area data stores

In distributed systems data are often replicated to several sites in order to improve data availability and fault tolerance, but also to provide users with fast data access. Data are in fact replicated close to the users that need to access them. When replicas can be modified, we need a mechanism to keep them consistent; this is the goal of a synchronisation mechanism, that can be achieved using different techniques. In this chapter we study different synchronisation techniques and their application to several application environments.

In Section 3.1 we review the main concepts of data replication, stressing the dif- ference between static replication and replica synchronisation. In Section 3.2 dif- ferent synchronisation techniques are studied. Since we are especially interested in synchronisation over wide area networks, we focus on optimistic replication, where replicas are kept consistent in a “relaxed way”. In the rest of the chapter we review some practical applications of these concepts. In Section 3.3 we present the SWARM middleware and its composable consistency model, while in Section 3.4 we present the TACT project. We conclude the chapter with a section about Oracle Streams, the Oracle solution for the synchronisation of Oracle databases, that, as we will see in Chapter 4, is extensively used at CERN to replicate Physics data.

3.1 Data Replication

Data replication is a well known strategy to improve performance and reliability of distributed computing platforms. When the users of a system are distributed over a wide area network (WAN), keeping data at a single location can affect data access in three ways:

Latency The data access time varies with the distance and link bandwidth of the user

from the data storage, and it is subject to network problems related to the WAN environment.

Availability Having a single data storage site is a risk for critical applications: when

storage is temporarily unavailable (for faults or maintenance reasons), or the storage site is not reachable due to network problems, users do not have access to data.

Congestion The single data storage site must sustain a potentially high number of

users requests; the hardware used for data storage can be very expensive or fail to satisfy user requests.

For these reasons, in many distributed environments data are replicated, i.e., copied, at different locations. In Figure 3.1 an environment with a single data provider is shown. In Figure 3.2 instead, we show the same system with the data storage repli- cated in three locations.

Data Storage

Wide Area Network

3.1. Data Replication 25

Wide Area Network

Data Storage Data Storage Data Storage

Figure 3.2: Distributed environment with replicated data

In the system that replicates the data store we do not have a single point of failure for data storage: if a storage component fails, users can still have access to the infor- mation through the other two replicas. Users also have data closer to their location, which speeds up data access. Besides, since the load on the storage devices is shared among the replicas, the storage devices can be built using less expensive hardware or, in any case, can better sustain user requests. To summarise, with data replication we achieve: fault tolerance, fast data access and load distribution.

Unfortunately, the benefits before mentioned come at a price. Having different replicas of a same data item, we need a way to locate them, hence saving their location and some metadata information. As we will see in Chapter 5, replica catalogues are used to this purpose, together with a well defined naming scheme. Replicas must also be created and placed according to the system configuration, or based on some dynamic criteria. For example, a replica could be created and saved close to a user just for the time the user needs that data, and then removed from the system. This is what is called dynamic replication, or sometimes replica optimisation. Moreover, we have to face problems of replica coherence, and synchronisation when replicas can be independently modified by users. In particular, replica coherence and replica synchronisation can be two faces of a more general problem, that of maintaining replica consistency.

Generally speaking, replica consistency means that replicated data have the same content. Then, since contents can diverge for different reasons, we distinguish be-

tween the two problems:

• replica coherence: even when replicas are read-only, from the point of view of a

user application, replica coherence can be broken when one of the replicas gets corrupted, either for hardware or software problems. In this case the content of the corrupted replica is different from the one of the other replicas. Replica coherence must be checked also when a new replica is introduced into the system: the content of the replica should be checked in order to ensure that it is a real copy of the replicas already present in the system. It is worth stressing the fact that replica coherence must be checked also in case of read-only replicas.

• replica synchronisation: synchronising replicas in only needed when replicas

are modifiable. When a replica is modified by a user, the other replicas become stale. In order to re-establish consistency among replicas we need to propagate the modification done on the first replica to the others, in a process that is usually called update propagation, the main part of a replica synchronisation process.

Replica coherence can be enforced with periodic checks on replicas, and control- ling the creation of new replicas. Replica synchronisation is more complex to deal with, and different approaches exist. Before studying these approaches in Section 3.2, we clarify, in the next section, the difference between replication and synchronisation, which is sometimes a source of misunderstandings.