Retrieving and Storing Data

To store data in a distributed fashion, we have to define a responsibility function mapping the position of a data item in the world to the node responsible for storing it. The self-stabilizing overlay was designed to provide such a mapping. Every node is responsible for storing all data items it is closer to than any other node. This means every nodes stores all data within its Voronoi cell. However, we cannot map the data space representing the area of the world directly onto the overlay ID space. Nodes near zone borders have much larger responsibility areas than nodes inside the zones creating an uneven distribution of load. Furthermore, we can take advantage of the zoning in the overlay to realize a replicated storing of data. Replication and redundancy are the main tools to provide reliability and resilience as shown in the following sections.

5.2.1 Data Space and Replication

A straight-forward way for a malicious node to prevent correct storing of data or to prevent honest nodes from retrieving the correct state is the storage and retrieval attack [111]. Instead of forwarding a store or retrieve message, a malicious node simply drops that message or replies with a faked result claiming responsibility for storing that data. One way to counter this attack is to place replicas of data in different locations in the key space so replicas can be stored and retrieved indepen- dently [59]. Since one malicious node on multiple paths to the replicas can still prevent the replica from being reached, it is beneficial to make this unlikely [5] or even guarantee paths to the different replicas to be disjoint [51].

We adapted our idea presented in [106] to retrieve replicas of data in a Pastry ring on disjoint paths to work in our two-dimensional overlay. Instead of mapping the world area directly onto the ID space, we map every position in the world to one position per overlay top-level zone. For every data item, one replica is stored in every

5.2 Retrieving and Storing Data 103

zone. Thus, every replica has a different position in the ID space. The replication factor r = r_x∗ r_y equals the number of zones depending on the bases b_x, b_y of the numbering systems used.

Referring back to figure 4.18 the world area is mapped onto the marked areas S_i,j with size s_x∗ s_y in the zones with size z_x∗ z_y. We more formally define two linear mappings s and t. The isomorphic mapping s maps the world area

W ={(x, y) ∈ R × R | 0 ≤ x < w_x, 0≤ y < w_y}

with size w_x∗ w_y onto the sub-area S_0,0: s : W → S_0,0 with

s ⎛ ⎝x y ⎞ ⎠₌ ⎛ ⎝wx/sx 0 0 w_y/s_y ⎞ ⎠_∗ ⎛ ⎝x y ⎞ ⎠

Afterwards, t translates the sub-area S_0,0 and the set of replica or zone identifiers

R ={(i, j) ∈ N × N | 0 ≤ i < r_x, 0≤ i < r_y}

to a key from the key space K ⊂ R × R using:

t ⎛ ⎝ ⎛ ⎝x y ⎞ ⎠_{, (i, j)} ⎞ ⎠₌ ⎛ ⎝i∗ zx j∗ z_y ⎞ ⎠₊ ⎛ ⎝x y ⎞ ⎠

We denote the compound mapping m = t◦ s.

Using s, any rectangular world area can be mapped onto any rectangular shape of

S_0,0. As the shape of S_0,0 depends on the zone shape depending on the chosen bases

b_x and b_y, any bases can be used for the ID space.

In each zone, all nodes hold a replica of all objects in the world so there is a copy of the world in each zone. This is also shown in figure 5.1 with s mapping an object

o in the world W to its replica o_0,0 in S_0,0 and t mapping o_0,0 to the replicas o_i,j in the other zones. By sending an area query to each of the zones, every replica can be retrieved independently and majority voting can be used to decide on the correct state as shown in the following section.

5.2.2 Data Retrieval on Disjoint Paths

When a node n tries to retrieve world state, it specifies an area to retrieve all objects lying in that area. This area A can be a single point A = p, p∈ W or a sub-area of

104 Chapter 5 A Virtual World Storage o_0,0 0 1 0 1 W S0,0 S1,0 S1,1 S0,1 o s t(x,(1,0)) o1,0 o1,1 o_0,1 t(x,(1,1)) t(x,(0,1))

Figure 5.1: Mappings_{s and t map object o to its replica IDs o0,0...o1,1}

the world A⊆ W . The location is then mapped to the r different zones using the function m generating areas A_0,0...A_{rx−1,ry−1}.

Every area A_i,j is a subset of the zone it is located in A_i,j ⊂ Z_i,j. A node only accepts retrieve operations for areas completely contained in W . Since s maps W onto S_0,0 and t just translates S_0,0 into the other sub-areas, all A_i,j are even subsets of the sub-area S_i,j in zone Z_i,j: A_i,j ⊆ S_i,j.

Sending Retrieval Messages

For each of these mapped area A_i,j , n sends a message with A_i,j as destination using the overlay. Depending on the type of area an areacast or a unicast is used. As the overlay supports convex polygon and circle shapes only for areacasts, these are also the only area shapes supported by the storage.

The message payload is a storage message of type RetrieveMessage specifying the type of operation to be performed by the receiver. Furthermore, n remembers the running retrieval and starts a timer specifying a timeout.

Assuming n lies in zone Z_a,b, then the top layer of its routing table contains nodes from all other zones Z_i,j, i = a, j = b. Since all destination areas A_i,j are in different zones, n will forward the messages to each of the nodes in its top layer. Only the message for destination A_a,bwill be forwarded to a node in zone Z_a,bin a lower layer

5.2 Retrieving and Storing Data 105 o_0,0 0 1 0 1 S_0,0 S_1,0 S_1,1 S0,1 o_1,0 o_1,1 o_0,1 n

Figure 5.2: Node_{n retrieving all replicas of an object o creates disjoint paths}

of the routing table. Therefore, in the next hop the messages will all arrive at one node in each of the different zones as shown in figure 5.2.

As long as prefix routing is used, the messages will also never leave these zones. The messages will only be forwarded to nodes matching the destination in more digits but not less. Therefore, the leading digits of all receiving nodes will always be the zone identifier of the zone the message was sent to by n. The only way for a message to leave its zone is switching from prefix-routing using routing tables to the neighbourhood-based routing.

However, the way IDs are distributed in the ID space actually prevents this. As we have shown in section 4.5.4, if there is one node in a zone Z_i,j, it is always closer to any point from S_i,j than any node from another zone. Since every A_i,j ⊆ S_i,j, the nodes overlapping A_i,j are all located in Z_i,j. Therefore, even when switching to neighbourhood-based routing, a message will not change zones.

Thus, the paths to the different replicas are completely disjoint except for node n. This contains the influence of malicious nodes because an attacker needs a malicious node on multiple paths to tamper with queries. For non-disjoint paths, an attacker only needs one common node on multiple paths to tamper.

106 Chapter 5 A Virtual World Storage

Processing Reply Messages

When the overlay has routed the messages, all nodes overlapping the destination areas receive the RetrieveMessage. They select all stored replicas lying in the requested area from their local data store and answer with a reply message of typeRetrieveResultMessage. This message will be sent directly to n using a shortcut connection the overlay will establish automatically.

Node n will record all incoming reply messages until it is able to finish the query and pass the final result back. If the query was a point-query, it will at most receive r of these messages each containing a set of replicas. When n receives such a message, it checks whether it has received more than r/2 messages in total. If yes, it performs a voting on the result where each message counts as one vote. If more than r/2 answers contain identical set of replicas, this set is returned. If not, n continues to wait for replies until r answers arrived or the timeout is reached. Then it returns the set of replicas with most votes. Furthermore, it signals its confidence in the result returning the share of votes the result achieved.

Area queries are handled in a similar way. However, for area queries the answer for one destination area A_i,j might consist of multiple messages if multiple nodes overlap the destination area. Therefore, the replica set of a destination area is created by assembling the result sets of the individual reply messages from one zone. Furthermore, n has to decide if all reply message for a destination area have been received. This is the case when replies from all nodes overlapping A_i,j have arrived. Since n does not necessarily know nodes in this area, this information has to be sent to n.

Therefore, all reply messages include the neighbourhood of its sender. Thus, n can check for each reply whether any neighbours of its sender also overlap the destination area. If a neighbour overlaps and its reply message has not been received yet, the replica set of this zone is still incomplete and n will wait for further replies. Only when more than r/2 replica sets have been received, the above voting procedure starts with each replica set having one vote. It returns, when a majority is reached or waits for further replies until the majority or the programmed timeout is reached.

5.2.3 Storing Data

The storage does not only allow retrieving static state. It also updates the world state. Consequently, the storage also allows storing of information. When a node