Distributed Production Deployment

In Figure 8.2 we illustrate the CATS component architecture designated for distributed production deployment. In the center we have the executable process CATS Peer Main, which contains the CATS Node, a network com- ponent embedding the Grizzly NIO framework [218], a timer component based on the Java timer service, a web interface component embedding the Jetty web server [219], and an Application component which may embed a command-line interface (CLI) or a graphical user interface (GUI).

The CATS Node provides a DHT service abstraction. By encapsulating all the protocol components – discussed in the previous section – behind the DHT port, the rest of the system is oblivious to the complexity internal to the CATS Node component. The benefit of encapsulation and nested hierarchical composition is even more pronounced in the whole-system simulation and stress testing architectures of Figure 8.6 where multiple CATS Nodesare manipulated as if they were simple components.

The JettyWebServer enables users to monitor the status of a node’s components and issue interactive commands to the node using a web browser. The CATS Node exposes its status through a Web port. The HTML page representing the node’s status will typically contain hyperlinks to its neighbor nodes and to the bootstrap and monitoring servers. This enables users and developers to browse the set of nodes over the web, and inspect the state of each remote node. An example is shown later in Figure 8.4.

8.2. DISTRIBUTED PRODUCTION DEPLOYMENT 131

CATS Bootstrap Server Click on a peer link to visit the peer.

Active peers

Count Peer Network address Last keep-alive

1 10000 [email protected]:12000 1s ago 2 20000 [email protected]:22000 1s ago 3 30000 [email protected]:32000 3s ago 4 40000 [email protected]:42000 3s ago 5 50000 [email protected]:52000 4s ago 6 60000 [email protected]:62000 2s ago 7 70000 [email protected]:9000 0s ago Runtime

Memory System Java Virtual Machine Operating System

Total: 7.08 MB Free: 3.48 MB

Uptime: 16d22h40m39s Revision 276

Java HotSpot(TM) 64-Bit Server VM Version 1.6.0_26 / 20.1-b02

Linux / amd64 Version 3.2.0-38-generic

Launch new peer

On machine cloud1.sics.se try to launch peer 1234 Launch!

Powered by

Figure 8.3.Interactive web interface exposed by the CATS bootstrap server.

On the left side of Figure 8.2 we have the architecture of a CATS client generating put and get workloads using the YCSB benchmark [60]. We have used this type of load generator clients in the performance evaluation of CATS presented in Chapter 9. The architecture of the CATS bootstrap server appears on the right side of Figure 8.2.

Figure 8.3 illustrates the web interface at the CATS bootstrap server. Here, users can see the list of active peers, with the last time a keep-alive message was receive from each. User can navigate to any live peer, and even launch a new peer on one of a set of preconfigured cluster machines.

Figure 8.4 shows the web interface exposed by a CATS peer. Here, users can interact with the local node, and inspect the status of its consistent hashing ring topology, its key range replication responsibilities, as well as failure detection statistics, and various information about the run-time system, such as the size of the heap and the amount of free memory.

Users may kill a peer – using its web interface – in order to test or to demonstrate a system reconfiguration. Users may also initiate put and get operations using the Distributed Hash Table input form. The interactive operation results and statistics are illustrated in Figure 8.5.

CATS Peer 30000

Peer network address: [email protected]:32000 Local DIGHT Service Bootstrap Server

Distributed Hash Table

Get key 30000 Get Put key 30000 with value abc Put

Key Ranges

Range Replication group Version Replica State Items

(20000, 30000] {30000, 40000, 50000, 60000, 70000} 10 0 READY 0

(10000, 20000] {20000, 30000, 40000, 50000, 60000} 0 1 READY 2

(70000, 10000] {10000, 20000, 30000, 40000, 50000} 0 2 READY 5

(60000, 70000] {70000, 10000, 20000, 30000, 40000} 10 3 READY 0

(50000, 60000] {60000, 70000, 10000, 20000, 30000} 10 4 READY 1

Peer state: INSIDE. Hover your mouse over the items count to see a list of all items in the range. Consistent Hashing Ring

Predecessor Self Successor Successor List

20000 30000 40000 [40000, 50000, 60000, 70000, 10000, 20000]

Failure Detector

Peer Network address Last RTT RTT avg RTT std RTTO RTTO show

70000 [email protected]:9000 0.47 ms 0.59 ms 0.21 ms 1.42 ms 10.00 s 20000 [email protected]:22000 0.48 ms 0.65 ms 0.26 ms 1.71 ms 10.00 s 40000 [email protected]:42000 0.59 ms 0.66 ms 0.20 ms 1.45 ms 10.00 s 50000 [email protected]:52000 0.51 ms 0.66 ms 0.14 ms 1.21 ms 10.00 s 60000 [email protected]:62000 0.60 ms 0.68 ms 0.21 ms 1.53 ms 10.00 s Runtime

Memory System Java Virtual Machine Operating System

Total: 7.80 MB Free: 1.93 MB

Uptime: 107d20h20m31s Revision 276

Java HotSpot(TM) 64-Bit Server VM Version 1.6.0_26 / 20.1-b02

Linux / amd64 Version 2.6.38-15-server

To kill this peer click on the top-right button. Powered by

Figure 8.4.Interactive web interface exposed by one of the CATS peers.

Figure 8.5 describes the results of interactive put and get operations. First, the operation response or an error message is printed. Then, the successor list returned by the lookup for the requested key is shown together with the latency of the lookup. Fine-grained timing statistics are shown for both the read and the write phases of the operation. This illustrates how quorums are formed in each phase, showing the latency of the round-trip times to each quorum member, in the order in which acknowledgements were received. In this example the replication degree was five, so a majority quorum consists of three nodes. Finally, the total operation latency is shown together with the consistent quorum accessed by the operation.

For the get operation, the returned item value or an error message is printed first. The remaining statistics are very similar to the ones output by a put operation. A notable difference is that for a get operation, if all