Scalability of P-CSI

We evaluated the P-CSI model for various system sizes, i.e. number of sites, to demonstrate its scalability under the scale-out model. For each system size, we measured the maximum transaction throughput and the average transaction latency. We performed this evaluation over both the local area MSI cluster as well as the wide-area cluster using Amazon EC2. In the evaluations using the MSI cluster, we evaluated the system with both the in-memory storage and HBase storage. In case of evaluations using EC2, we used the in-memory storage.

We also compare the performance of partial replication with the full replication scheme. For full replication scheme, the database contained equal number of items as that of the partial replication configuration, however each item was replicated on all sites, i.e. the replication degree was set the number of total sites. Note that this configuration corresponds to the basic CSI/PSI model. Thus, this evaluation also presents the comparative evaluation of CSI/PSI protocol and the P-CSI protocol.

Figures 4.6 and 4.7 show the transaction throughput data for systems using in-memory storage and HBase storage, respectively. This data corresponds the evaluation performed using the MSI cluster. Table 4.2 shows the throughput scalability data for experiments performed on Amazon EC2 using in-memory storage. For the figures we can see that P-CSI provides near-linear scalability; the maximum throughput achieved scales almost linearly with increase in number of sites. The throughput provided by full replication is much smaller compared to the throughput provided by partial replication. Moreover, in contrast to partial replication, full replication offers poor throughput scalability. For example, if we double the number of sites from 10 to 20, in full replication throughput increases only by a factor of roughly 1.29 for in memory storage and roughly 1.39 with HBase storage. The same for partial replication is 1.95 and 1.94, respectively. The throughput gain achieved in full replication diminishes even further if we scale the system to 30 sites.

0 5000 10000 15000 20000 10 15 20 25 30

Transactions per second

Number of Sites Partial-Replication Full-Replication

Figure 4.6: Transaction Throughput Scalability with InMemory Storage

500 1000 1500 2000 2500 3000 3500 10 15 20 25 30

Transactions per second

Number of Sites Partial-Replication

Full-Replication

Figure 4.7: Transaction Throughput Scalability with HBase Storage

provided by full replication approximately by a factor ranging between 2 to 4. This factor tends to be higher with increase in system size mainly due to poor scalability offered by full replication. One would expect that this factor to be close the ratio of replication degree of partial replication to the replication degree of full replication. For example, in case of 30 sites we would expect the throughput of partial replication with replication degree of 3 to be 10 times higher than the the throughput of full replication, which has replication degree of 30. However, the throughput is affected due to various other factors, such as two-phase commit overhead, overhead of executing read/write operations, etc., which induce same amount of overhead in both full and partial replication. Therefore, the relative throughput gain we observe in partial replication is lower than expected.

Table 4.2: System Performance on Amazon EC2 (Using InMemory Storage) Num. of Max Throughput Avg. Response Avg. Visibility

Sites (txns/sec) Time Latency Partial Replication 4 1938 198 ms 5.41 sec 8 3205 220 ms 5.46 sec Full Replication 4 768 318 ms 8.2 sec 8 938 370 ms 21.1 sec

Figure 4.8 shows the average transaction response times for partial and full replication schemes with in-memory and HBase storage. Table 4.2 shows the same for experiments per- formed on Amazon EC2 using in-memory storage. In case of partial replication, the response times remain roughly constant with increase in number of sites. The response times with HBase storage are higher than the response times with in-memory storage. The response times observed in case of Amazon EC2 experiments are higher due to higher communication latencies of wide- area network. The partial replication configuration provides lower response times compared to full replication. Moreover, in case of full replication, the response times typically increase with increase in the number of sites, indicating that full replication does not provide response time scalability.

Impact of scale-out on Visibility Latency

Another important question is whether increasing the system size, i.e number of sites in the system, has an impact on the amount of update delay and visibility latency. One of the advantage of the PCSI protocol is that the updates need to be propagated only to the sites containing the update partitions. This allows the system to be scaled to obtain almost-linear increase in transaction throughput, as demonstrated above. However, if scaling the system causes increase in the visibility latency, it can nullify the advantage of scaling the system. Thus, ideally one would expect that visibility latencies remain unaffected under scaling.

Figure 4.9 shows the average visibility latencies for above mentioned configurations. In full replication, visibility latency increases significantly with increase in system size due to the

0

50

100

150

200

250

300

350

400

450

10

15

20

25

30

Transaction Latency (ms)

Number of Sites

InMem-Partial

InMem-Full

HBase-Partial

HBase-Full

Figure 4.8: Transaction Latency

increase in update propagation overhead. The same trend can be observed in geographic-scale replication using EC2, as shown by the data presented in Table 4.2 In contrast, the visibility latency remains roughly constant in partial replication for both cluster-based as well geo-scale replication. This is because the P-CSI protocol requires propagating updates only to the sites containing the modified partitions, and therefore, the visibility latency depends only on the workload characteristics and not on the system size. In summary, with partial replication one can achieve near-linear scalability without increase in the visibility latency.