ResilientTxBench

Using the parallel workers framework, we designed theResilientTxBenchprogram as a flexible framework for evaluating the throughput of the transactional store with

§5.4 Performance Evaluation 143

different configurations. The program is located at x10.dist/samples/txbench in our repository. Table 5.7lists available parameters inResilientTxBenchand the values used in our evaluation.

Table 5.7: ResilientTxBench parameters.

Description Used values

p number of worker places

16 32 64 128

t number of producer threads per place 4

d duration in ms for a benchmark execution 5000

c the concurrency control mechanism RL EA UL

RV LA WB

r the range of keys in the store 218

u percent of update operations 0

50

h,o

h×o is the transaction size.

h is the number of participating places

and o is the number of operations per participant 2×8 4×4 8×2

f transaction coordinator is one of the participants true

The program startsp∗t transaction producer threads, wherep is the number of worker places and t is the number of producer threads per place. Each producer thread starts a loop that generates one transaction per iteration and terminates afterd

milliseconds have elapsed. According to the given transaction size, a producer thread selectsh different participants andorandom keys at each participant. The readset and the writeset of the transaction are randomly selected from the range ofr keys of the store, based on the configured update percentageu. The keys are evenly partitioned among the places. By settingf =true, we request that the first participant is always the place initiating the transaction; the remaining h−1 participants are selected randomly. We believe this strategy is well aligned with thePGASparadigm, as users are expected to favor locality by initiating transactions from one of the participating places.

Each producer thread records the number of completed transactions Tij and the

exact elapsed time in milliseconds Eij, where i is the place id and j is the thread

id within that place. On completion, the first place collects these values from each thread and estimates the throughput(operations per millisecond)as:

throughput= ∑ p i=1∑ t j=1Tij∗h∗o ∑p i=1∑ t j=1Eij ×p×t

Although we implemented the sixCCmechanisms described in Section5.2.3.3, we focus our evaluation on only two of these mechanisms: RL EA ULandRV LA WB, which, respectively, represent the two extremes of pessimism and optimism in lock- based concurrency control. RL EA ULacquires the read locks and write locks before accessing any key and keeps acquiring the locks until the transaction terminates. On the other hand,RV LA WBdoes not acquire any locks until the commit preparation phase, which provides more concurrency to the application. In non-resilient mode,

RL EA ULhas an advantage over RV LA WB, since the former does not require a preparation phase for non-resilient transaction [Bocchino et al., 2008]. In resilient mode,RL EA ULcan only avoid the preparation phase in read-only transactions.

We started four producer threads per place and allocated two cores per producer thread. Therefore, each X10 place was configured with 8 worker threads usingX10 NTHREADS=8. We found that this level of parallelism is necessary for achieving scalable performance, because it provides enough capacity to a place to coordinate its own transactions and participate in transactions originated at other places. We measured the throughput with increasing numbers of producer threads: 64, 128, 256, and 512, and we expect the ideal performance to result in an increase in the throughput that is proportional to the number of worker threads. We evaluated the transactional store throughput in non-resilient mode and in resilient mode using the two implementations of the optimistic finish protocol: optimistic place-zero (O-p0) and optimistic distributed (O-dist).

Figure 5.11 shows the throughput results for read-only transactions, and Fig- ure5.12shows the throughput results for transactions performing 50% update opera- tions. For each configuration, we report the median, the 25th and the 75th percentile. Table 5.8shows a summary of the results with 512 producer threads.

From these results, we draw the following conclusions:

• The centralized optimistic finish implementation scales very poorly in this benchmark, due to the large number of concurrent transactions. The distributed implementation achieves much better scaling performance for this benchmark. • Read-only transactions scale well in both non-resilient and resilient (O-dist)

modes with bothCCmechanisms.

• The scalability with write transactions varies depending on theCCmechanism used. WhileRV LA WBscales reasonably well,RL EA ULstruggles to scale as the number of threads increases.

• With read-only transactions,RL EA ULenables the program to achieve higher throughput than withRV LA WB. For example, with 512 threads,RL EA UL

achieves an increase in throughput between 16% to 33% compared toRV LA WB. However, the situation is reversed with write transactions.

• With write transactions, RV LA WB enables the program to achieve higher throughput than withRL EA UL. For example, with 512 threads,RV LA WB

§5.4 Performance Evaluation 145

achieves an increase in throughput between 80% to 222% throughput increase compared toRL EA UL.

• With 512 threads, the throughput loss due to the resilience overhead of the X10 runtime and the resilient two-phase commit protocol ranges between 40%–49% for read-only transactions and between 41%–65% for write transactions. The overhead increases as the number of transaction participants increases.

Bocchino et al. [2008] performed comparative evaluation between the six CC

mechanisms for a non-resilientDTMsystem for the Chapel language. They found that

RL EA * mechanisms were always superior to other mechanisms because they avoid the commit preparation phase. On the other hand, our results demonstrate that the relative efficiency of different CCmechanisms depends on the application worload. Write-intensive applications can benefit more from an optimistic concurrency control mechanism where locking is delayed to the commitment time. The performance of the SSCA2-k4 clustering application, which we will explain in the next section, confirms this finding and provides more detailed analysis. Our work also demonstrates that

RL EA * cannot always avoid the preparation phase when used in resilient mode. There are many implementation differences that prevent direct comparison with the work inBocchino et al.[2008]. For example, they propose a low-level implementa- tion at the memory manager level, while our implementation is done at the runtime and application level. While we use a read-write wait-die lock, they use an exclusive write lock. Also, fault tolerance is not among their design goals.

We believe the achieved resilience overhead is acceptable, given the underlying tracking and commitment mechanisms on replicated data. We expect the use of small- sized transactions in compute-intensive applications to result in minimal resilience overhead in realistic application scenarios. The SSCA2-k4 application, which we will explain next, does not fit in this category; it is a communication intensive application that relies heavily on distributed transactions.

Table 5.8:ResilientTxBench transaction throughput (operations per ms) with 512 threads, in non-resilient and resilient mode withO-dist. The resilience overhead is shown in parentheses.

2x8 4x4 8x2

non-res. O-dist non-res. O-dist non-res. O-dist RL EA UL (u=0%) 7678 4278 (44%) 3951 2129 (46%) 2077 1064 (49%) RV LA WB (u=0%) 5776 3464 (40%) 2974 1714 (42%) 1570 914 (42%) RL EA UL (u=50%) 1082 533 (51%) 628 341 (46%) 489 214 (56%) RV LA WB (u=50%) 2893 1715 (41%) 1669 960 (42%) 1085 385 (65%)