Notes about the Experimental Evaluation

The experiments conducted for the performance evaluations in Chapters 5 to 7 have several things in common, which I will describe in this section (e. g., the selection of benchmarks). However, the experiments differ in other aspects such as the hardware they have been executed on, which I will described in the sections about the respective experiments.

The lack of available TM benchmarks is a big problem for TM research in general. The STAMP TM benchmark suite [11] is basically the only set of C/C++ benchmarks that is freely available, used widely by the TM community, and is not meant to test pathological workloads or corner cases. Given that TM has so far seen little use by C/C++ programmers—or at least that those programmers have not made their uses public, it is also not possible to just take existing non-benchmark programs that use transactions and transform them into benchmarks. There are other applications with transactions that have been used by research groups as experimental base for publications, such as those used in a study [84] by Pankratius and Adl-Tabatabai, but those have not been made publicly available. Besides the benchmarks that try to resemble real programs, there also exist microbenchmarks that typically test TM performance when transactions are used to synchronize access to shared data structures. A frequently used group of such microbenchmarks tests concurrent operations on different implementations of sets of integers. Therefore, and due to the lack of better candidates, I use both the STAMP and the integer set benchmarks.

IntegerSet benchmarks. The transactional workload that is created by these benchmarks is a sorted set of integer values that is accessed and modified by several threads. Each thread continuously runs a transaction that either inserts a new element into the set, removes an element from the set, or tests whether a certain element is contained in the set. This set is not a multiset, so a new element is only inserted if it is not yet present in the set; thus, insert and remove operations are not guaranteed to perform transactional write operations.

Each of the IntegerSet benchmarks implements the set either using a skip list, a red-black tree, a sorted linked list, or a hash table. The skip list uses at most 8 levels. The hash table uses open hashing, 217 buckets, and a multiplicative hash function; each element in the hash table resides in a separate list node

Benchmark Comments Benchmark parameters Genome 16M segments -g16384 -s64 -n16777216

Sim: 16K segm. -g256 -s16 -n16384

Genome-4M 4M segments -g16384 -s64 -n4194304

Genome-8M 8M segments -g16384 -s64 -n8388608

KMeans-Lo 64K input -m40 -n40 -t0.00001 -i random-n65536-d32-c16.txt

Sim: 2K input -m40 -n40 -t0.05 -i random-n2048-d16-c16.txt

KMeans-Hi 64K input -m15 -n15 -t0.00001 -i random-n65536-d32-c16.txt

Sim: 2K input -m15 -n15 -t0.05 -i random-n2048-d16-c16.txt

Vacation-Lo 4M transactions -n2 -q90 -u98 -r1048576 -t4194304

Sim: 4K txns. -n2 -q90 -u98 -r16384 -t4096

Vacation-Hi 4M transactions -n4 -q60 -u90 -r1048576 -t4194304

Sim: 4K txns. -n4 -q60 -u90 -r16384 -t4096

Vacation-1M 1M transactions -n10 -q90 -u80 -r65536 -t1048576

Vacation-2M 2M transactions -n10 -q90 -u80 -r65536 -t2097152

Vacation-2M-Lo 2M transactions -n2 -q90 -u98 -r1048576 -t2097152

Vacation-2M-Hi 2M transactions -n4 -q60 -u90 -r1048576 -t2097152

SSCA2 -s20 -i1.0 -u1.0 -l3 -p3

Sim: -s13 -i1.0 -u1.0 -l3 -p3

Table 3.2: STAMP benchmark configurations. The configurations annotated as “Sim” in the second column are used for experiments executed in a simulator (see Chapter 7), all other configurations are used by non-simulated executions.

referenced by one of the buckets, so adding an element to the table requires one call ofmallocto dynamically allocate memory.

Table 3.1 shows the benchmark configurations that I use. The operations performed by each transaction are chosen randomly such that the potentially updating transactions with insert and remove operations occur with the prob- ability shown in the second column of the table; insert and remove operations always occur with the same probability. Note that as discussed previously, the probability of transactions that actually modify state might be lower (but ele- ment lookups are always read-only transactions). The values of elements used as arguments to operations are chosen randomly in the range of zero to the value shown in the right-most column of the table. During benchmark initialization, the integer sets are populated with half as many elements as the upper bound of the range from which their values are picked; therefore, the number of elements in the sets will remain roughly constant during the execution of the benchmark. Each benchmark is executed for five seconds except if a simulator is used to execute experiments as in Chapter 7, in which case the benchmarks perform a fixed and lower number of operations to keep simulation time reasonable. Finally, the implementations of the IntegerSet benchmarks used to evaluate TM metadata colocation differ slightly (see Section 6.3.2 for details).

STAMP benchmarks. The STAMP benchmarks that I selected for my ex- periments are Genome, KMeans, Vacation, and SSCA2. Genome’s transactions access a mix of pointer data structures and a large number of character strings.

Note that STAMP’s manual instrumentation of memory accesses in transactions does not treat the frequent string comparisons in the benchmark’s transactions as transactional accesses because these strings do not change in the respective phase of the benchmark. Thus, this is a programmer-supplied optimization, which is not available to TM compilers such as DTMC. This increases the num- ber of transactionally accessed memory locations in each transaction compared to what has been reported for STAMP originally. Genome also uses almost 1.5GB of memory on 64b systems with its default non-simulator configura- tion. KMeans mostly operates on arrays of primitive types, has rather short transactions that access few memory locations, and spends only little of its to- tal execution time in transactions. Vacation simulates a reservation system; its transactions mostly operate on a couple of red-black trees and linked lists. SSCA2’s transactions are used to build graph data structures in parallel that are implemented using arrays; it also spends only little of its total execution time in transactions.

Table 3.2 shows the benchmark configurations that I use. The configurations annotated as “Sim” are used for the experiments in Chapter 7; they have shorter execution times, which keeps simulation time reasonable.

The implementation of these benchmarks is based on STAMP version 0.9.6 but contains a few modifications and bug fixes. First, the original STAMP used map data structures with 32b integer keys and values also to map from or to pointers—these data structures were changed to instead use pointers as keys and values. This allows the STAMP benchmarks to also work correctly when compiled as 64b-pointer programs, and improves the quality of the compiler analyses used in Chapter 6. Second, the barriers that control when application threads start to execute transactions have been changed to a spinning imple- mentation, which allows for more precise measurements of the execution time when used with a simulator; the barriers now also work with thread counts that are not a power-of-two value. Finally, some of the entry points to the main data structures in the benchmarks were placed on separate cache lines to avoid unnecessary hardware transaction false conflicts.

Other benchmark, software, and execution parameters. The specific hardware or simulator used for the experiments in Chapters 5 to 7 are different and described in these chapters. Notwithstanding, benchmarks are never exe- cuted with more threads than logical CPUs provided by the hardware because TinySTM++’s implementation only uses spinlocks; as explained previously, this is not a general limitation, nor is it a case that would be very important in practice. Threads are pinned to logical CPUs in the STM experiments (see Section 5.2.2 for details) and the HTM experiments (see Section 7.4; threads are pinned to cores of the simulated CPU).

All performance measurements have been executed several times, and the data shown is the average of the individual measurement results.

As mentioned previously, the TM runtime libraries are statically linked to the benchmarks, and link-time optimizations are enabled (using the default LLVM optimization passes). Chapters 5 and 7 use the same DTMC version (i. e., based on LLVM 2.8) and the most recent TinySTM++ and 64b benchmark applications. The benchmarks in Chapter 6 use an older DTMC version based on LLVM 2.1, an older TinySTM version, and are 32b programs. The glibc

version used to execute experiments is 2.15 in Chapter 5, 2.10 in Chapter 7, and 2.3.6 in Chapter 6. All benchmarks use glibc’s default memory allocator except the HashTable experiments of Chapter 7, which use the Hoard memory allocator [8].

Chapter 4

Integrating TM with

C/C++ Programs

Language integration is essential for TM to provide usability and composability. Requiring programmers to manually instrument the program code to use TM like a library would be an obstacle to reaching those goals. Instead, programmers should be able to just demarcate which regions in the code are supposed to be transactions, and have the compiler automatically transform these code regions. A simple option to demarcate regions are calls to special marker functions:

DTMC BEGIN(); x++; DTMC COMMIT();

This is fairly straightforward to implement in a compiler and also was what I had chosen to do in early versions of DTMC. It does not require extensions to the programming language but has shortcomings exactly because of that. First, the compiler would still have to be aware of the program’s source code during the transformations because otherwise, it cannot precisely detect what is transactional code (e. g., if transforming intermediate code created by other parts of the compiler). Second, it still needs to be specified how transactions interact with other language components such as exception handling. Overall, the compiler’s frontend has to support TM and we need an extended language specification.

Thus, one can also extend the language in the first place because it would need a similar kind of compiler support and level of understanding by the pro- grammers. The resulting lack of compatibility with older compilers and related tools is likely to be outweighed by the increased clarity in both syntax and semantics of programs with transactions.

The central TM language construct in C/C++ are transaction statements, which are executed as a transaction and consist of either a single statement or a block of statements (see Section 2.3 for examples). To specify their semantics and to implement support for them, we need to also consider other specifications and interfaces, which are shown in Figure 4.1.

The programming language’s memory model, which defines how programs access memory and how multi-threaded programs synchronize concurrent ac- cesses, has to be extended with the semantics of transactions. An extended memory model then specifies the orderings guaranteed by transactions and how they are related to the rest of the model. One particular TM concern is how

Transactional language constructs

(e.g., __transaction_atomic { ... } ) C++11/C11 memory model

TM runtime library ABI

Extends

TM compiler

Hardware memory model, HTM TM runtime library

Figure 4.1: Relation of the TM specification to other interfaces.

transactional memory accesses interact with nontransactional memory accesses, covering questions such as strong versus weak isolation or publication and pri- vatization.

For reasons further explained in Section 4.2, we want the compiler support to be fairly independent of actual TM implementations, and rather let the compiler target a common Application Binary Interface (ABI) as intermediate interface. TM runtime libraries then implement this ABI for some set of hardware archi- tectures, which also provide different hardware memory models. The compiler and the TM runtime libraries thus have to jointly implement the C++ mem- ory model extended with transactions, on top of the memory models provided by hardware architectures (e. g., by using hardware instructions with memory barrier semantics to ensure certain orderings required at the programmming language level).

4.1

Specifying TM for C/C++

In this section, I will explain a draft specification of transactional language con- tructs for C++ [63] (“specification” for short), which is a joint effort by several companies (HP, Intel, IBM, Oracle, and Red Hat) and individuals, including people working on the major C++ compilers. It is based on and extends the C++11 standard [65]. I will also describe the extensions to the C++11 memory model [6, 10] that are necessary for the additional TM language constructs. I will then, in the following section, derive what this specification actually means in terms of requirements on a TM compiler and runtime library. Finally, I will compare this specification to other approaches for modeling the correctness of concurrent operations in Section 4.3.

Overview of the C++ memory model. To put it simply, the memory model uses per-thread program order together with the synchronization relations present in the program to derive a happens–before relation, which then also describes how memory locations are modified and which values are read by load operations. Programs have to be free of data races (i. e., all accesses must be properly ordered by happens–before); if they are not, then the behavior of the program is undefined. Thus, the memory model defines which executions are

allowed for a given program and how a program’s threads can synchronize with each other.

From a formal perspective [6], the model consists of several relations that are either derived from a program’s source code (and thus are fixed for a given program and control flow path), or can be chosen freely to represent the different executions a program might have (i. e., to model the indeterminism that arises when executing a multi-threaded C++ program).

First, the sequenced–before, data–dependency, and additional–synchronizes– with relations are determined by the program’s source code and the assumed control flow in the program. sequenced–before basically is the per-thread order of operations in the program, and thus takes a central role. The other two relations are not as important to understand how TM fits into the memory model (e. g., additional–synchronizes–with models orderings such as between a child thread and the operation in the parent thread that created them).

Second, reads–from, modification–order , and sequentially–consistent are “wit- ness relations” that represent the relation or ordering between accesses by sev- eral threads. The relations are the witnesses of some chosen execution and are used to enumerate all possible executions. modification–order orders opera- tions that modify the same location, and reads–from defines which modification to a location is observed by a reading operation to the same location. Loca- tions can either be nonatomic (normal program state), atomic (accessible by atomic operations such as CAS), or mutexes. Atomic operations accept an additional memory order modifier that affects the strength of its synchroniza- tion, including which other operations it potentially synchronizes with (e. g., memory order relaxed does not impose an order, whereas memory order seq cst is stronger). sequentially–consistent orders sequentially consistent operations (e. g., locking and unlocking a mutex).

Together, these six relations can be used to enumerate the possible candidate executions of a program. The model also uses them to derive other relations, of which synchronizes–with and happens–before are the most important ones from the perspective of our discussion. synchronizes–with contains the order that is enforced by synchronization operations in the program: for example, two atomic operations in different threads on the same memory location, one a store withmemory order release and the other a load withmemory order acquire, result in a synchronizes–with edge from the store to the load (see Table 2.1 for the notations I use in algorithms). happens–before is, roughly speaking, the transitive closure of synchronizes–with and sequenced–before (it is not transitive though for chains including a certain kind of memory order modifier on atomic operations). Thus, happens–before is the top-most specification of ordering in some execution of a program, but it is not a total order.

Next, these six relations and the derived relations are used to determine which candidate executions are consistent. The precise consistency constraints are described in the formal model and I will not explain them in detail. How- ever, they are pretty intuitive and basically constraints on the various relations and combinations of them; for example, reads–from must be consistent with modification–order and happens–before in that reads observe the most recent value written to a location according to the orderings specified by happens– before and modification–order . Any inconsistent candidate executions are not further considered.

race condition (e. g., data races due to conflicting accesses by different threads that are not ordered by happens–before), then the behavior of the program is undefined; otherwise, behavior will be equal to one of the candidate executions. Note that this also highlights the “catch fire” semantics of race conditions and undefined behavior. In particular, if there can be a case where a program is not race-free, then the program’s execution enviroment (e. g., the compiler or the TM runtime library) does not even need to ensure that all other candidate executions would execute correctly.

Programmers thus have to essentially use the right amount of synchroniza- tion in their programs to prevent data races and other incorrect yet race-free behavior. They can do that using ordered atomic operations, locks, or also transactions as we will see later. The compiler and all other parts in the C++ execution enviroment, including TM runtime libraries, are responsible for trans- lating race-free source programs into native code that is also race-free and only yields executions that are equivalent to consistent candidate executions in the model. This can restrict compiler transformations (e. g., code movement across operations that contribute to synchronizes–with), and the compiler’s code gen- erator and the runtime libraries have to use suitable native code (e. g., memory barriers) that correctly implements the C++ memory model on top of the mem- ory model provided by the targeted hardware platform.

Please also note that the memory model relies on ordering as expressed in happens–before and not on linearizability, which is commonly used to reason about concurrent data structures (see Section 2.1). Furthermore, the C++ standard is rather vague regarding progress guarantees (e. g., it allows I/O op- erations to actually finish after the respective I/O function has returned, which would not be a linearizable operation). However, if the operations of a concur- rent data structure are indeed linearizable, then ordering in happens–before also becomes straightforward.

TM Language Constructs. The main transactional language construct are transaction statements, consisting of either the transaction atomic keyword or the transaction relaxed keyword followed by a compound statement (see Section 2.3 for examples). Alternatively, transactions can also have the form of transaction expressions or function transaction blocks. The former make paren- thesized expressions transactional, whereas the latter execute whole function bodies as a transaction; both can be expressed with transaction statements, so I will not consider them further.

Informally, atomic transactions (using the transaction atomic keyword) can be thought of as executing instantaneously (i. e., atomically and in isolation from other threads) if there are no race conditions with other nontransactional operations. These transactions are called atomic transactions and can only exe- cute code that can execute safely in an atomic transaction or can be transformed so that it is safe code.

Alternatively, transactions can also be annotated as relaxed transactions (us- ing the transaction relaxedkeyword) and can then execute unsafe code. Ex- amples for unsafe code are accesses to volatile memory locations or C++ atomic variables, file I/O, or functions in libraries only available as native code. Thus, relaxed transactions do not provide full atomicity but are atomic only with re- spect to other atomic or relaxed transactions; in contrast to atomic transactions,

they can communicate with other threads from within the transaction via unsafe code. In a typical STM implementation, those relaxed transactions that exe-