Bulk-Synchronous Patterns in Accelerator Workloads

Accelerators place different constraints on caches and coherence management rel- ative to contemporary general-purpose CMPs. An opportunity exists, chiefly driven by characteristics of accelerator workloads, to exploit these differing con- straints. However, optimizations that exploit characteristics of current workloads may come into conflict with the desire to support the variety present in emerging future workloads. To enlarge the space of applications accelerators target, they must not only support the data-parallel execution model prevalent today, but irregular task-parallel computation not well-suited to contemporary accelerators such as single-instruction multiple-data (SIMD) GPUs. Support for flexible and evolving task management models is not easily implemented with area-efficient hardware mechanisms. Therefore, we are motivated to investigate software mech- anisms when possible, and general hardware mechanisms as required, for support- ing a memory model tuned for accelerators.

4.2.1

Parallelism Structure

We observe that the programming styles adopted by developers for accelerator ap- plications share a common structure, similar to bulk synchronous processing [60]. These large-scale parallel applications are composed of collections of concurrently executing tasks comprising mostly data-parallel units of work. Tasks execute be- tween two barrier operations. The period between two barriers we define as an interval. The tasks exchange little or no data within an interval. At a barrier, modified shared data is made globally visible, and the next phase of computation begins.

Updates by a task during an interval can only be assumed by the programmer to be visible after the current interval has ended. Moreover, the programmer

cannot assume updates are invisible during the interval in which the update oc- curs. That is, updates are not buffered nor deferred and may take effect any time between the update and the end of the interval. Sharing modified data within an interval requires explicit programmer annotation.

We observe that popular programming models used in developing large-scale data-parallel applications do not depend on the hardware support provided by conventional systems, i.e., hardware coherence, for arbitrary sharing. In a barrier- synchronized, mostly data-parallel, task-based shared-memory programming model, coherence management is required to enable sharing; however, the mechanisms found in conventional CMP architectures to support arbitrary sharing through cache coherence are of marginal utility. As such, mechanisms for enabling some data, such as work queues or data structures, to be shared are required but need not cover all of memory at all times. Furthermore, the common structure present in these parallel applications is rooted in the programmer’s attempt to create scal- able code in a manner that is conceptually simple; thus there is minimal sharing.

4.2.2

Sharing Patterns

We provide analysis of a set of parallel visual computing workloads from VIS- Bench [12] and from the Rigel kernel benchmark suite. VISBench consists of a set of full applications that we run on the x86 platform. Analysis of these work- loads shows similar data sharing and synchronization patterns across workloads and between the two different platforms we target. Specifically, we investigate the sharing patterns of our workloads across synchronization boundaries. Figure 4.1 shows the naming scheme of these memory operations pictorially. Figure 4.2 gives the number of unique memory references that are shared across intervals, marked as input and output, and within an interval, marked as conflict, for VISBench

5 John H. Kelm ST Z LD Z ST X LD X ST Y LD Y LD Y Input Read by t2, written by t1in previous interval Output Produced before a barrier, read after it

Private Read and written by only one task

Conflict Written by t1, read by

t2within an interval

Barrier

Barrier

Figure 4.1: Categorization of memory access types in Rigel and VISBench bench- marks. Letters X, Y, and Z represent distinct addresses, and t1 and t2 represent

distinct tasks.

applications, and Figure 4.3 gives the same for the Rigel benchmark suite. Note that the analysis results for MRI versions differ due to the larger degree of regis- ter spilling on x86, resulting in more private reads on x86 compared to the Rigel variant. We exclude work distribution-related sharing from results to highlight application-level characteristics.

Figures 4.2 and 4.3 give the frequency of non-private loads and stores, which are data produced by one task and consumed by one or more other tasks. Non- private accesses are further broken down into whether the values are shared be- tween tasks within an interval, which we call conflict reads and writes, or across intervals, which we call input reads and output writes. The figures show that the majority of non-private loads are reads to data produced before the current in- terval began, i.e., input reads. At the same time, both conflict reads and writes to data shared within an interval are rare. Output writes, which are writes from one task in the current interval consumed by another task in the next interval, are more common in real applications than true shared writes which require intra- interval synchronization; moreover, true shared writes constitute a small fraction

h 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Wr ite Rea d Wr ite Rea d Wr ite Rea d Wr ite Rea d Wr ite Rea d Wr ite Rea d

PovRay MRI FaceDetect H.264 Blender ODE

Output Conflict Private Input

Figure 4.2: Relative fraction of memory misses for each access type for VISBench.

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Wr ite Rea d Wr ite Rea d Wr ite Rea d Wr ite Rea d Wr ite Rea d Wr ite Rea d Wr ite Rea d

CG DMM GJK HEAT KMEANS MRI SOBEL

Output Conflict Private Input

Figure 4.3: Relative fraction of memory misses for each access type for the Rigel benchmarks.

of overall execution. Also note that the number of unique output writes is much smaller than the number of input reads in the figure due to one-to-many sharing across intervals.

4.2.3

Accelerator Workload Characteristics

To summarize, we list five common characteristics in accelerator workloads: 1. Large amounts of immutable, read-shared data are present within an inter-

model descriptions or blocks of streaming media data.

2. Synchronization is coarse grained. Coarse-grained synchronization moti- vates our investigation of bulk coherence management at task boundaries. Indicative of this pattern are output writes and corresponding input reads in Figure 4.2 and Figure 4.3, which demonstrate that modified data is often read by a task after the interval in which the data was written ends.

3. There exist only small amounts of write-shared data within an interval. The small amount of write-shared data indicates that tasks are highly data- parallel with few data dependences between tasks within an interval. The effect is indicated in Figure 4.2 and Figure 4.3 as a lack of conflict reads and writes. Furthermore, the conflicts that do exist are structured, such as the histogramming operation on kmeans and reduction operations in cg. The structured conflicts are supported by collective operations such as atomic accumulate instructions in our workloads.

4. Fine-grained synchronization is present but rare. An example of such syn- chronization is atomic updates to shared data structures. Although not shown in the figures directly, we observe that much of the fine-grained syn- chronization that we do find is used for task management and not for appli- cation code.

5. When write sharing within an interval does exist, it is usually between few sharers.

Collectively, these characteristics demonstrate that little coherence manage- ment is required within an interval, indicating the potential for pushing coherence management into software to be performed logically at the end of an interval. At the same time, mechanisms must be present to allow small amounts of fine-grained

synchronization and data sharing within an interval for supporting task manage- ment efficiently. Our findings further motivate the use of shared caches that can amortize the costs associated with data access to read-shared data, a prevalent access pattern in our target workloads.

4.2.4

Cache Coherence Management

A mechanism for maintaining coherence, whether fully supported in hardware or a software protocol partially supported by hardware, cannot simply be omitted from the design of future accelerators, but the constraints placed on accelerators with respect to coherence differ from those of general-purpose CMPs. CMPs rely on cache coherence and global synchronization mechanisms to provide shared re- source management. Alternatively, an accelerator architecture can employ relaxed memory models, explicit local and global memory operations, and a task-based programming model to execute the coherence actions needed to enforce the mem- ory model at barriers, thus providing structure without sacrificing performance. As a substitute for hardware cache coherence, we investigate the use of software enforcement of our Task-Centric Memory Model, as described in Chapter 5. In the next section, we provide a more detailed overview of coherence management costs for accelerators.