Extending SAMRAI with CudaPatchData

The SAMRAI library uses object-oriented design patterns to allow for easy in- teraction with user-supplied code [78]. Each of the basic structural units of the AMR hierarchy: patches, patch levels, and the patch hierarchy itself; are provided as fundamental software constructs by SAMRAI. TheSi+?class is a container for all the data living in a particular mesh region, and provides a way to access and manipulate this data. All the data on a patch are handled us- ingSi+?.iobjects, each of which represents some simulation quantity on the mesh. TheSi+?.iinterface uses the Strategy design pattern [161], and defines a set of operations that a class must provide in order to be interoper- able with SAMRAI’s data management and communication routines. We use this interface to develop a library capable of storing patch-based data in GPU

PatchData - Box d_box - Box d_ghost_box - IntVector d_ghosts - double d_timestamp + getBox() : Box + getGhostBox () : Box + getGhostCellWidth() : IntVector + setTime(timestamp : double) : void + getTime() : double

+ copy(src : PatchData) : void + copy2(dst : PatchData) : void

+ copy(src : PatchData, overlap : BoxOverlap) : void + copy2(dst : PatchData, overlap : BoxOverlap) : void + canEstimateStreamSizeFromBox() : void + getDataStreamSize(overlap : BoxOverlap) : void

+ packStream(stream : MessageStream, overlap : BoxOverlap) : void + unpackStream(stream : MessageStream, overlap : BoxOverlap) : void + getFromRestart(restart_db : Database) : void

+ putToRestart(restart_db : Database) : void + getDim() : Dimension

Figure 6.2: The SAMRAI PatchData interface.

memory whilst still using SAMRAI for mesh management, communication and visualisation.

SAMRAI's PatchData Interface

TheSi+?.iinterface is used to define the operations a class needs to sup- port to allow SAMRAI to gather data from the patch in order to transfer it to other patches in the hierarchy. Figure 6.2 documents the fullSi+?.iinter- face. During a simulation,Si+?objects store an array of pointers to the nec- essarySi+?.iobjects. The functions that theSi+?.iroutines perform include copying data from oneSi+?.iobject to another, packing the data corresponding to a given region of the patch into a message buffer, and unpack- ing data from a message buffer into a given patch region. These methods are the key points of theSi+?.iinterface that we implement.

whilst still accessing the full capability of SAMRAI to manage a simulation on an adaptive mesh has been used to its full potential in a number of applications. For example, in [163], Wijesinghe et al. present a hybrid hydrodynamics code that uses a combination of continuum methods and Direct Simulation Monte Carlo to perform simulations at a range of scales. This implementation relies on a custom datatype, but the AMR capability of the SAMRAI library.

By allowing an application to fully control data management, SAMRAI is easy to use in an existing application. In the case of our GPU-based exten- sions, the abstraction provided by theSi+?.iinterface let us store simula- tion data in the GPU memory at all times and only copy data across the PCI bus when necessary.

CudaPatchData Library

The library we have developed contains two packages, modelled after those in the main SAMRAI distribution. The first,T/i, contains three differentSi+?.i

implementations for managing data in GPU memory. The second,;2QK, pro- vides a collection of coarsen and refine routines that are essential when copy- ing data between patches at different levels of the hierarchy. We describe both packages in the remainder of this section.

The three differentSi+?.iimplementations we have developed are collectively called*m/Si+?.i, and the three implementations exist since they are specialised for the three data centrings required for the hydrodynam- ics scheme used in CleverLeaf. The common data store for each class is the

*m/``v.iobject. This class is responsible for allocating a contiguous ar- ray of data in GPU memory, corresponding to a given box size. This class also contains data-parallel routines to copy data, pack a region of the array into a buffer, and unpack a buffer into a region of the array. Each data-centring passes a slightly different"Qtto the*m/``v.iobject it owns, allowing the nec- essary data to be stored. The three centrings required for CleverLeaf are: cell- centred, node-centred, and side-centred. Figure 6.3 shows the design and data

CudaArrayData double* d_cuda_buffer; CudaCellData CudaArrayData* d_array_data; CudaNodeData CudaArrayData* d_array_data; CudaSideData CudaArrayData* d_array_data; PatchData

Figure 6.3: The SAMRAI CudaPatchData datatypes. stored by each class.

During an AMR simulation, boundary conditions can be filled in one of three ways: (i) using the physical boundary conditions; (ii) with data from a neighbouring patch on the same level; or (iii) with data from a patch on the next coarser level. Filling the boundary cells with the physical boundary conditions is handled by the application, and requires no additional features to be added to our library. When data must be transferred between two patches at the same level of refinement, a copy routine is used. If the two patches involved in the copy operation are located on different nodes the required data must be trans- ferred using MPI. Supporting MPI is essential for any modern scientific code, and by including the necessary routines in our library we can run on multiple GPUs.

The data-parallel copy and packing operators are designed as follows. Each operation will receive a"Qtas one of its parameters. This box contains the region of the patch that needs to be operated on (whether data is being copied, packed, or unpacked). In all cases, the size of this box controls the number of CUDA threads that will be launched. Each thread will then be responsible for copying, packing, or unpacking one array element.

double* cuda_stream; packStream() (3,3) (4,4) CudaArrayData double* d_cuda_buffer; CudaCellData CudaArrayData* d_array_data; T2 T0 T1 T3

Figure 6.4: Data-parallel buffer packing for MPI operations.

to pack data from the required region into a contiguous buffer in GPU mem- ory. This buffer is then copied to the host memory and passed to SAMRAI, which handles the MPI communications. To unpack received data, the buffer is copied into the GPU memory and then unpacked in parallel using another CUDA kernel. Once the data has been transferred, a newSi+?.iobject is created locally and the copy operators described previously can be used to fill the boundary cells on the receiving processor. We launch one CUDA thread per element to be packed into the buffer, ensuring the maximum amount of parallelism is exposed. As an example, Figure 6.4 shows how the overlapping region is copied into the contiguous buffer in parallel.

The operators described by the Si+?.iinterface are sufficient for transferring data between objects at the same level of refinement. However, to transfer data between objects at different refinement levels, we must use a refinement operator (if data is moving to a higher refinement level) or a coarsen operator (if data is moving to a lower refinement level). In SAMRAI, these oper- ations are handled by two interfaces: *Q`b2MPT2`iQ`and_27BM2PT2`iQ`, that provide the necessary methods for coarsening or refining data. To allow the*m/Si+?.iclasses to be used in an AMR simulation, we must provide operators to coarsen and refine data between different levels of the hierarchy.

We provide four data-parallel operators for coarsening and refining data. As with the copy, pack, and unpack routines, each method executes using mul- tiple CUDA threads in a highly parallel fashion. These are, to the best of our

knowledge, the first data-parallel implementations for each of these operators. As an illustrative example, we consider linear interpolation for node-centred data. The code listing for our data-parallel algorithm is shown in Figure 6.5. In a typical implementation, data dependencies exist between temporary variables in different loop iterations and the algorithm is not immediately amenable to the data-parallel programming model of a GPU. Through substitutions and op- eration re-ordering, we remove these dependencies and develop an algorithm thatisfully data-parallel (see Figure 6.6). When this kernel is launched to re- fine some region of data, one CUDA thread can be used per fine node, offering massive parallelism. We also provide conservative linear refine operators for the cell- and side-centred data, as well as a node-centred injection operator.

Following the same general design as SAMRAI, we have partitioned our library into two packages. The first,T/i, contains all the data types described so far, the second;2QKcontains the operators used to coarsen and refine data on a Cartesian grid. Together, theT/iand;2QKpackages provide all the nec- essary components for a block-structured AMR simulation to be solved on a GPU. All that the user code must provide is a black-box integrator that can advance the simulation on a single patch.