Changes to the library

Some subroutines in the BoxLib library do not work with non-cubic boxes and hence require a change in the signature and bodies. We list below the changes we made to the library routines for them to work seamlessly with non-cubic blocks. To give precedence to these subroutines over the default routines, we make changes to them and copy the appropriate source files to the current project working directory to make sure our versions of the routines are invoked. We first give the file name to which the subroutine belongs and then describe the necessary change. In addition to these changes, we discuss some precautions, compilation options and the method to profiling BoxLib based applications using Scalasca [121] (see Chapter 3) in Appendix B.

– make new grids.f90: One of the parameters passed to the subroutine make new grids() subroutine is the scalar max grid size that is used to specify a cubic block shape. This must be changed to a vector i.e. max grid size(:) (in Fortran90). When the boxarray maxsize() subroutine is called inside the body of this routine, the vector is di- vided by the refinement ratio to give the correct box-size at the finer resolution. Similarly vectors max grid size 2(:) and max grid size 3(:) must be passed to the subroutine enforce proper nesting() instead of scalars. The original subroutine declares a local integer variable by the name of max grid size 3 which must be changed to an allocatable vector (dynamic array) using allocate(max grid size 3(mba%dim)), where mba denotes the variable of type Multilevel BoxArray and dim represents the dimension field in this structure.

– multifab physbc.f90: This file contains the boundary conditions for periodic, aperiodic or exterior boundaries etc. The way most codes are written in BoxLib does not require the updates of the physical boundaries using multifab physbc(). Since we use a cell-centered

scheme, we must update the Dirichlet boundary conditions and hence we incorporate the condition u0+ug

2 = ua for all the boundaries of a 3-D unit cube. Here u0, ug and ua are the near-to-boundary data, ghost data lying outside the actual boundary and the actual boundary condition, respectively, as explained in the introduction of Section 5.7. This subroutine is called from several other subroutines and hence it becomes necessary to change that subroutine call to point to our version of this subroutine. Further, since we use the mesh resolution in setting the boundary conditions, an array containing equal elements in 2-D or 3-D must be passed to this subroutine. This is necessary in the discretized version of the problem because the loop indices by themselves do not translate to the actual distance on the physical domain. Since our implementation is cell-centered, we add a value of half to the loop index and then multiply it with the mesh resolution to get the actual distance on the physical domain.

– fillpatch.f90: This file contains several subroutines which call the multifab physbc() subroutine. Since we modify the latter routine for non-cubic blocks and carry out Dirichlet boundary updates for cell-centered implementation, we need to change the invocations of this subroutine in this particular file. The first change comes in the signature of the subroutine fillpatch() which must now also contain the mesh spacing for the fine (dx fine) and coarse (dx coarse) grid and also the lower (prob lo in) coordinates of the problem domain. The fine and coarse mesh spacings are passed as a scalar and the lower coordinates of the problem domain is passed as a vector having three components (as we implement 3-D problems). The scalars must be copied to vectors having three components (which are equal), as the call to our version of multifab physbc() requires vectors. Depending on the call to the fine or coarse grid, the appropriate vectors are passed into the multifab physbc() subroutine.

– multifab fill ghost cells.f90: The routine multifab fill ghost cells() contained in this file calls the subroutine fillpatch() in fillpatch.f90. Thus, we need to change the signature of multifab fill ghost cells() to address this change. We again pass the fine grid mesh spacing, coarse grid mesh spacing and the lower coordinates of the problem domain to multifab fill ghost cells(), which are then passed to the fillpatch() subroutine and also utilized in calls to multifab physbc() subroutine. As mentioned above we rename the subroutines by adding an appropriate prefix, rename the files and copy them to the working directory.

5.9

Experimental Results

We now discuss the experimental results for the test problems that we described for single uniform grids and AMR. It should be noted that for comparison of various topologies the application was executed on the same set of cores to eliminate process placement issues. Along

(a) Topology 1 × 4 × 6, cells = 24 × 6 × 4 (b) Topology 4 × 3 × 2, cells = 6 × 8 × 12 Figure 5.10: Sub-domain shapes/sizes resulting from two of several MPI Cartesian Topologies on a 243 domain possible using Listing 5.3

with comparing the execution timings, we extract and compare the cache-misses for both single grids and AMR. Since for AMR there are many sub-routines that are called by BoxLib, we only present the aggregate sum of cache-misses in the top-level subroutine. The test platform for all the tests is the ARC3 facility described previously (see Chapter 3).