Set-up

After initializing BoxLib with boxlib initialize(), we need to specify the domain size (n cell), the maximum number of AMR levels allowed (max levs) and the dimension of the problem (dim). We impose that an entire box is refined even if there is a single tagged cell in it (i.e. true block structured AMR behaviour). Further, there is no safe layer i.e. if a block is refined, no cells outside the block are tagged for refinement. Thus, amr buf width is set to 0. For a cubic block, the max grid size variable can be set such that the box has equal cells in all directions and n cell is generally perfectly divisible by max grid size. If the box-shape is non-cubic, then the sizes are specified in a 1-D array having three elements namely, chunk dims(1:3). Thus, for a cubic block, chunk dims(1:3)=max grid size. The cluster min eff indicates the minimum fraction of cells which must be refined so that the en- tire block is refined. Thus, since the total cells in a box are chunk dims(1) * chunk dims(2) * chunk dims(3), the cluster min eff is set to

1.0

chunk dims(1) ∗ chunk dims(2) ∗ chunk dims(3).

Setting the value of cluster min eff in such a way should have the same effect as tagging all the cells in a block in the file tag boxes.f90. We stress that it is important to experi- ment with both these factors and verify the result after plotting and visualizing the domain with a visualization package such as VisIt [140]. The BoxLib user manual [92] defines the cluster minwidth as the minimum number of cells in each direction of the newly formed grid. This is defined as a scalar variable in BoxLib. Thus, the implementation and the User Guide assumes that the user generally uses cubic-blocks to minimize communication and hence the cluster minwidth remains the same in all directions. This is not the case with non-cubic blocks and hence ideally the cluster minwidth should be a dim-dimensional vector depend- ing on the dimensionality (dim) of the problem. Our correspondence with the developers of BoxLib made it clear that they might make the cluster minwidth a vector in the future implementations but it was not a priority as most users used cubic-block sizes. Thus, for

Table 5.1: Set-up Variables: Declared variables during Set-up phase

Variable Meaning

lo(3), hi(3) Low, High end of box is periodic(3) Periodicity in each direction prob lo(3), prob hi(3) Physical domain of problem

phys bc(dim,2) Physical boundary types dx(max levs) Mesh spacing at each level phi(max levs), oldphi(max levs) Jacobi update MultiFabs

rhs(max levs) RHS array

la array(max levs) Layout array at each level error(max levs) Error MultiFab at each level

our experiments when chunk dims(1) 6= chunk dims(2) 6= chunk dims(3) and the value of chunk dims(2) lies between chunk dims(1) and chunk dims(3), we set the cluster minwidth to 2 * chunk dims(2). The result was always verified by observing the resulting box-shapes with VisIt and the chosen non-cubic block shape was consistently observed at all levels of refinement. The cluster blocking factor gives a value such that all newly formed grid di- mensions in each direction are divisible by cluster blocking factor. This was set equal to the cluster minwidth after experimentation. To summarize, it requires experimentation and visualization to determine the appropriate values of the factors cluster min width and cluster blocking factor to ensure that the non-cubic block size is maintained at all levels of refinement.

The boundaries in our experiments are all Dirichlet boundaries and their values can be set in the file bc.f90. Further, the Dirichlet boundary is represented by the integer 15 by BoxLib. 3-D arrays (or MultiFabs) are allocated to represent various conditions such as the low (minimum indices of coordinates) and high end (maximum index of coordinates) of a box, its physical dimensions etc., as shown in Table 5.1. The physical domain of our test problem (prob lo(3), prob hi(3)) ranges from 0 to 1 in each direction (unit cube). The mesh spacing for the coarsest grid is calculated as prob hi(1)−prob lo(1)_{n cell} . Since we use a unit cube domain, the direction we use to calculate the mesh spacing does not matter.

Built-in subroutines are called to set the cluster minwidth, cluster blocking factor and the cluster min eff. The refinement ratio between levels is set to two by using the amr ref ratio init(max levs,dim,2) subroutine call. Since AMR has multiple levels, in- stead of building a single BoxArray object, a multilevel BoxArray object is built using the subroutine call ml boxarray build n(mba,max levs,dim), where mba is of type ml boxarray i.e. a Multilevel BoxArray. The refinement ratio must be passed to the mba variable using ml boxarray set ref ratio(mba) and this uses the ratio set by the amr ref ratio init call above. The default ratio used by BoxLib is two, i.e. the resolution/mesh-spacing of the grid at

level 2 is half that of the mesh at level 1. In general, the mesh spacing at level l +1 is half that of at level l. The subroutine bc tower init() allocates the array defined in define bc tower.f90 but does not initialize it. In general, the bc tower datatype defines the boundary conditions at each level. Thus, we must pass the max levs, dim and phys bc to the initializing subroutine. The Multilevel BoxArray object mba has a field named pd which must point to the Box at a particular level. Thus, we set mba%pd(1)=bx, where the variable bx represents the box spanning the initial given domain. This is followed by setting the mba%pd(level) to appropriate boxes that are obtained by refining the box at level 1. Another field in the mba object gives the size of the boxes at a particular level and is given by mba%bas(level). Since initially the whole domain is represented as a single box at level 1, we execute boxarray build bx(mba%bas(1),bx) to internally set mba%bas(1)%bxs(1)=bx. The call boxarray maxsize(mba%bas(1),chunk dims) breaks the box at level one into boxes having sizes chunk dims(1:3).

The first level layout can now be built using the layout build() subroutine by passing the multilevel box array pointer, the problem domain and the periodicity of the problem. The boundary conditions tower can be built next using the call to bc tower level build() and passing into it the layout array which was built in the previous step. Next all the MultiFabs are allocated memory and the initial solution/guess is initialized to zero. MultiFabs can be passed into user-defined low level subroutines which are accessed as either 2-D or 3-D arrays, element by element in the column major order.

When statically refining the grid, as in our test problem, the refinement criteria is tested to see if the base level grids/ coarsest grids need to be refined. If this is the case then they are re- fined and the new grids are built by invoking the make new grids() subroutine. This is carried out along with allocating new MultiFabs for the newly created levels and addition of the bound- ary conditions to that level using the bc tower level build() subroutine. The MultiFabs are rebuilt again if the proper nesting conditions are violated and this nesting is forced using the enforce proper nesting() subroutine. Since the number of levels now may be less than the max levs specified earlier, the restricted layout is built using ml layout restricted build(). The Set-up phase is not trivial, especially when the proper nesting condition is violated which results in re-building of data structures. This leads us to the next phase where the solution is approximated at each iteration i.e. the Solve phase.