Adaptivity and Dynamic Load Balancing
2.7 Two Further Paradigms
We complete this chapter with a discussion of two more algorithms concerning the application of dynamic load-balancing to adaptive solvers. These have been selected because they both relate to the use of mesh adaptivity for the parallel solution of time-dependent problems in three space dimensions and are hence of great practical interest.
2.7.1 Algorithm of Oliker & Biswas
In [77] Oliker and Biswas present an important method which dynamically min- imises the amount of load imbalance which arises due to the adaptive nature of a particular solver used to solve a given class of CFD problems on parallel machines. This appears to be the very rst complete algorithm which can accomplish all of the typical phases associated with such an adaptive approach; mesh adaptation, repar- titioning, processor assignments, and remapping are all done rapidly and eciently so as not to cause a signi cant overhead to the numerical simulation.
For the repartitioning stage they use ParMETIS ([77]). As far as the processor assignment stage is concerned they make use of two cost functions: TotalV and MaxV. TotalV minimises the total volume of data moved along all processors, while MaxV minimises the maximum ow of data to or from any single processor.
They nally execute a remapping phase which is responsible for physically mov- ing data when it is reassigned to a di erent processor. This remapping phase is further divided into two sub-phases: marking and subdivision. In the marking stage the edges are simply marked for bisection (based on an error indicator). Once the marking stage is complete, the weight of the dual graph can be adjusted and based on the new weights the load balancer may proceed in generating a new partition- ing. The newly redistributed mesh is then subdivided (and subsequently re ned) based on the marking patterns. Since the actual re nment is performed only after the subdivision stage it is believed that a relatively small amount of data must be moved.
2.7.2 Algorithm of Vidwans
et al.
In [104] Vidwans et al. present their own divide-and-conquer based algorithm de- signed to solve three-dimensional Navier-Stokes problems on unstructured meshes in parallel using adaptive techniques.
The initial computational domain is partitioned among the available processors (which they assume is a power of two) by a partitioning algorithm based on the orthogonal recursive bisection method. At some later stage of the solution process when there is a load imbalance due to adaptivity they split the processors into two equal groups based upon their IDs. The group with the higher load is termed as the
sender groupwhilst the other is known as thereceiver group. After that half of the
di erence of loads is transferred from thesender groupto that of thereceiver group. The recursion is continue in the sense that each of the two groups at a given stage are further divided into two subgroups. The recursion is terminated when the size of a particular group becomes 2, in which case the processors simply balance the sum of their individual loads by exchanging the loads across their common boundary. The algorithm is deterministic in the sense that recursion will terminate after log
kPk steps, with kPk being the number of processors, irrespective of the amount of
imbalance and its distribution across the processors.
As far as the migration of nodes (also known as cells) from the sender groupto
the receiver group are concerned, they employed two di erent approaches.
The grid-connectivity-based approach. In this approach they start with those
cells which have at least one face on the initial interprocessors boundary. After that they select those cells which are neighbours of the cells selected in the previous stage. The process is repeated until they have the desired number of cells to be migrated. This is obviously done in \layers" starting from the outermost layers of cells. In adapted regions a large number of layers relatively occupy small physical space. On the other hand in the dense region of the mesh small number of layers occupy the large physical space. So this method may leads to jagged and long boundaries.
The coordinate-based approach. In this approach all those cells which have
their centroids within a particular region in 3-d space are marked for migra- tion. This region is de ned to be adjoining the interprocessors boundary. As
the number of cells in the region is not known a priori, the width of the re- gion has to be determined by trial-and-error approach so that it contains the required number of cells. They say that this method is better than the grid- connectivity-based approach. This trial and error approach requires manual intervention not only for di erent meshes, but also for a given mesh the man- ual intervention is required for ever changing boundaries of the partitions. So a dynamic load balancing algorithm based on coordinate-based approach can never be a robust one and also the trial and error nature of it will make the algorithm less ecient.
Apart from improving the selection criteria for migrating cells, there are two other steps in the algorithm which can be improved.
The restriction that at each stage the two subdomains should have same
number of processors may reduce the quality of the mesh-partition especially if the mesh is non-uniform in nature. If this restriction can be relaxed in a meaningful way than this may improve the quality of the mesh-partition.
The division of a group into two groups which is based simply on the IDs of
the processors in the original group may also reduce the quality of the load- balancer especially if this produces groups in which some of the processors are physically unrelated from the other processors in the group.
The new dynamic load balancing algorithm presented in the next chapter uses this philosophy of Vidwans et al. together with a signi cant number of improvements. There, the two subgroups are not required to have the same number of processors and the division of a given group into two groups is not simply based on the IDs of the processors present in the group. Instead a sorted version of the Fiedler vector is used for the purpose of bisection. Also a gain-based approach (see Chapter 3) is used as the basis for the purpose of migrating cells.
Experimentation shows that these three steps improve the quality of the new partitioner, especially in the case where the underlying mesh is of a non-uniform nature (i.e. in some region of the mesh the elements are much ner and in other parts of the region they are relatively coarse).