Note that we did not claim that we have solved the problem of architecture- and workload- aware graph partitioning. Instead, the most important contribution of this thesis is a demon- stration of the importance of architecture-awareness (heterogeneity and contentiousness) for distributed graph computation on modern HPC clusters as well as a set of possible solu- tions. In the rest of this section, we will discuss the limitations of our proposed solutions. Addressing these limitations is left as future work.
Time Complexity Comparison Ideally, we would like to include a formal comparison of the time complexity for different graph (re)partitioners. However, for distributed graph (re)partitioners, the (re)partitioning overhead may rely more on the amount of communi- cation required by the (re)partitioning algorithm. As a result, the time complexity may not always be a good indicator to look at. The fact that many graph (re)partitioners sim- ply do not have such complexity analysis in their published papers further increases the difficulty of comparison. Additionally, the memory usage of the graph (re)partitioners is an- other important factor one should consider while choosing the (re)partitioners. Many graph (re)partitioners simply do not work with large graphs either because it takes too much time to compute a partitioning or it consumes too much memory. To a certain point, we have made an indirect comparison of this in Table2.1. The lightweight property we talked about covers all the three dimensions: time complexity, space complexity, and data communication. Heterogeneity-Awareness Modeling Clearly, all our proposed architecture-aware graph (re)partitoiners require a fairly accurate measurement of the relative network communication costs among the computing elements. Although Planar+ has provided an efficient way to measure the costs, its effectiveness was only evaluated on a relatively small cluster with a simple network topology in our experimental study. For larger clusters with complicated network topologies, the solution may not work well. In addition to this, the variability of the relative network communication costs may increase, as the size of the cluster and the complexity of the network topology increase. Nevertheless, both Paragon and Planar+ allow customized solutions for relative network communication cost measurement.
Contention-Awareness Modeling Another limitation of our proposed solutions is that we require users to do some profiling of the target workload on the computing infrastructures to determine the ideal λ value, the degree of contentiousness. The parameter λ depends on many factors, such as the characteristics of the graph dataset that the workload is operating on, the characteristics of the graph algorithm, as well as the characteristics of the underlying computing infrastructures. As a result, the profiling may become an extra burden for the users. In addition to this, the set of computing elements used for profiling may be different from the set of computing elements used for actual computation in many cases, and users may not have control over this. Thus, one of the things people could look at is to automate this profiling process (using some machine learning techniques for example).
Workload-Awareness Modeling Currently, we encode the characteristics of the target workload into weights/sizes of the vertices and edges for (re)partitioning. Each vertex/edge of the graph can only be assigned with a single weight/size. However, vertices of the graph may have different computation and communication requirements in different time periods of the computation. Thus, a single weight/size could not always accurately reflect the requirements. This could also be a part of the future work people could explore. In fact, the modeling of the computation and communication requirements of the workload is also not an easy task, especially for scientific simulations.
Additionally, we use the knowledge of the execution traces for skew-resistant graph par- titioning. However, the solution we adopted in Sargon is somewhat simplistic. A more complete solution (e.g., a machine learning model to predict the characteristics) is required to increase the potential benefit/generality of Sargon.
Number of Partitions In the current implementation of our proposed solutions, especially Paragon, Planar, and Planar+, we assume that the number of partitions remains the same as int the initial partitioning. However, users may sometimes want to repartition the graph into a different number of partitions. To the best of our knowledge, a large body of existing graph repartitioners, including ours, do not support this. This could also be an interesting problem to look at. Along these lines, it would also be interesting to examine the problem of how to determine the optimal number of partitions for a given workload and a
given graph dataset.
In addition to this, all the experimental studies of our proposed solutions were designed and performed with the assumption of one partition per core in mind. Sometimes, it may make sense to do over decomposition, that is, assigning more than one partitions to a core. Theoretically, all our proposed solutions can also work in the over decomposition case. How- ever, it has not been thoroughly evaluated.
Vertex-Cut Based Graph Partitioning The focus of this thesis is the edgecut-based graph partitioning problem, where vertices of the graph are distributed across partitions by cutting edges if needed. However, it would also be interesting to investigate the architecture- and workload-aware vertex-cut based graph partitioning problem, where edges of the graph are assigned to partitions by cutting vertices if needed.