Note that the dot operation obtains an inconsistent scalar, and therefore a global synchro- nization/communication is required to obtain the corresponding consistent value.
preconditioner application: As the definition of the recursion is maintained, the operations to apply the preconditioner, explained in the previous chapter, remain valid. However, to complete the recursion step in the task-parallel case, the preconditioner DAG has to be crossed twice per solve zj+1 := M−1rj+1 at each iteration of the PCG: once from bottom to top, and a second time from top to bottom. Each transition requires a synchronization/ communication step, adding vectors when the DAG is traversed upward, and copying vectors in the reverse case. Moreover, the concurrency increases/decreases as we move towards/away from the leaves. In this operation, the preconditioner (M ) is consistent, the operand vector is inconsistent, and the result vector will be consistent.
transforms: It is possible to change the distribution of each operand as follows. For example, if we need a data structure with a specific type of consistency, and this is the result from a previous operation, but with an inappropriate consistency type, we can convert the structure. We have two types of transforms:
xi → xc: To change from inconsistent to consistent, we proceed as in the preconditioner application, but with M = I. Thus, no changes are made inside each node, and only the additions and copies related to each DAG transition are applied.
xc→ xi: The transformation of a data structure from consistent to inconsistent only requires a local adjustment.
In Figure 4.6 we illustrate the PCG algorithm with the combination of different types of dis- tribution (consistent and inconsistent) for each operation, and the additional reductions required to complete the dots. Moreover, we have changed the order of some operations to merge the two operations that cross the DAG (O06 + O10).
Note that the number of leaves in the DAG grows exponentially with the number of nested dissection steps, so that the degree of concurrency can be easily increased by expanding additional levels. However, there exists a balance between the number of levels, that determines the number of independent tasks, and the convergence rate of the procedure. Concretely, each dissection step introduces additional numerical levels in the computation, yielding both a different DAG and a distinct preconditioner. While the numerical properties of all these preconditioners are similar, in many cases the number of iterations of the PCG solver grows significantly after a few levels (8 and more) are expanded.
4.2
Parallel Programming Models
The past few decades have witnessed a steady advancement in computer performance. This improvement is due, among other factors, to the introduction of parallel architectures. Micropro- cessors drove performance increments and cost reductions in computer applications for more than two decades. However, around 2003, the performance increases stopped because heat dissipation and energy consumption issues limited the CPU clock frequencies and the number of tasks that can be performed within each clock period [68]. The solution adopted was to switch to a model where the microprocessor has multiple processing units, known as cores [108].
Nowadays, there are multicore architectures integrating a few cores into a single microprocessor, and manycore architectures consisting of a large number of cores. Task parallelism is among the best alternatives to take benefit of the additional hardware concurrency in these new architectures.
O0. A → M O0. Preconditioner computation Initialize r0, p0, x0, σ0, τ0; j := 0
while (τjc> τmax) Iterative CG solve
O01. vji := Aipcj O01. spmv
O02. δij := (pcj)Tvji O02. dot
Reduction (δi j)
O03. αcj := σjc/δcj O03. scalar operation
O04. xcj+1:= xcj + αjpcj O04. axpy
O05. rij+1:= rji − αjvji O05. axpy
O06. zj+1c := (Mc)−1rj+1i O06. preconditioner application
rcj+1← ri
j+1 Transformation
O07. σj+1i := (rij+1)Tzcj+1 O07. dot
O10. τi
j+1:= (rc)Tj+1rij+1 O10. 2-norm
Double Reduction (σij, τji)
O08. βjc:= σcj+1/σjc O08. scalar operation
O09. pcj+1:= zcj+1+ βcjpcj O09. xpay (similar to axpy)
j := j + 1 end while
Figure 4.6: Algorithmic formulation of the PCG method taking into account the consistency of the data structures.
Therefore, the applications have to be rewritten by the developer, partitioning the workload into tasks, and mapping these tasks to the workers (cores).
Traditionally, parallel systems have been divided into two broad categories: shared memory and distributed memory [176]. The first type provides a single memory address space accessible to all the processors. In the second type, instead of having a global address space, each processor has its own memory. More recently, hybrid shared-distributed memory systems have been built, combining the features of both architectures.
The conventional parallel programming models include a pure shared memory model [68, 176] called OpenMP [150, 58, 59], and a pure message-passing model [68, 176] named MPI [90, 91, 152]. However, today it is common to mix both shared and distributed memory models to improve the performance on current hybrid architectures. In addition, the availability of General Purpose computation on GPUs, and other manycore accelerators, has lead to the Heterogeneous Parallel Programming (HPP) model, which takes advantage of the capabilities of multicore CPUs and manycore GPUs.
The objective of this dissertation is the parallelization of ILUPACK on multicore and manycore processors/accelerators. For that purpose, we first developed a data-flow version of this code with considerable levels of thread-concurrency using OmpSs, and then we implemented a hybrid version of ILUPACK using MPI+OmpSs to exploit the benefits of each programming model.
In this section we revisit the parallel programming models which are the target of our work: OpenMP, OmpSs, and MPI.
4.2. PARALLEL PROGRAMMING MODELS
4.2.1 OpenMP
OpenMP [58, 59] is a shared-memory API that provides a portable and scalable model to facilitate shared-memory parallel programming. OpenMP is implemented as a combination of a set of compiler directives (#pragmas), and a runtime providing both management of the thread pool and a set of library routines. These directives can be added to a sequential program in Fortran, C, or C++ to describe how the work is to be shared among the threads that will execute on different processors or cores and to orchestrate accesses to shared data as needed. Therefore, OpenMP requires specialized compiler support to understand and process these directives.
The use of threads is highly structured in OpenMP because it was designed specifically for parallel applications. In particular, the switch between sequential and parallel sections of code follows the fork/join model. Thus, when a parallel region is reached, a single thread of control splits into a number of independent threads (fork), and the sequential execution is resumed when all the threads have completed the execution of their tasks (join). OpenMP is specially suited to exploit loop parallelism, and from version 3.0, it can also address the task parallelism.
The success of OpenMP can be attributed to a number of factors. One is its strong emphasis on structured parallel programming. Another is that OpenMP is comparatively simple to use, since the burden of working out the details of the parallel program is up to the compiler. It has the major advantage of being widely adopted, so that an OpenMP application will run on many different platforms from small to large Symmetric Multi-Processing (SMP) architectures and other multi-threading hardware. Nowadays, the recent versions of GNU and Intel compilers give support to the OpenMP standard.
4.2.2 OmpSs
OmpSs [2, 74] is a task-based programming model developed at Barcelona Supercomputing Center (BSC). The model supports automatic detection of data dependencies between tasks, deter- mined at execution time by hints given in the form of directionality clauses embedded into compiler directives. Armed with this information, a task graph is generated during the execution that is then scheduled by the OmpSs runtime to the cores, exploiting the inherent task parallelism. This feature is now also integrated in the OpenMP 4.0 standard.
The use of data-dependencies between the different tasks of the program in OmpSs enables asyn- chronous parallelism. When a function is annotated with the task construct, each invocation of that function becomes a task creation point. The task construct allows to express data-dependencies using in (standing for input), out (standing for output) and inout (standing for input/output) clauses to this end. Each time a new task is created, its in and out dependencies are matched against those of existing tasks. If a Read-after-Write (RaW), Write-after-Write (WaW) or Write- after-Read (WaR) dependency is found, the task becomes a successor of the corresponding tasks. This process creates a task dependency graph at runtime, so that tasks are scheduled for execu- tion as soon as all their predecessors in the graph have finished, or at creation, if they have no predecessor.
OmpSs can be combined with MPI, providing a highly asynchronous programming model where communication tasks are overlapped with computation tasks avoiding the need for global synchro- nization. Another interesting property of OmpSs is the support for heterogeneous platforms, as it can combine CUDA or OpenCL in regular C/C++ or Fortran codes, relieving the programmer from performing data transfers, allocating memory or even compiling the kernels, which are all automatically performed by the OmpSs runtime. OmpSs has been successfully applied to appli-
cations of different nature, and especially in complex scientific codes targeted in projects such as Montblanc [138], DEEP [64] and INTERTWinE [115].
4.2.3 MPI
MPI [90, 91, 152] is a standard library interface for writing parallel programs. Its specification uses message-passing operations where communication between processes is done by ex-changing messages. Although it is oriented to distributed systems and manycore architectures, it can also be employed in shared-memory architectures and multicore processors, expanding its area of use. The goal of MPI is to establish a portable, efficient, and flexible standard for message-passing that will be widely adopted. In fact, it has become the “industry standard” for writing message-passing programs on HPC platforms. MPI favors the Single Program Multiple Data (SPMD) and the Master/Worker program structure patterns.
In the message-passing model, the processes executed in parallel have separate memory address spaces, and the programmer has to handle the tasks which are computed by each process. Hence, communication occurs when part of the address space of one process is copied into the address space of another process. This operation is done cooperatively when the first process executes a send operation and the second executes a receive operation. Communication modes in MPI comprise point-to-point, collective, one-sided (since MPI-2), and parallel I/O operations.
In conclusion, MPI is well suited for applications where portability, both in space and in time, is important. MPI is also an excellent option for task-parallel computations and for applications where the data structures are dynamic. Today, there exist several public implementations of the MPI standard, among which MPICH [140], OpenMPI [151] and MVAPICH2 [142] stand out.