While each of these parallel computing architectures is different in nature and each has individual challenges when programming applications for it, they are conceptually similar. All of the parallel architectures contain multiple cores which read data, execute instructions and write to memory. The architectures can be separated into two categories - shared- and distributed-memory machines.
Shared-memory machines store all the data in a single shared memory location which the cores read from and write to. The threads executing on these cores must synchronise with each other to ensure correct computation. Distributed memory machines have separate memory locations which may be accessed by only one or more cores. Threads running on these machines must synchronise with each other and exchange data through some communication channel.
These parallel architectures require the use of a parallel programming language and library. Mul- tiple languages/libraries may be required for hybrid parallel machines. These languages are all different but once again have a number of similarities. To determine how these similarities can be exploited to automatically construct finite-difference simulations using these languages, the specific implementations of the simulations must be reviewed. These implementations are discussed and compared in Chapter 5.
4.8 Conclusions
A number of popular parallel computing architectures have been introduced and described. Some of the languages and libraries used for implementing parallel programs on these architectures have been presented. Each of these architectures and languages has individual strengths and weaknesses in terms of cost, performance, program complexity and scalability.
To utilise the computational processing power of these architectures for simulating computational models, the simulations must be decomposed into tasks which can be processed in parallel. The exact manner of this parallelisation depends on the simulation, the architecture and the language/libraries used to implement them. Algorithms for computing these simulations in parallel are presented in Chapter 5.
The architectures and languages presented in this chapter are only a sample of the current tech- nology available. It can be expected that new languages and architectures will emerge in the future. Thus if code for these simulations is to be generated, the system must be capable of producing code for any language or architecture but also be easily extensible to the new languages which will be developed in the future.
“An algorithm must be seen to be believed.”
Donald Knuth
5
Parallel Algorithms
5.1 Introduction
he numerical methods discussed in Chapter 3 describe how the PDEs in Chapter 2 can be numerically discretised in time and space. For these simulations to be computed, the numerical equations must be implemented in programming code. This code must deal with creating, storing and managing the lattices representing the system, computing the equations and integrating the system over time.
The code to compute the equations can be easily written using mathematical operators and code to access the lattice values required by the finite-differencing stencils. Many different integration methods can be used to integrate these systems as discussed in Chapter 3. Each of these methods have different requirements in terms of computational cost and memory usage. Multi-stage methods may require intermediate lattices to be stored and additional stages computation. Algorithms that compute simulations using these multi-stage methods must consider the order of computation and data synchronisation to ensure correct results.
Computing simulations with large lattice sizes and many time steps required to show certain behaviour can be a very computationally-intensive task. The correct use of parallel architectures and languages such as those discussed in Chapter 4 is vital to producing simulation results in a reasonable time-frame. These simulation implementations require the use of various languages and methods of splitting the computation.
To automate the process of constructing these simulation implementations, the common features and the language specific features must be identified. All the different implementations must split the computation into tasks which can be computed in parallel. Identifying which parts of the simulation computation can be parallelised will be common to all the simulations and each implementation will require one of the decomposition methods discussed in Section 5.5. The implementations will also require language and architecture specific supporting code which will not be common to other implementations. Identifying these features is vital to determining how to automatically generate simulations.
CHAPTER 5. PARALLEL ALGORITHMS
tions on parallel computers [23]. This work includes implementations on Distributed Comput- ers [21, 57–59, 136, 137], Multicore Processors [138], Graphical Processing Units [53, 56, 139] and GPU clusters [127, 135] with various methods of parallel decomposition [140, 141]. This chapter discusses both sequential and parallel implementations of finite-differencing simulations. Many of these im- plementations have been discussed in previous publications [75,78,102,125–127,142] but are collated here.
5.2 Equation Computation
To create any sequential or parallel simulation, the first necessary functionality is to compute the equations for a given system. The numerical methods discussed in Chapter 3 transform the equations into a discrete form which can be stored and calculated by computer. All of the architectures and languages presented in this thesis use C-style syntax. These languages were chosen to make it easy to compare different implementations on different architectures and because of the author’s personal preference. One of the advantages of this approach is that evaluating the equations for a system can be performed using almost the same code. The code fragments that evaluate the three equations from Chapter 2 are presented in the following sections.
5.2.1
Cahn-Hilliard Equation
In order to compute the Cahn-Hilliard equation, the neighbouring values required by the discrete stencils must first be fetched from memory. The Cahn-Hilliard equation contains a biharmonic sten- cil which requires not just the nearest neighbours to be fetched from memory but also their nearest neighbours. These values are then substituted into the equation with the appropriate coefficients from the stencil to compute the equation for a particular cell. Listing 5.1 shows the code that com-
putes the Cahn-Hilliard equation for a single cell(x, y)in a two-dimensional lattice with dimensions
(X, Y). The code has been formatted to make the stencil structure easily visible.
Listing 5.1: The C-syntax code to evaluate the Cahn-Hilliard equation at position(x, y)for a discrete
two-dimensional floating-point latticeuwith dimensions(X, Y).
f l o a t u ym2x = u [ ( y−2)∗X + x ] ; f l o a t u ym1xm1 = u [ ( y−1)∗X + ( x−1 ) ] ; f l o a t u ym1x = u [ ( y−1)∗X + x ] ; f l o a t u ym1xp1 = u [ ( y−1)∗X + ( x + 1 ) ] ; f l o a t u yxm2 = u [ y ∗X + ( x−2 ) ] ; f l o a t u yxm1 = u [ y ∗X + ( x−1 ) ] ; f l o a t u yx = u [ y ∗X + x ] ; f l o a t u yxp1 = u [ y ∗X + ( x + 1 ) ] ; f l o a t u yxp2 = u [ y ∗X + ( x + 2 ) ] ; f l o a t u yp1xm1 = u [ ( y+1)∗X + ( x−1 ) ] ; f l o a t u yp1x = u [ ( y+1)∗X + x ] ; f l o a t u yp1xp1 = u [ ( y+1)∗X + ( x + 1 ) ] ; f l o a t u yp2x = u [ ( y+2)∗X + x ] ; M∗( (−B∗( u ym1x + u yxm1 + (−4∗u yx ) + u yxp1 + u yp1x ) ) +
(U∗( ( u ym1x∗u ym1x∗u ym1x ) +
5.2. EQUATION COMPUTATION
( u yp1x∗u yp1x∗u yp1x ) ) ) −
(K∗( u ym2x +
(2∗u ym1xm1 ) + (−8∗u ym1x ) + (2∗u ym1xp1 ) + u yxm2 + (−8∗u yxm1 ) + ( 20∗u yx ) + (−8∗u yxp1 ) + u yxp2 +
(2∗u yp1xm1 ) + (−8∗u yp1x ) + (2∗u yp1xp1 ) + u yp2x
) ) ) ;
It should be noted at this point that this code does not account for boundary conditions. Executing this code without extra handling for boundary conditions could cause array index out of bounds errors. Implementing boundary conditions in code is discussed in Section 5.3.
5.2.2
Ginzburg-Landau Equation
The Time-Dependent Ginzburg-Landau equation can be computed in a similar fashion. The TDGL equation uses the smaller Laplacian stencil which requires only the nearest-neighbour values to be
fetched from memory. It should be noted thatuis a lattice of complex numbers and this code assumes
a complex number class where appropriate operators have been provided. The code to compute the TDGL equation can be seen in Listing 5.2.
Listing 5.2: The C-syntax code to evaluate the Time-Dependent Ginzburg-Landau equation at posi-
tion(x, y)for a discrete two-dimensional complex latticeu.
complex u ym1x = u [ ( y−1)∗X + x ] ; complex u yxm1 = u [ y ∗X + ( x−1 ) ] ; complex u yx = u [ y ∗X + x ] ; complex u yxp1 = u [ y ∗X + ( x + 1 ) ] ; complex u yp1x = u [ ( y+1)∗X + x ] ;
− ( P/ i )∗( u ym1x + u yxm1 + (−4∗u yx ) + u yxp1 + u yp1x ) − ( q/ i )∗ ( abs ( u yx∗u yx )∗u yx ) + y∗u yx ;
If there is no complex number class available, the real and imaginary parts of the model must be
computed as separate equations. Such a system must be stored as two separate lattices (u randu i)
representing the real and imaginary parts of the field. Performing the computation in this way can also offer performance benefits on some architectures [78]. The code to compute the model in this separated form is given in Listing 5.3.
Listing 5.3: The C-syntax code to evaluate the Ginzburg-Landau equation at position(x, y)using
two separate lattices and calculations for the real and imaginary parts. The system is stored in two
floating point latticesu randu i.
f l o a t uym1x r = u r [ ( y−1)∗X + x ] ; f l o a t uyxm1 r = u r [ y ∗X + ( x−1 ) ] ; f l o a t uyx r = u r [ y ∗X + x ] ; f l o a t uyxp1 r = u r [ y ∗X + ( x + 1 ) ] ; f l o a t uyp1x r = u r [ ( y+1)∗X + x ] ; f l o a t uym1x i = u i [ ( y−1)∗X + x ] ; f l o a t uyxm1 i = u i [ y ∗X + ( x−1 ) ] ; f l o a t u y x i = u i [ y ∗X + x ] ;
CHAPTER 5. PARALLEL ALGORITHMS
f l o a t uyxp1 i = u i [ y ∗X + ( x + 1 ) ] ;
f l o a t uyp1x i = u i [ ( y+1)∗X + x ] ;
− p i∗( uym1x r +
uyxm1 r + (−4∗uyx r ) + uyxp1 r + uyp1x r )
− p r∗( uym1x i +
uyxm1 i + (−4∗u y x i ) + uyxp1 i + uyp1x i )
− q i∗( uyx r∗uyx r∗uyx r + u y x i∗u y x i∗uyx r )
− q r∗( uyx r∗uyx r∗u y x i + u y x i∗u y x i∗u y x i ) + ( y∗uyx r ) ;
p r∗( uym1x r +
uyxm1 r + (−4∗uyx r ) + uyxp1 r + uyp1x r )
− p i∗( uym1x i +
uyxm1 i + (−4∗u y x i ) + uyxp1 i + uyp1x i )
+ q r∗( uyx r∗uyx r∗uyx r + u y x i∗u y x i∗uyx r )
− q i∗( uyx r∗uyx r∗u y x i + u y x i∗u y x i∗u y x i ) + ( y∗u y x i ) ;
5.2.3
Lotka-Volterra Equation
Finally the code to compute the Lotka-Volterra equation is given in Listing 5.4. This code uses the
Laplace operator but now there are two latticesu0andu1which approximate the system. To com-
pute the equation for the cell at position (x,y), the neighbouring values from both lattices must be fetched. Two separate computations are required to update the cell in both of these lattices. This is similar to the TDGL implementation which stores and computes the real and imaginary parts of the system separately.
Listing 5.4: The C-syntax code to evaluate the Lotka-Volterra equation at position(x, y) for two
coupled, discrete two-dimensional floating-point latticesu0andu1.
f l o a t u0 ym1x = u0 [ ( y−1)∗X + x ] ; f l o a t u0 yxm1 = u0 [ y ∗X + ( x−1 ) ] ; f l o a t u0 yx = u0 [ y ∗X + x ] ; f l o a t u0 yxp1 = u0 [ y ∗X + ( x + 1 ) ] ; f l o a t u0 yp1x = u0 [ ( y+1)∗X + x ] ; f l o a t u1 ym1x = u1 [ ( y−1)∗X + x ] ; f l o a t u1 yxm1 = u1 [ y ∗X + ( x−1 ) ] ; f l o a t u1 yx = u1 [ y ∗X + x ] ; f l o a t u1 yxp1 = u1 [ y ∗X + ( x + 1 ) ] ; f l o a t u1 yp1x = u1 [ ( y+1)∗X + x ] ; (A∗u0 yx ) − ( B∗u0 yx∗u1 zyx ) + ( D0∗( u0 ym1x + u0 yxm1 + (−4∗u0 yx ) + u0 yxp1 + u0 yp1x ) ) ; (C∗u0 yx∗u1 yx ) − (D∗u1 yx ) + ( D1∗( u1 ym1x + u1 yxm1 + (−4∗u1 yx ) + u1 yxp1 + u1 yp1x ) ) ;
5.3. BOUNDARY CONDITIONS
5.3 Boundary Conditions
Methods for implementing the three simple boundary conditions discussed in Chapter 3 are pre- sented here. These are relatively simple boundary conditions but they are sufficient for the purpose of these example simulations. There are various possible ways to implement these conditions but only one implementation for each is presented here. The following sections discuss the three bound- ary conditions - Periodic, Dirichlet and Neumann.
5.3.1
Periodic Boundaries
Periodic boundary conditions are very easy to implement in code. All the required neighbouring values are stored in the lattice, only the calculation of their indexes must be changed. If the neigh- bouring index is less than 0 or greater than the size of the lattice, the boundary condition must be applied. This can be performed by adding or subtracting the lattice length to the computed index. In
the x-dimension, the neighbouring index−1becomesX−1andXbecomesX−X.
Listing 5.5 shows a code fragment to apply periodic boundaries to the index calculation in two- dimensions. This code computes the neighbours of a site (x,y) by calculating the two neighbouring
indexes of the site in each dimension (xm1,xp1,ym1andyp1).
Listing 5.5: The C-syntax code to apply periodic boundary conditions in two-dimensions. ym1 = ( y == 0 ) ? Y−1 : y−1;
xm1 = ( x == 0 ) ? X−1 : x−1; xp1 = ( x == X−1) ? 0 : x + 1 ; yp1 = ( x == Y−1) ? 0 : y + 1 ;
The code in Listing 5.5 uses the ternary operator because of its higher performance compared to
ifstatements. This performance different is particularly noticeable for GPU implementations.
5.3.2
Dirichlet Boundaries
Dirichlet boundary conditions can also be applied to a simulation relatively easily. The value of the field outside its boundary is a fixed value or function. If an index is outside the range of the field, the lattice will not be accessed but instead a function is called which returns the boundary value at that point. This may be a fixed value or some computed value.
Listing 5.6 gives an example of how Dirichlet boundaries can be implemented. The code applies
Dirichlet boundaries in two-dimensions and uses four functions (by0(y, x),byY(y, x),bx0(y, x)and
bxX(y, x)) to compute the values on each boundary.
Listing 5.6: The C-syntax code to apply Dirichlet boundaries in two-dimensions. u ym1x = (ym1< 0 ) ? by0 (ym1 , x ) : u [ym1∗X + x ] ;
u yxm1 = ( xm1< 0 ) ? bx0 ( y , xm1 ) : u [ y∗X + xm1 ] ; u yxp1 = ( xp1>= X) ? bxX ( y , xp1 ) : u [ y∗X + xp1 ] ; u yp1x = ( yp1>= Y) ? byY ( yp1 , x ) : u [ yp1∗X + x ] ;
CHAPTER 5. PARALLEL ALGORITHMS
This implementation also uses the ternary operator to either calculate a boundary value or fetch a value from the lattice. Like the periodic boundary implementation, this option was selected for performance reasons.
5.3.3
Neumann Boundaries
Implementing Neumann boundaries are more complex. The exact implementation will depend on the nature of the computation model and the stencils it uses for computation. In general the equa- tion can be reformulated to exclude the lattice site outside the boundary and include the Neumann
αterm. The exact reformulation will depend on the model but the boundary conditions can be im-
plemented by a series ofifstatements.
Listing 5.7 shows the series ofifstatement for necessary to enforce Neumann boundary con-
ditions on a two-dimensional lattice. There are eight possible conditions when the boundaries are necessary. Four for the boundary in each dimension and another four when boundaries in both directions are encountered.
Listing 5.7: The C-syntax code to enforce Neumann boundary conditions on a two-dimensional lat- tice. i f( ( y == 0 ) && ( x == 0 ) ) { / / Top L e f t } e l s e i f( ( y == 0 ) && ( x == X−1)) { / / Top R i g h t } e l s e i f( ( y == Y−1) && ( x == 0 ) ) { / / Bottom L e f t } e l s e i f( ( y == Y−1) && ( x == X−1)) { / / Bottom R i g h t } e l s e i f( x == 0 ) { / / L e f t } e l s e i f( x == X−1) { / / R i g h t } e l s e i f( y == 0 ) { / / Top } e l s e i f( y == Y−1) { / / Bottom } e l s e { / / I n t e r i o r }
5.4 Sequential Implementation
Programming these simulations on a single-threaded CPU is relatively simple. For every step the CPU will iterate over the lattice or lattices and compute the equation for each lattice cell. Using an appropriate integration method it will compute a new value to be written into the cell of another lattice representing the system after the time step. This iteration process may have to be performed several times during each step depending on the number of stages of the integration method. Imple- mentations of three integration methods described in Chapter 3 are presented for the CPU - Euler, RK2 and RK4.
5.4. SEQUENTIAL IMPLEMENTATION
5.4.1
Euler
The Euler integration method is the least-accurate explicit method, but it is fast, has low memory requirements and is easy to implement. Euler only requires one computation stage per time step.
For every time step the CPU will iterate over every cell in the latticeu0and calculate the equation for
that cellf(u0). The new value is then calculated using the Euler method (yt+h=yt+h×f(yt)) and
written to the output latticeu1. This will perform a single simulation time step. The code to perform
this computation can be seen in Listing 5.8.
Listing 5.8: Code for a finite-differencing simulation using the Euler integration method imple- mented for a single-core using C.
void e u l e r (double ∗u0 , double ∗u1 ) {
f o r(i n t i y = 0 ; y<Y ; i y ++) { f o r(i n t i x = 0 ; i x <X ; i x ++) { / / u1 = u0 + f ( u0 )∗h } } }
i n t main (i n t argc , char∗∗ argv ) {
double u0 = new double[Y∗X ] ;
double u1 = new double[Y∗X ] ; . . . f o r(i n t t = 0 ; t< n o s t e p s ; t ++) { c a h n h i l l i a r d e u l e r ( u0 , u1 ) ; swap(&u0 , &u1 ) ; } . . . }
As the Euler integration method only has one stage, only one iteration over the lattice is required. More complex integration methods with multiple stages will require multiple iterations over the lattice to compute the methods. These methods also require additional memory to store the inter- mediate stages while the Euler method only requires two - the input and output lattices. A simple example of a multi-stage Runge-Kutta integration method is the RK2 method.
5.4.2
Runge-Kutta 2
ndOrder
The Runge-Kutta 2ndorder integration method implementation requires an additional iteration over
the lattice and additional memory to store the intermediate stage. Because this method uses the derivative at the midpoint between time steps, the lattice must first be computed. Then the deriva- tive of this midpoint lattice is used for the calculation of the final lattice at the next time step. The implementation of the update function can be seen in Listing 5.9.
Listing 5.9: The single-threaded CPU implementation of the RK2 integration method. void runge kutta 2nd (double ∗u0 , double∗u1 , double ∗u2 ) {
f o r(i n t i y = 0 ; y<Y ; i y ++) { f o r(i n t i x = 0 ; i x <X ; i x ++) { / / u1 = u0 + f ( u0 )∗h / 2 } } f o r(i n t i y = 0 ; y<Y ; i y ++) { f o r(i n t i x = 0 ; i x <X ; i x ++) { / / u2 = u0 + f ( u1 )∗h }
CHAPTER 5. PARALLEL ALGORITHMS
} }
This simulation implementation will take longer to execute and require more memory than the Euler implementation. Because the midpoint field must be first calculated, two iterations over the lattice are required, so this method can be expected to take approximately twice as long. It will also
require an additional lattice to be stored in memory, requiring an additional50%of memory usage
over the Euler method.
5.4.3
Runge-Kutta 4
thOrder
The Runge-Kutta 4thorder method can be implemented in a very similar way to the RK2 method,
but the number of stages is increased. The RK4 method has four stages and requires additional lattice iterations and memory space. Because the function evaluations of the RK4 method are reused in the