Iterative methods

Iterative methods use an initial approximation of the solution to calculate the next approxima- tion. An iterative method is said to converge when the difference between the actual solution and current approximation tends to zero on increasing the iterations [22]. These methods are

useful for large sparse systems of linear equations when the memory and computational time requirements of the direct methods become overwhelmingly high. We now describe the two simplest iterative methods for solving sparse systems of linear equations.

2.3.2.1 Jacobi

The Jacobi method one of the simplest iterative methods. Consider a linear system of three unknown variables having the form Au = f as shown in Equation (2.32).

a11u1+ a12u2+ a13u3= f1 a21u1+ a22u2+ a23u3= f2 a31u1+ a32u2+ a33u3= f3

(2.32)

The ith_{equation in Equation(s) (2.32) can be used to express u}

i in terms of the RHS and the remaining unknowns. This is shown below.

u1= 1 a11 (f1− a12u2− a13u3) u2= 1 a22 (f2− a21u1− a23u3) u3= 1 a33 (f3− a31u1− a32u2) (2.33)

Thus, in general, the unknown ui, i = 1, 2, 3, ..., n can be expressed as

ui= 1 aii (fi− i−1 X j=1 aijuj− n X j=i+1 aijuj). (2.34)

When the solution at the kth _{iteration is used to compute the approximate solution at the} (k + 1)thiteration, we can re-write Equation (2.34) as Equation (2.35) below:

uk+1_i = 1 aii (fi− i−1 X j=1 aijukj− n X j=i+1 aijukj). (2.35)

The number of computations on the RHS can reduce drastically if the coefficient matrix is sparse i.e. most of aij’s are zero. Jacobi iterations can also be written in a matrix form by splitting the coefficient matrix (A) into diagonal (D), strictly lower (L) and strictly upper triangular (U) matrices. The coefficient matrix is thus expressed as A = D − L − U . Substituting this in the matrix form of linear equations (see Equation (2.28)), we get

The notation of Equation (2.36) allows us to write Jacobi iteration (2.35) in matrix form as

uk+1= D−1(L + U )uk+ D−1f. (2.37)

The matrix D−1(L + U ) above is called the point Jacobi iteration matrix. The word “point” stems from the fact that the iterative step computes the value of the solution at a single point in terms of the solution at other points [22]. A variation of the Jacobi iteration method is the weighted Jacobi (ω-Jacobi) iterative method where the solution at point uk_i also contributes to the approximate solution uk+1_i . For some ω > 0, the weighted Jacobi iteration can be written as: uk+1_i = (1 − ω)uk_i + ω( 1 aii (fi− i−1 X j=1 aijukj − n X j=i+1 aijukj)). (2.38) 2.3.2.2 Gauss-Seidel

The Gauss-Seidel iterative method is a variation of the Jacobi iterative method in the sense that it utilizes the most recently computed approximation of the unknowns to update the solution. The following equation expresses this idea:

uk+1_i = 1 aii (fi− i−1 X j=1 aijuk+1j − n X j=i+1 aijukj). (2.39)

Equation (2.39) suggests that to update any uk+1_i , the most recently computed values of uk+1₀ , uk+1₁ , ..., uk+1_i−1 and the old values of uk

i+1, uki+2, ..., ukn are used. Using Equation (2.39), we can directly write the matrix form in terms of the diagonal (D), strictly lower triangular (L) and strictly upper triangular matrices (U) as

uk+1= D−1(f − (−Luk+1− U uk)). (2.40)

Pre-multiplying Equation (2.40) by D and re-arranging the terms for uk+1_{on the LHS, we get}

(D − L)uk+1= U uk+ f. (2.41)

Pre-multiplying Equation (2.41) by (D − L)−1, we obtain

uk+1= (D − L)−1U uk+ (D − L)−1f, (2.42)

which gives us the point Gauss-Seidel iteration matrix as (D − L)−1_{U . Although the Gauss-} Seidel method generally converges faster than the Jacobi method, it has its drawbacks in the sense that uk+1_i cannot be computed unless uk+1₀ , uk+1₁ , ..., uk+1_i−1 have been computed. Thus, this necessitates an ordering on updating the solution - a limitation which is not present in Jacobi’s method as it uses the approximate solution at the kth_{iteration to compute the value}

at the (k +1)th_{iteration. This property of the Jacobi method makes it very suitable for applica-} tions in parallel computing although a variant of the Gauss-Seidel method called the Red-Black Gauss-Seidel (RBGS) [37] method can be applied when computing the numerical solutions in parallel environments.

To discuss the RBGS method in a parallel environment, consider a mesh in two dimensions. A mesh point having indices (i, j) is given the color Red if i + j is even and the color Black if i + j is odd. This coloring scheme can also be reversed in the sense that a mesh point can be colored Black if i + j is odd, and Red when i + j is even. The algorithm for updating the solution consists of two phases. In the first phase, the red points are updated using only the value of the solution at the neighbouring black points. For example, using a 5-pt stencil in 2-D and the unweighted Jacobi iterative method, the solution at a red point is updated according to the weighted average of the solution at the four neighbouring black points. The updated values of the solution at the red points next to the sub-domain boundary are then communicated to the neighbouring processes. In the second phase, the solution at the black points is updated using the latest value of the solution at the red points. It is important to note that the update of red points (or black points) can be done in any order but the neighbouring processes must synchronize and communicate the updated values of the solution at the red points (or black points) before starting the solution update at the black points (or red points). Although the Red-Black ordering described above works correctly with a 5-pt stencil, it fails with a 9-pt stencil in 2-D. The reason is that the corner points needed for the update of a red point (or black point) are also red (or black). This problem can be overcome by using two additional colors as described in [38]. The technique of multi-color ordering can be extended to more than two dimensions [37, 38].

A further variation of the Gauss-Seidel iteration method is the Successive Over-relaxation (SOR) method where the (k + 1)th_{approximation is the sum of the k}th _{approximation and a} correction in a single Gauss-Seidel iteration [22]. Adding and subtracting uk_i from the RHS of Equation (2.39), we obtain: uk+1_i = uk_i + 1 aii (fi− i−1 X j=1 aijuk+1j − n X j=i aijukj). (2.43)

The term in parenthesis on the RHS of Equation (2.43) can be seen as a change (or correction or displacement) made to uk

i by one Gauss-Seidel iteration. If the successive corrections are one-signed, the convergence can be accelerated by using a larger correction term. This is the idea behind successive over-relaxation and is expressed in the general form as shown in Equation (2.44) below: uk+1_i = uk_i + ω aii (fi− i−1 X j=1 aijuk+1j − n X j=i aijukj). (2.44)

The factor ω in Equation (2.44) is called the acceleration or relaxation parameter and generally 1 < ω < 2. For ω = 1, the SOR method reduces to the Gauss-Seidel method [22]. To obtain the iteration matrix of the SOR method, we first subtract uk _{from both sides of the Equation} (2.40) and multiply the RHS by ω to obtain:

uk+1− uk= ωD−1(f + Luk+1+ U uk− Duk). (2.45) Re-arranging the terms for uk+1and uk in Equation (2.45), we obtain the point SOR iteration matrix H(ω) as:

H(ω) = (I − ωD−1L)−1((1 − ω)I + ωD−1U ). (2.46)

2.3.2.3 Other Iterative Methods

Several other iterative methods exist. A sophisticated and efficient class of non-stationary iter- ative methods called the Krylov subspace methods do not have constant iteration matrices such as Jacobi, Gauss-Seidel and SOR. The idea behind Krylov subspace methods is to generate systematic approximate solutions uk _{∈ u}0_{+ κ}

n(A, r0) of the solution to Au = f , where uk is the kth _{iterate of the approximate solution, u}0 _{is the initial approximation, r}0 _{= f − Au}0 _is the initial residual and κn(A, r0) is the nthKrylov subspace generated by A from r0. Formally, κn(A, r0) = span(r0, Ar0, A2r0, ..., An−1r0) [39].

The problem of solving the linear system of equations of the form Au = f can be visualized as a problem of minimizing ||f − Au|| for u ∈ Rm_{, where ||.|| denotes some norm (generally the} L2 norm) and Rmis the set of all real vectors. The Krylov subspace κn ⊆ Rmand κm= Rm. For n = 1, u1 _{= αr}

0 and we need to choose an α which minimizes ||f − Au1||2. Similarly for n = 2, u2_{= αr}

0+βAr0and we need to choose both α and β such that ||f −Au2||2is minimized. Krylov methods can generate uk from uk−1 efficiently and they are successful because we can find an n << m such that ||f − Aun_||

2< , where  is sufficiently small. Well known methods such as Conjugate Gradient (CG), Arnoldi, Lanczos, Generalized Minimum Residual (GMRES), Biconjugate Gradient Stabilized (BiCGSTAB) etc., all belong to the Krylov family [33, 40, 41]. Out of these the CG method is the most efficient but only applicable to symmetric positive definite matrices. GMRES is the most general method applicable to all matrix types but is less efficient as compared to the CG method. Another method called the MinRES method is applicable to symmetric matrices which need not be positive definite. A detailed description of these methods falls out of the scope of the thesis, but the interested reader can refer to [33].

2.3.2.4 Multilevel Iterative Methods

Iterative methods such as the Geometric Multigrid (GMG) and Algebraic Multigrid (AMG) are specializations of the general class of multilevel iterative methods. Multilevel iterative methods use a hierarchy of approximations of decreasing resolution. Although iterative methods such as

ω−Jacobi and RBGS can effectively remove the high frequency error components on a fine grid, they fail to effectively eliminate the low frequency error components. The idea behind using these coarser approximations in multilevel iterative methods is to accelerate the reduction of the lower frequency components of the error (as high frequency error components of a coarser representation) and thus improve the overall convergence. Multilevel methods can be classified as Additive or Multiplicative. The main difference between these is that in Multiplicative methods, the update of the solution using iterative methods is carried out sequentially i.e. one level after the other, whereas in the Additive Multilevel schemes, these operations can be performed in parallel for various levels. In both the methods, the inter-grid transfer operations are carried out sequentially [25, 42, 43]. Multiplicative methods are generally used as stand- alone solvers and can be applied to asymmetrical problems, while Additive methods are used as preconditioners to accelerate other iterative methods such as the Conjugate Gradient (CG). The standard or classical Multigrid is an example of a Multiplicative Multilevel method [25]. We expand on the Multigrid methods briefly, later in this chapter, and describe the Geometric Multigrid method in detail in Chapter 6. For a detailed discussion of Multiplicative and Additive methods, the interested reader can refer to [42].