Relaxation Methods for Iterative Solution to Linear Systems of Equations

(1)

Relaxation Methods for Iterative Solution

to Linear Systems of Equations

Gerald Recktenwald Portland State University Mechanical Engineering Department

[email protected]

Primary Topics

• Basic Concepts

• Stationary Methods a.k.a. Relaxation methods � Gauss Seidel

� Jacobi � SOR

(2)

Direct and Iterative Methods

Solve

Ax = b

where A is n_{× n matrix, x and b are n element column vectors.} Gaussian elimination with backward substitution is aDirect Method

• A Direct Method obtains the solution in a finite number of steps • Solution is equivalent to

x = A−1b

Iterative Methodsobtain a sequence of approximations to the solution xk k = 0, 1, 2, . . .

such that (b− Axk₎ _{→ 0 as k → ∞.}

Relaxation Solution to Linear Systems page 2

Residual

The residual

r = b− Ax

is zero when x is the solution to Ax = b.

For an iterative method the residual at iteration k is

rk = b_{− Ax}k k = 0, 1, 2, . . .

(3)

Relaxation Methods

(1)

Overview

• Simple to program

• Converges slowly for large systems of equations (large n) • Not a useful stand alone solution method

• Key ingredient to multigrid methods Examples

• Jacobi • Gauss-Seidel • SOR

Relaxation Methods

(2)

Basic Idea

• Update elements of the solution vector one element at a time • “Solve” a nodal equation by assuming other nodal values are known

N

P

S

E W

Apply the finite volume method to get

−aSφS− aWφW + aPφP − aEφE − aNφN = b

If k is the iteration counter, we “solve” for φk+1_P

φk+1_P = 1 aP

�

b + aSφk_S+ aWφk_W + aEφk_E + aNφk_N

(4)

Matrix Notation

(1)

On a structured mesh the interior nodes can be numbered sequentially with a mapping like

i = I _{− 1 + (J − 2)n}x

where nx is the number of cells in the x

direction and xi is the value of the ith

dependent field variable which is located at the cells with indices I and J.

I=1 I=nx+2 J=ny+2 J=1 2 2 3 3

i=1 i=2 i=nx

i=nx+1

x y

i=nxny

The compass point notation

−aSφS− aWφW + aPφP − aEφE− aNφN = b

becomes

−aS,ixi−nx− aW,ixi−1+ aP,ixi− aE,ixi+1− aW,ixi+nx = bi

Matrix Notation

(2)

Thenatural orderingof nodes on a two-dimensional structured mesh yields a coefficient matrix with five diagonals. Coefficients on the diagonals correspond to aP coefficient of

φP ↔ xi and the four neighbor coeﬃcients aS, aW, aE, and aN.

(5)

Matrix Notation

(3)

In an unstructured mesh, the coeﬃcient matrix is also sparse, but the non-zero elements of the coeﬃcient matrix do not lie along diagonals.

Unstructured Mesh Non-zeros in coeﬃcient matrix

0 10 20 30 40 50 60 70 80 90 0 10 20 30 40 50 60 70 80 90 nz = 361

To simplify the presentation, we’ll use a structured mesh.

Relaxation Methods

Use relaxation methods to solve

Ax = b

Relaxation methods do not work for all A, but they do work for standard finite-diﬀerence and finite-volume discretizations of the Poisson equation.

• Jacobi • Gauss-Seidel

(6)

Jacobi Iteration

(1)

A discrete model of the Poisson equation yields a system of equations that can be written Ax = b. The equation for the ith nodal value is

n

�

j=1

aijxj = bi

where n is the total number of nodes in the domain. Extract the diagonal term from the sum

aiixi+ n � j=1 j�=i aijxj = bi

The preceding equation will not be satisfied until, in the limit as the number of iterations increased to∞.

Jacobi Iteration

(2)

“Solve” for the next guess xi.

xi = 1 aii � b− n � j=1 j�=i aijxj � (�)

In terms of the compass-point notation for the two-dimensional finite-volume method, the Equation (�) is φ(k+1)_P = 1 aP � b + aSφ(k)_S + aWφ(k)_W + aEφ(k)_E + aNφ(k)_N �

Equation (�) is not really the solution for xi until all the other xj are known. In other

words, the value of xi from Equation (�) will be correct only when the system of

equations Ax = b is solved.

If matrix A has favorable properties, the value of xi from Equation (�) will be closer to

(7)

Jacobi Iteration

(3)

Let k be the iteration counter. The Jacobi iteration formula is x(k+1)_i = 1 aii � bi− n � j=1 j�=i aijx(k)_j � i = 1, . . . , n

This formula implies that two copies of the x vector are maintained in memory: one for iteration k and one for iteration k + 1.

for k=1:maxit

% -- For nodes away from the boundaries for i=(nx+1):n-nx

xnew(i) = (b(i) + as(i)*x(i-nx) + aw(i)*x(i-1) ... + an(i)*x(i+nx) + ae(i)*x(i+1) ) / ap(i); end

x = xnew; end

Jacobi Iteration

(4)

The Jacobi iterative formula can be written as a matrix equation. Let D = diag(a11, . . . , ann) and B = D− A then

x(k+1)_i = 1 aii � b− n � j=1 j�=i aijx(k)_j � is equivalent to x(k+1)= D−1 � b + Bx(k) � (1) where x(k+1) and x(k)are the solution vectors at iteration k + 1 and k, respectively.

(8)

Jacobi Iteration

(5)

The matrix form of the iteration can be rewritten

x(k+1)= Hx(k)+ d (2)

where

H = D−1B

d= D−1b

The matrix formulation is primarily useful in the analysis of iterative methods and in toy implementations in Matlab. The eigenvalues of H determine the convergence rate of the iterations.

Stopping Criteria

(1)

Recall that the goal is to solve

Ax = b

where A is n× n matrix, x and b are n element column vectors.

Iterative Methodsobtain a sequence of approximations to the solution xk k = 0, 1, 2, . . .

such that b_{− Ax}k_{→ 0 as k → ∞.} The residual

r = b_{− Ax}

(9)

Stopping Criteria

(2)

For an iterative method the residual at iteration k is

rk = b_{− Ax}k k = 0, 1, 2, . . .

If an iterative methods converges, then rk_{→ 0 as k → ∞.} Since r is a vector, what does it mean that rk → 0?

Use a convenient vector norm to test whether_{�r� is small enough.}

Jacobi Iteration

(3)

Solution to a toy problem ∂2T ∂x2 + ∂2T ∂y2 = 0 on 0 _{≤ x ≤ L, 0 ≤ y ≤ W} T = 0 on three boundaries uniform q on fourth boundary

Solution 0 0.5 1 1.5 0 0.5 1 1.5 2 3 3.5 4 4.5

(10)

Jacobi Iteration

(4)

Convergence of Jacobi iterations slow as mesh is refined.

0 200 400 600 800 1000 1200 1400 10−4 10−3 10−2 10−1 100 Iteration || r || 1 / || r 0 ||1 8 x 8 16 x 16 32 x 32

Gauss-Seidel Iteration

(1)

In the Jacobi method x(k+1)is obtained from the frozen values of x(k).

The Gauss-Seidel method uses a similar formula for updating x(k+1)_i , but the new value of x(k+1)_i is used as soon as it is available

x y values from iteration k values from iteration k+1

(11)

Gauss-Seidel Iteration

(2)

Replace nodal values as sweep progresses through the domain

for k=1:maxit

% -- For nodes away from the boundaries for i=(nx+1):n-nx

x(i) = (b(i) + as(i)*x(i-nx) + aw(i)*x(i-1) ... + an(i)*x(i+nx) + ae(i)*x(i+1) ) / ap(i); end

end

Gauss-Seidel Iteration

(3)

Consider the sweep through the nodal values in order of increasing i: x(k+1)₁ = 1 a11 � b1− n � j=2 a1jx(k)j � x(k+1)₂ = 1 a22 � b2− a21x(k+1)1 − n � j=3 a1jx(k)_j � x(k+1)₃ = 1 a33 � b3− a31x(k+1)₁ − a32x(k+1)₂ − n � j=4 a1jx(k)_j � ... 1 � �i−1 �n �

(12)

Gauss-Seidel Iteration

(4)

Coefficients that

multiply new values Coefficients that_{multiply old values}

i A x = b ne w va lue s ol d va lue s = Again, let D = diag(a11, . . . , ann) and define

L = lower triangular part of A U = upper triangular part of A Then Gauss-Seidel iterations correspond to the splitting of A

A = D_{− L − U}

Note that A _{�= LU, i.e.,, L and U are} not the usual factors of A.

Gauss-Seidel Iteration

(5)

Equation (3) can be rewritten as aiix(k+1)_i = bi− i−1 � j=1 aijx(k+1)_i − n � j=i+1 a1jx(k)_j or aiix(k+1)i − i−1 � j=1 aijx(k+1)i = bi− n � j=i+1 a1jx(k)j or Dx(k+1)_{− Lx}(k+1) = b + U x(k) =⇒ (D − L)x(k+1)= U x(k)+ b or x(k+1)= (D_{− L)}−1(U x(k)+ b) (4)

(13)

Gauss-Seidel Iteration

(6)

Note: The order in which the nodes are processed in the Gauss-Seidel iterations will affect the path toward the solution. In other words, for two different orderings of the nodes, the intermediate iterates of x wil be different.

The Jacobi iteration does not depend on the order in which the nodes are numbered. Much of the classical theory of relaxation methods assumes that the nodes and numbered in natural order.

SOR: Successive Over-Relaxation

(1)

Let ˆx(k+1)_i be the updated value of xi from the Gauss-Seidel formula, viz.

ˆ x(k+1)_i = 1 aii � bi− � j<i aijx(k+1)_j − � j>i aijx(k)_j � (5)

Instead of taking ˆx(k+1)_i as the value of xi at the k + 1 step, use

x(k+1)_i = ω ˆx(k+1)_i + (1_{− ω)x}(k)_i (6) where ω is a scalar weighting factor. The basic step in Equation (5) is called relaxation.

ω = 1 for Gauss-Seidel

ω < 1 causes the iterations to more slowly move toward the solution. This is called under-relaxation.

(14)

Relaxation as an update

We can rewrite Equation (6) as

x_i(k+1)= x(k)_i + ω�xˆ(k+1)_i _{− x}(k)_i � (7) or

x(k+1)_i = x(k)_i + ω∆x(k+1)_i (8)

where

∆x(k+1)_i = ˆx(k+1)_i − x(k)i

∆x(k+1)_i is the update to xi in iteration k.

∆x(k+1)_i is the direction in which the solution is changing.

ω∆x(k+1)_i is a magnified (ω > 1) or reduced (ω < 1) change to the solution.

SOR: Successive Over-Relaxation

(2)

Substitute Equation (6) into Equation (5) x(k+1)_i = (1− ω)xki + ω aii � bi− � j<i aijx(k+1)j − � j>i aijx(k)j � (9) Multiply through by aiiand rearrange to get

aiix(k+1)_i + ω � j<i aijx(k+1)_j = (1− ω)aiixki − ω � j>i aijx(k)_j + ωbi

Use the splitting A = D_{− L − U and the preceding equation becomes} Dx(k+1)− ωLx(k+1)= (1− ω)Dx(k)+ ωU x(k)+ ωb

(D− ωL)x(k+1)=�(1− ω)Dx(k)+ ωU�x(k)+ ωb

(15)

SOR: Successive Over-Relaxation

(3)

The update equation is

x(k+1)= (D_{− ωL)}−1�(1_{− ω)Dx}(k)+ ωU�x(k)+ ω(D_{− ωL)}−1b Thus x(k+1)= Hx(k)+ d where H = (D− ωL)−1�(1− ω)Dx(k)+ ωU� d= ω(D_{− ωL)}−1b

The speed of convergence depends on the eigenvalues of H.

Compare Relaxation Convergence

On a coarse 8× 8 mesh 101 102 103 104 ||r|| 1 Jacobi GaussïSeidel SOR

(16)

Compare Relaxation Convergence

On a coarse 16× 16 mesh 0 50 100 150 200 250 300 350 400 101 102 103 104 105 Iteration ||r|| 1 Jacobi GaussïSeidel SOR