Motivation. Solving Linear Systems:Iterative Methods. Iterative Methods. Iterative Methods: Overview. u xx + u yy = 0. x (k) = Cx (k 1), +d

(1)

Solving Linear Systems:Iterative Methods

Motivation

Jacobi Iteration

Gauss Seidel Iteration

Successive Over Relaxation

Determinants

Matrix Inversion

Analysis

ITCS 4133/5133: Intro. to Numerical Methods 1 Iterative Methods

Motivation

◦ Consider the 2D potential field equation defined over a mesh of points by a partial differential equation(PDE):

uxx+ uyy= 0

given the boundary values and a mesh with∆x = ∆y = 0.25.

◦ These systems are large and verysparse, and thus more appropri-ate foriterativesolutions.

Iterative Methods

Solution of linear systems using atrial and errorapproach.

Assume a set ofunknowns, and successivelyrefinethe estimates.

Can produce exact solution; precision dependent on number of iter-ations.

Major Advantage: Can be used to solve non-linear equations, not possible using direct (elimination) approaches.

Iterative Methods: Overview

To solve Ax = B, convert to x = Cx + d and generate a set of approximations,x(1)_{, x}(2)_{, . . . , x}(n)_{, where}

x(k)_{= Cx}(k−1)_{, +d}

(2)

Jacobi Iteration

Must start iteration withinitialestimates for theXis

After computation ofXis,i = 1, 2, · · · , n, use the new estimates for the next iteration.

Convergence: Successive estimates are acceptably small.

Jacobi Iteration

Consider a linear system:

a11X1+ a12X2+ · · · + a1nXn = C1 a21X1+ a22X2+ · · · + a2nXn = C2 ... ... ... ... an1X1+ an2X2+ · · · + annXn = Cn Solve for each unknown,Xi

X1 = C1− a12X2− a13X3_a11 − · · · − a1nXn X2 = C2− a21X1− a23X3_a22 − · · · − a2nXn

... ...

Xn = Cn− an1X1− an2X3_ann− · · · − an−1,nXn=1

Jacobi Iteration: Example

◦ Note:New values of variables are not used until a new iteraction is begun.

(3)

Jacobi Iteration: Application

Finite Difference Solution to a PDE

Jacobi Iteration: Application

Finite Difference Solution to a PDE

Gauss-Seidel Iteration

Modification of the Jacobi iteration

Use themost recent estimatesofXis, instead of waiting for compu-tation of allXis.

Results in faster convergence

Gauss-Seidel Iteration: Example

2x + y = 6, x = −1

2y + 3 x + 2y = 6, y = −1

2x + 3 Iterations proceed as follows (assumex(1)_{= y}(1) _{= 1/2}

x(2)_{= −}1

2y(1)+ 3 = 11/4, y(2)= − 1

(4)

Gauss-Seidel Method: Algorithm

Gauss-Seidel Method: Example

Successive Over Relaxation(SOR)

Goal: To further accelerate Gauss-Seidel iterations using an addi-tional parameter,ω.

Use a weighted combination of the previous and current updates of x.

0 < ω < 1(successive underrelaxation),1 < ω < 2(SOR),ω = 1is Gauss-Seidel.

Successive Over Relaxation(SOR)

Consider



a11_a21 a12_a22 a13_a23 = b1_{= b2} a31 a32 a33 = b3   SOR Equations are

x(new)1 = (1 − ω)x(old)1 +_a11ω (b1− a12x(old)2 − a13x(old)3 ) x(new)

2 = (1 − ω)x(old)2 +_a22ω (b2− a21x(new)1 − a23x(old)3 ) x(new)

(5)

Successive Over Relaxation(SOR):Algorithm

Successive Over Relaxation(SOR):Example

Iterative Methods: Analysis

Consider a matrix A decomposed intoE andF,

A = E + F (E + F)x = b Ex = −Fx + b x = −E−1_{Fx + E}−1_b Or, iteratively, x(k+1) _{= −E}−1_Fx(k)_{+ E}−1_b

Analysis: Jacobi Iteration

In Jacobi’s iteration, the diagonal terms are isolated

(6)

Analysis: Error

Error term across successive iterations as

e(k+1) _{= −E}−1_{F e}(k) For convergence,

lim k→∞e

(k)_{= 0}

Ife(k)_{− e}(k+1)_{is positive, the solution is converging}

Use thenormofeto determine its magnitude (2-norm,∞norm)

Jacobi Iteration: Convergence

Sufficient Condition: MatrixAbe strictly diagonally dominant.

Necessary and Sufficient Condition: Magnitude of largest eigen value of iteration matrixCbe larger than 1.

Order of equations in Jacobi iteration matters!

Jacobi Iteration: Computational Complexity

Each iteration involves one matrix-vector multiplication,(n−1)2 mul-tiplies

Total computation can be significantly less thanO(n3₎_{, required of} GE.

Particularly appropriate forparallelcomputation.

Gauss Seidel Iteration: Analysis

Acan be decomposed as follows:

A = L + D + U For Gauss-Seidel iteration,

Ax = b

(D + L)x = −Ux + b

(7)

Gauss Seidel Iteration: Convergence

Form of the matrix and eigen values determine convergence,but not convenient for analysis.

Frequently, matrix is real, symmetric with positive diagonal elements (PDE systems solutions) – convergence guaranteed if eigen values are real and positive.

if A is positive definite, convergence is guaranteed for any initial vector,x(0)_.

When iteration matrix is non-negative, Jacobi and Gauss-Seidel ei-ther converge or diverge; Faster convergence using Gauss-Seidel.

Computational Effort: Roughly half the number of iterations of Ja-cobi.

Successive Overrelaxation: Analysis

Similar to Gauss-Seidel, multiply by relaxation parameter,ω

ω(D + L)x = −ωUx + ωb Multiply both sides by(1 − ω)Dx,

(D − ωL)x = ((1 − ω)D − ωU)x + ωb

x = (D − ωL)−1((1 − ω)D − ωU)x + ω(D − ωL)−1b

⇒ SOR iteration matrixC = (D − ωL)−1((1 − ω)D − ωU).

Successive Overrelaxation: Analysis

SOR iteration matrixC = (D − ωL)−1((1 − ω)D − ωU).

Can be shown that|C| = (1 − ω)n

Determinant ofCis product of its eigen values.

Outside of the interval0 ≤ ω ≤ 2, at least one eigen value is greater than 1.

Chooseωto be between 0 and 2.

SOR method is designed to reduce the residual, r = b − Ax(_k) more rapidly than Gauss-Seidel method.

Successive Overrelaxation: Positive Definite

Matrices

SOR method is guaranteed to converge for anypositive definite ma-trix.

Positive Definite Matrix: Following apply:

⇒ xT_{Ax > 0}

⇒ All eigen values are positive

(8)

Method of Determinants

Consider the linear system

   

a11 a12 · · · a1n a21 a22 · · · a2n ... ... ... ... an1 an2 · · · ann

        X1 X2 ... Xn     =     C1 C2 ... Cn     We can use Cramer’s rule to determine theXis, as

Xi= |Ai|_|A|

where|A|is thedeterminantofAand |Ai|is the determinant of matrix A, whoseith column is replaced by theCvector.

Method of Determinants

For example, X3=

a11 a12 C1 · · · a1n a21 a22 C2 · · · a2n ... ... ... ... ... an1 an2 Cn · · · ann

a11 a12 a13 · · · a1n a21 a22 a23 · · · a2n ... ... ... ... ... an1 an2 an3 · · · ann