Introduction
Direct methods for solving system of linear equations
Gauss Elimination method
Gauss Elimination with Partial Pivoting LU-Decomposition (Factorization) Iterative Method for Solving System of Lin- ear equation
Jacobi Method
Gauss-Seidel Iteration Method EIGEN VALUE PROBLEMS
Power Method Inverse Power Method
3 — System of Equation
3.1 Introduction
Consider the systems of algebraic equations a 11 x 1 + a 12 x 2 + . . . + a 1n x n = b 1
a 21 x 1 + a 22 x 2 + . . . + a 2n x n = b 2
... ... (3.1)
a n1 x 1 + a n2 x 2 + . . . + a nn x n = b n
In matrix notation we write the system as
Ax = b (3.2)
where
A =
a 11 a 12 . . . a 1n
a 21 a 22 . . . a 2n
... ... ... ...
a n1 a n2 . . . a nn
, X =
x 1
x 2
...
x n
, b =
b 1
b 2
...
b n
A particularly useful representation of the equations for computational purposes is the augmented coefficient matrix, obtained by adjoining the constant vector b to the coefficient matrix A in the following fashion:
[ A|b] =
a 11 a 12 . . . a 1n b 1
a 21 a 22 . . . a 2n b 2
... ... ... ... ...
a n1 a n2 . . . a nn b n
A system of n linear equations in n unknowns has a unique solution, provided that the determinant of the coefficient matrix is nonsingular, i.e., if |A| �= 0. The rows and columns of a nonsingular matrix are linearly independent in the sense that no row (or column) is a linear combination of other rows (or columns). If the coefficient matrix is singular, the equations may have an infinite number of solutions, or no solutions at all, depending on the constant vector.
Numerical methods for solving systems of linear equations divide into two categories: direct
methods and iterative methods.
• Direct methods would give exact solution of problem after finite number of elementary algebraic operations without rounding error.
• In Iterative methods, the algorithm begins with initial value (a guess for solution) , say x 0 , and successively construct a sequence of values,x k , called iterates points, until a prescribed condition is met so that the last iterate value approximates the desired solution with a reasonable degree of accuracy.
3.2 Direct methods for solving system of linear equations
Elementary row transformations (operations) The following operations on the rows of a matrix A are called the elementary row transformations (operations).
1. Interchange of any two rows. If we interchange the i th row with the j th row, then we usually denote the operation as R i ↔ R j .
2. Division/multiplication of any row by a non-zero number p. If the i th row is multiplied by p, then we usually denote this operation as pR i .
3. Adding/subtracting a scalar multiple of any row to any other row. If all the elements of the j th row are multiplied by a scalar p and added to the corresponding elements of the i th row, then, we usually denote this operation as R i → R i + pR j . Note the order in which the operation R i + pR j is written. The elements of the j th row remain unchanged and the elements of the i th row get changed.
3.2.1 Gauss Elimination method
The method is based on the idea of reducing the given system of equations Ax = b, to an upper triangular system of equations Ux = z, using elementary row operations.
�
Example 3.1 Use Gaussian elimination to solve the following system.
0.143x 1 + 0.357x 2 + 2.01x 3 = −5.173
−1.31x 1 + 0.911x 2 + 1.99x 3 = −5.458 11.2x 1 − 4.30x 2 − 0.605x 3 = 4.415
After each intermediate calculation, round the result to three significant digits.
�Solution: Applying Gaussian elimination to the augmented matrix for this system produces the following.
0.143 0.357 2.01 −5.17
−1.31 0.911 1.99 −5.46 11.2 −4.30 −0.605 4.42
R
1→(1/0.143)R
1−−−−−−−−−→
1 2.50 14.1 −36.2
−1.31 0.911 1.99 −5.46 11.2 −4.30 −0.605 4.42
R
2→R
2+1.31R
1−−−−−−−−−→
R
3→R
3−11.2R
1
1 2.5 14.1 −36.2
0 4.91 20.5 −52.9 0 −32.3 −159 409
−−−−−−−−→ R
2→(1/4.91)R
2
1 2.5 14.1 −36.2
0 1 4.89 −12.6
0 −32.3 −159 409
R
3→R
3+32.3R
2−−−−−−−−−→
1 2.5 14.1 −36.2
0 1 4.89 −12.6
0 0 −1 2
Thus x 3 = −2.00, and using back-substitution we obtain x 2 = −2.82 and x 1 = −0.950.
3.2 Direct methods for solving system of linear equations 29
�
Example 3.2 Consider the following system 2
3 x 1 + 2 7 x 2 + 1
5 x 3 = 43 1 15
3 x 1 + 1 7 x 2 − 1
2 x 3 = 5 1 6
5 x 1 − 3 7 x 2 + 2
5 x 3 = − 12 5
�
Solution:
Exercise: Using Gaussian elimination and back substitution show that the exact solution is x 1 = 1,x 2 = 7 and x 3 = 1
Now write the system using four decimal digit rounding arithmetic 0.6667x 1 + 0.2857x 2 + 0.2000x 3 = 2.867
0.3333x 1 + 0.1429x 2 − 0.5000x 3 = 0.8333 (3.3)
0.2000x 1 − 0.4286x 2 + 0.4000x 3 = −2.400 Remarks:
• To obtain the 4-digit floating point approximation of a number, we use the rounding method (e.g. 2 3 = 0.66666... gives 0.6667.)
• Whenever performing a calculation (e.g. addition or multiplication) involving two or more numbers, the 4-digit rounded form is used for each number and then the result of the calculation is rounded as well. For example, 0.1234 × 1.567 = 0.193368··· = 0.1934.
This is the source of roundoff error accummulation.
• When performing row operations to get the echelon form, the entries below the pivot are implicitly set equal to zero, as opposed to being explicitly calculated, in order to avoid unnecessary calculations
Thus,
0.6667 0.2857 0.2 2.867 0.3333 0.1429 −0.5 0.8333
0.2 −0.4286 0.4 −2.4
−−−−−−−−−→ R
2→R
2−
0.33330.6667R
1R
3→R
3−
0.20000.6667R
1
0.6667 0.2857 0.2 2.867
0 0.0001 −0.6 −0.5997
0 −0.5143 0.34 −3.26
R
3→R
3−
−0.51430.0001R
1−−−−−−−−−−→
0.6667 0.2857 0.2 2.867
0 0.0001 −0.6 −0.5997
0 0 −3086 −3087
which gives the solution:
x 3 = 1.000 (exact)
x 2 = 3.000 (relative error=57%) and x 1 = 2.715 (relative error=171.5%) Remark:
The large errors obtained in the previous example were obtained because of a cancellation (or
loss of significance) error which occurred while working on the first column. This introduced a
small pivot, 0.0001, in the second column which amplified the error. We cannot always avoid
cancellation errors but at least we can try to avoid the use of very small pivots.
3.2.2 Gauss Elimination with Partial Pivoting
1. Find the entry in the left column with the largest absolute value. This entry is called the pivot.
2. Perform a row interchange, if necessary, so that the pivot is in the first row.
3. Use elementary row operations to reduce the remaining entries in the first column to zero. The completion of these four steps is called a pass. After performing the first pass, ignore the first row and first column and repeat the four steps on the remaining submatrix.
Continue this process until the matrix is in row-echelon form.
�
Example 3.3 Apply the technique of Gaussian elimination with partial pivoting to system
(3.3) and verify if the error has improved.
�
0.6667 0.2857 0.2 2.867 0.3333 0.1429 −0.5 0.8333
0.2 −0.4286 0.4 −2.4
−−−−−−−−−→ R
2→R
2−
0.33330.6667R
1R
3→R
3−
0.20000.6667R
1
0.6667 0.2857 0.2 2.867
0 0.0001 −0.6 −0.5997
0 −0.5143 0.34 −3.26
R
2↔R
3−−−−→
0.6667 0.2857 0.2 2.867
0 −0.5143 0.34 −3.26
0 0.0001 −0.6 −0.5997
R
3→R
3+
0.00010.5143R
2−−−−−−−−−→
0.6667 0.2857 0.2 2.867
0 −0.5143 0.34 −3.26
0 0 −0.5999 −0.603
Partial pivoting gives the solution x 3 = 1.001, x 2 = 7.000 and x 1 = 1.000.
�
Example 3.4 Solve the system of equations 2x 1 + x 2 + x 3 − 2x 4 = −10
4x 1 + 2x 3 + x 4 = 8 3x 1 + 2x 2 + 2x 3 = 7 x 1 + 3x 2 + 2x 3 − x 4 = −5
using the Gauss elimination with partial pivoting
�Solution: The augmented matrix is given by
[ A|b] =
2 1 1 −2
4 0 2 1
3 2 2 0
1 3 2 −1
� �
� �
� �
� �
−10 8 7
−5
−−−−→ R
1↔R
2
4 0 2 1
2 1 1 −2
3 2 2 0
1 3 2 −1
� �
� �
� �
� � 8
−10 7
−5
R
2→R
2−1/2R
1R
3→R
3−3/4R
1−−−−−−−−→
R
4→R
4−1/4R
1
4 0 2 1
0 1 0 −5/2
0 2 1/2 −3/4 0 3 3/2 −5/4
� �
� �
� �
� � 8
−14 1
−7
R
2↔R
4−−−−→
4 0 2 1
0 3 3/2 −5/4 0 2 1/2 −3/4
0 1 0 −5/2
� �
� �
� �
� � 8
−14 1
−7
R
3→R
3−(2/3)R
2−−−−−−−−−→
R
4→R
4−1/3R
2
4 0 2 1 8
0 3 3/2 −5/4 −7
0 0 −1/2 1/12 17/3 0 0 −1/2 −25/12 −35/3
R
4→R
4−R
3−−−−−−→
4 0 2 1 8
0 3 3/2 −5/4 −7
0 0 −1/2 1/12 17/3
0 0 0 −13/6 −52/3
Using back substitution, we obtain x 1 = 5, x 2 = 6 x 3 = −10 x 4 = 8
3.2 Direct methods for solving system of linear equations 31 Exercise 3.1 Consider the following system
3x 1 + x 2 + 4x 3 − x 4 = 7 2x 1 − 2x 2 − x 3 + 2x 4 = 1 5x 1 + 7x 2 + 14x 3 − 8x 4 = 20 x 1 + 3x 2 + 2x 3 + 4x 4 = −4
1. Show, by direct substitution, that the exact solution is (1,-1,1,-1).
2. Solve the system using Gaussian elimination without pivoting and calculate the relative errors.
3. Solve the system using Gaussian elimination with partial pivoting and calculate the relative errors.
�
Ill-Conditioned System of Linear Equations
Definition 3.2.1 The system of equation AX = b is said to be ill-conditioned or unstable system if it is highly sensetive to small change in A or b. i.e. a small change in A or b causes a big difference in the solution of the system
On the other hand if small changes in A and/or b give small changes in the solution, the system is said to be stable, or, well conditioned. Thus in an ill-conditioned system, even the small round off errors affect the solutions very badly. Unfortunately it is quite difficult to recognize an ill-conditioned system.
�
Example 3.5 Consider the system 2x + y = 3
2x + 1.001y = 0
that have the solution x = 1501.5,y = −3000.
�Now, we make slight changes in the given system of equations. The new system becomes 2x + y = 3
2x + 1.002y = 0 Here we get x = 751.5, y = −1500.
Note that a 0.1% change in the coefficient of y produced a 100% change in the solution. Hence the given system is ill-conditioned.
Exercise 3.2 Show that the system of equations
� 2.1 1.8 6.2 5.3
�� x 1
x 2
�
=
� 2.1 6.2
�
is ill-conditioned.
�3.2.3 LU-Decomposition (Factorization)
Consider the system of equations, written in matrix form Ax = b where A is n × n square matrix, and x,b is n × 1 matrix.
The LU decomposition (or factorisation) of the matrix A consists of writing the matrix A as the product of a lower triangular matrix L and an upper triangular matrix U, such that
A = LU where
l 11 0 0 ... 0 l 21 l 22 0 ... 0 l 31 l 32 l 33 . . . 0
...
l n1 l n2 l n1 . . . l nn
, and
u 11 u 12 u 13 . . . u 1n
0 u 22 u 23 . . . u 2n
0 0 u 33 . . . u 3n
...
0 0 0 . . . u nn
Thus, Ax = b ⇒ LUx = b. Let y = Ux so that Ly = b.
R The LU decomposition is guaranteed when the matrix A is positive definite.
Definition 3.2.2 A matrix A is positive definite, if x T Ax > 0 for any vector x �= 0.
Further, x T Ax = 0 i f x = 0.
Positive definite matrices have the following important properties :
• The eigenvalues of a positive definite matrix are all real and positive.
• All the leading minors of A are positive.
Doolittle’s Method
When l ii = 1, the method is called the Doolittle’s method.
Having determined the matrices L and U, the system of equation Ax = b becomes LUx = b
�
Example 3.6 Solve the following system of equation x 1 + 4x 2 + 5x 3 = 1
4x 1 + 20x 2 +32x 3 = 8 x1 + 32x 2 +64x 3 = 14
using the LU decomposition method (Doolittle’s Method).
�Solution: Using Gauss elimination
1 4 5 4 20 32 5 32 64
R
2→R
2− 4R
1R
3→R
3−5R
1−−−−−−−→
1 4 5 0 4 12 0 12 39
−−−−−−−→ R
3→R
3−3R
2
1 4 5 0 4 12 0 0 3
Thus the LU factorization of the matrix A is
A = LU =
1 0 0 4 1 0 5 3 1
1 4 5 0 4 12 0 0 3
3.2 Direct methods for solving system of linear equations 33 Now, let y = Ux, then Ax = b ⇒ LUx = b ⇒ Ly = b
From Ly=b, we get
1 0 0 4 1 0 5 3 1
y 1
y 2
y 3
=
1 8 14
Using the forward substitution method, we obtain
y 1 = 1, 4y 1 + y 2 = 8 ⇒ y 2 = 4 and 5y 1 + 3y 2 + y 3 = 14 ⇒ y 3 = −3 From Ux = y we get
1 4 5 0 4 12 0 0 3
x 1
x 2
x 3
=
1 4
−3
Using the backward substitution method we obtain x 3 = −1, 4x 2 + 12x 3 = 4 ⇒ x 2 = 4
⇒ x 1 + 4x 2 + 5x 3 = 1 ⇒ x 1 = −10 Therefore, the solution is [−1,4,−10]
�
Example 3.7 Solve the following system of equation x + y − z = 2
3x − 3y − 2z = 3
−3x + 4y + 2z = 1
using the LU decomposition method (Doolittle’s Method).
�Solution: Using Gauss elimination
1 1 −1
3 −3 −2
−3 4 2
R
2→R
2− 3R
1R
3→R
3−−3 R
1−−−−−−−−→
1 1 −1 0 −6 1
0 7 −1
−−−−−−−−→ R
3→R
3−−
76R
2
1 1 −1 0 −6 1 0 0 1 6
Thus the LU factorization of the matrix A is
A = LU =
1 0 0
3 1 0
−3 − 7 6 1
1 1 −1 0 −6 1 0 0 1 6
Now, let y = Ux, then Ax = b ⇒ LUx = b ⇒ Ly = b From Ly=b, we get
1 0 0
3 1 0
−3 − 7 6 1
y 1
y 2
y 3
=
3 2 1
Using the forward substitution method, we obtain
y 1 = 3, 3y 1 + y 2 = 2 ⇒ y 2 = −7 and − 3y 1 − 7 6 y 2 + y 3 = 1 ⇒ y 3 = 11 6
From Ux = y we get
1 1 −1 0 −6 1 0 0 1 6
x y z
=
3
−7 11 6
Using the backward substitution method we obtain 1
6 z = 11
6 = ⇒ z = 11, −6y + z = −7 ⇒ y = 3
⇒ x + y − z = 3 ⇒ x = 11 Therefore, the solution is [11,3,11]
Exercise 3.3 Solve the following system of equations x + 2y + 3z = 1
x − y + 4z = 4 2x − 2y + z = 8
using LU-decomposition.
�Crout’s Method
When u ii = 1, the method is called the Crout’s method.
�
Example 3.8 Solve the following system of equations 3x + 2y − 3z = 6
2x + 2y + 5z = −3 x + y − z = 8
using LU-decomposition.
�Solution: Let
A =
3 2 −3
2 2 5
1 1 1
, L =
l 11 0 0 l 21 l 22 0 l 31 l 32 l 33
and U =
1 u 12 u 13
0 1 u 23
0 0 1
A = LU, becomes
3 2 −3
2 2 5
1 1 1
=
l 11 0 0 l 21 l 22 0 l 31 l 32 l 33
1 u 12 u 13
0 1 u 23
0 0 1
⇒
3 2 3 2 2 5 1 1 1
=
l 11 l 11 u 12 l 11 u 13
l 21 l 21 u 12 + l 22 l 21 u 13 + l 22 u 23
l 31 l 31 u 12 + l 32 l 31 u 13 + l 32 u 23 + l 33
⇒
l 11 = 3, l 11 u 12 = 2, l 11 u 13 = 3 l 21 = 2, l 21 u 12 + l 22 = 2, 21 u 13 + l 22 u 23 = 5
l 31 = 1, l 31 u 12 + l 32 = 1, l 31 u 13 + l 32 u 23 +l 33 = 1 T hus,
3.3 Iterative Method for Solving System of Linear equation 35
L =
3 0 0
2 2
3 0
1 1 3 − 3 2
and U =
1 2
3 − 1 0 1 21 2
0 0 1
Let y = Ux then Ly = b
3 0 0
2 2
3 0
1 1 3 − 3 2
y 1
y 2
y 3
=
6
−3 2
Using the forward substitution method, we obtain 3y 1 = 6 ⇒ y 1 = 2,
2y 1 + 2
3 y 2 = −3 ⇒ y 2 = −21 2 y 1 + 1
3 y 2 − 3
2 y 3 = 1 ⇒ y 3 = −7 3 From Ux = y, we get
1 2
3 − 1 0 1 21 2
0 0 1
x 1
x 2
x 3
=
2
−21 2
−2 3
Using the backward substitution method we obtain z = − 7
3 , y = 14, x = − 29 3
Exercise 3.4 Solve the following system of equations using LU-decomposition.
1.
2 1 −4 1
−4 3 5 −2
1 −1 1 −1
1 3 −3 2
x 1
x 2
x 3
x 4
=
4
−10 2
−1
2.
x 1 + 6x 2 + 2x 3 = 9 2x 1 + 12x 2 + 5x 3 = −4 x 1 − 3x 2 − x 3 = 17
�
3.3 Iterative Method for Solving System of Linear equation
The iterative methods are not always successful to all systems of equations. If the methods are to succeed, each equation of the system must possess one large coefficient and the large coefficients are along the leading diagonal of the coefficient matrix. That is iterative methods will be successful only when the system is diagonally dominant system. That is, an n × n matrix A is diagonally dominant if
|a ii | =
∑ n j=1, j�=i |a i j |
Under the category of iterative method, we shall describe the following two methods:
1. Gauss-Jacobi’s method
2. Gauss-Seidel method.
3.3.1 Jacobi Method
In order to apply this these iterative techniques, the equations must satisfy diagonal dominance criteria.
Consider the systems of algebraic equations
a 11 x 1 + a 12 x 2 + . . . + a 1n x n = b 1
a 21 x 1 + a 22 x 2 + . . . + a 2n x n = b 2
... ...
a n1 x 1 + a n2 x 2 + . . . + a nn x n = b n
Dividing each equation by the leading diagonal term, we get x 1 = b 1
a 11 −
∑ n j=2, j�=1
a 1 j x j
a 11
x 2 = b 2
a 22 −
∑ n j=1, j�=2
a 2 j x j
a 22
...
x i = b i
a ii −
∑ n j=1, j�=i
a i j x j
a ii
Let x ( k−1) =
x ( 1 k−1) x ( 2 k−1)
...
x ( n k−1)
be the (k −1) th approximation of the given system of equation. Then the k th approximation is
x (k) 1 = b 1
a 11 −
∑ n j=2, j�=1
a 1 j x ( j k−1) a 11
x (k) 2 = b 2
a 22 −
∑ n j=1, j�=2
a 2 j x ( j k−1) a 22
...
x (k) i = b i
a ii −
∑ n j=1, j�=i
a i j x ( j k−1) a ii
The process is continued till convergence is secured. That is, if the error �
�x (k) − x ( k−1) �
� < ε R
• In the absence of any better estimates, the initial approximations are taken as : x (0) 1 = 0,x (0) 2 = 0,...,x (0) n = 0
• If some of the diagonal elements are zeros, or the system is not diagonally dominant,
we may have to rearrange the equations.
3.3 Iterative Method for Solving System of Linear equation 37
�
Example 3.9 Solve the system of equations using the Jacobi method correct to two decimal place.
5x 1 + x 2 + 2x 3 = 10
−3x 1 + 9x 2 + 4x 3 = −14 x 1 + 2x 2 + 7x 3 = −33.
Take the initial approximation, x (0) =
0 0 0
.
�Solution: We have
A =
5 1 2
−3 9 4
1 2 7
|a 11 = 5 > |a 12 | + |a 13 | = 1 + 2 = 3
|a 22 = 9 > |a 21 | + |a 23 | = 3 + 4 = 7
|a 33 = 7 > |a 31 | + |a 32 | = 1 + 2 = 3
Thus, the system is diagonally dominant. Rearranging the equations we get x (k) 1 = 2 − 1
5
� x ( 2 k−1) + 2x ( 3 k−1) �
x (k) 2 = − 14 9 −
1 9
� −x ( 1 k−1) + 4x ( 3 k−1) �
x (k) 3 = − 33 7 −
1 7
� x ( 1 k−1) + 2x ( 2 k−1) �
First approximation: Substituting the initial guesses of the unknowns in the above governing equation, we get the first approximations as:
x (1) 1 = 2 − 1 5
� x (0) 2 + 2x (0) 3 �
= 2 x (1) 2 = − 14
9 − 1 9
� −x (0) 1 + 4x (0) 3 �
= −1.5556 x (1) 3 = − 33
7 − 1 7
� x (0) 1 + 2x 2 (0) �
= −4.7143
Second approximation: Substituting the first approximation of the unknowns in the above governing equation, we get the second approximations as:
x (2) 1 = 2 − 1 5
� x (1) 2 + 2x (1) 3 �
= 4.1968 x (2) 2 = − 14
9 − 1 9
� −3x (1) 1 + 4x (1) 3 �
= 1.2063 x (2) 3 = − 33
7 − 1 7
� x (1) 1 + 2x 2 (1) �
= −4.5556
Third approximation: Substituting the second approximation in the above governing equation, we get the third approximations as:
x (3) 1 = 2 − 1 5
� x (2) 2 + 2x (2) 3 �
= 3.581 x (3) 2 = − 14
9 − 1 9
�
−3x (2) 1 + 4x (2) 3 �
= 1.8681 x (3) 3 = − 33
7 − 1 7
� x (2) 1 + 2x 2 (2) �
= −5.6585
Fourth approximation:
x (4) 1 = 2 − 1 5
� x (3) 2 + 2x (3) 3 �
= 3.8898 x (4) 2 = − 14
9 − 1 9
� −3x (3) 1 + 4x (3) 3 �
= 2.153 x (4) 3 = − 33
7 − 1 7
� x (3) 1 + 2x 2 (3) �
= −5.7596 Fifth approximation:
x (5) 1 = 2 − 1 5
� x (4) 2 + 2x (4) 3 �
= 3.8732 x (5) 2 = − 14
9 − 1 9
� −3x (4) 1 + 4x (4) 3 �
= 2.3008 x (5) 3 = − 33
7 − 1 7
� x (4) 1 + 2x 2 (4) �
= −5.8851 Sixth approximation:
x (6) 1 = 2 − 1 5
� x (5) 2 + 2x (5) 3 �
= 3.8939 x (6) 2 = − 14
9 − 1 9
� −3x (5) 1 + 4x (5) 3 �
= 2.3511 x (6) 3 = − 33
7 − 1 7
� x (5) 1 + 2x 2 (5) �
= −5.925 Seventh approximation:
x (7) 1 = 2 − 1 5
� x (6) 2 + 2x (6) 3 �
= 3.8998 x (7) 2 = − 14
9 − 1 9
� −3x (6) 1 + 4x (6) 3 �
= 2.3757 x (7) 3 = − 33
7 − 1 7
� x (6) 1 + 2x 2 (6) �
= −5.9423 Eighth approximation:
x (8) 1 = 2 − 1 5
� x (7) 2 + 2x (7) 3 �
= 3.9018 x (8) 2 = − 14
9 − 1 9
� −3x (7) 1 + 4x (7) 3 �
= 2.3854 x (8) 3 = − 33
7 − 1 7
� x (7) 1 + 2x 2 (7) �
= −5.9502 Ninth approximation:
x (9) 1 = 2 − 1 5
� x (8) 2 + 2x (8) 3 �
= 3.903 x (9) 2 = − 14
9 − 1 9
�
−3x (8) 1 + 4x (8) 3 �
= 2.3896 x (9) 3 = − 33
7 − 1 7
� x (8) 1 + 2x 2 (8) �
= −5.9532
Since, � �x (9) − x (8) � � =
|3.903 − 3.9018|
|2.3896 − 2.3854|
| − 5.9532 − (−5.9502)|
=
0.0012 0.0042 0.003
< ε = 0.5 ×10 −2
Thus, the approximate root correct to two decimal place is x (9) .
3.3 Iterative Method for Solving System of Linear equation 39
�
Example 3.10 Solve the system of equations using the Jacobi method correct to two decimal place.
5x 1 − x 2 + x 3 = 10 x 1 + x 2 + 5x 3 = −1
x 1 + 2x 2 = 6.
Take the initial approximation, x (0) =
2.64 1.7
−1.12
.
�Solution: The given system is not diagonally dominant. So rearranging the system we get 5x 1 − x 2 + x 3 = 10
x 1 + 2x 2 = 6 x 1 + x 2 + 5x 3 = −1 Here, we have
A =
5 −1 1
1 2 0
1 1 5
|a 11 = 5 > |a 12 | + |a 13 | = 1 + 1 = 2
|a 22 = 2 > |a 21 | + |a 23 | = 1 + 0 = 1
|a 33 = 5 > |a 31 | + |a 32 | = 1 + 1 = 1
Thus, the system is diagonally dominant. Rearranging the equations we get x (k) 1 = 2 − 1
5
� −x 2 ( k−1) + x ( 3 k−1) �
x (k) 2 = 1 3 −
1 6 x ( 1 k−1) x (k) 3 = − 1
5 − 1 5
� x ( 1 k−1) + x 2 ( k−1) �
First approximation: x (1)
x (1) 1 = 2 − 1 5
� −x (0) 2 + x (0) 3 �
= 2.564 x (1) 2 = 3 − 1
2 x (0) 1 = 1.68 x (1) 3 = − 1
5 − 1 5
� x (0) 1 + x (0) 2 �
= −1.068 Second approximation: x (2)
x (2) 1 = 2 − 1 5
� −x (1) 2 + x (1) 3 �
= 2.549 x (2) 2 = 3 − 1
2 x (1) 1 = 1.718 x (2) 3 = − 1
5 − 1 5
� x (1) 1 + x (1) 2 �
= −1.0488
Third approximation x (3) :
x (3) 1 = 2 − 1 5
� −x (2) 2 + x (2) 3 �
= 2.553 x (3) 2 = 3 − 1
2 x (2) 1 = 1.725 x (3) 3 = − 1
5 − 1 5
� x (2) 1 + x (2) 2 �
= −1.054
Fourth approximation x (4) :
x (4) 1 = 2 − 1 5
� −x (3) 2 + x (3) 3 �
= 2.5558 x (4) 2 = 3 − 1
2 x (3) 1 = 1.7235 x (4) 3 = − 1
5 − 1 5
� x (3) 1 + x (3) 2 �
= −1.0556
Since, �
�x (4) − x (3) �
� =
|2.5558 − 2.553|
|1.725 − 1.7235|
| − 1.0556 − (−1.054)|
=
0.0028 0.0015 0.0016
< ε = 0.5 ×10 −2 Thus, the approximate root correct to two decimal place is x (4) .
3.3.2 Gauss-Seidel Iteration Method
This method is a modification of the Jacobi’s method. Because the new values can be immediately stored in the location that held the old values, the storage requirement for x with the Gauss-Seidel method is half what it would be the Jacobi method and the rate of convergence is more rapid.
Consider the systems of algebraic equations
a 11 x 1 + a 12 x 2 + . . . + a 1n x n = b 1
a 21 x 1 + a 22 x 2 + . . . + a 2n x n = b 2
... ...
a n1 x 1 + a n2 x 2 + . . . + a nn x n = b n
is written as
x 1 = b 1
a 11 − 1
a 11 ( a 12 x 2 + a 13 x 3 + ··· + a 1n x n ) x 2 = b 2
a 22 − 1
a 22 ( a 21 x 1 + a 23 x 3 + ··· + a 2n x n ) ...
x i = b i
a ii − 1 a ii
� a i1 x 1 + a i2 x 2 + ··· + a i(i−1) x i−1 + ··· + a in x n �
3.3 Iterative Method for Solving System of Linear equation 41
Let x ( k−1) =
x ( 1 k−1) x ( 2 k−1)
...
x ( n k−1)
be the (k −1) th approximation of the given system of equation. Then the k th approximation is
x (k) 1 = b 1
a 11 − 1 a 11
� a 12 x ( 2 k−1) + a 13 x ( 3 k−1) + ··· + a 1n x ( n k−1)
�
x (k) 2 = b 2
a 22 − 1 a 22
� a 21 x (k) 1 + a 23 x ( 3 k−1) + ··· + a 2n x ( n k−1)
�
x (k) 3 = b 3
a 33 − 1 a 33
� a 31 x (k) 1 + a 32 x (k) 2 + ··· + a 3n x ( n k−1)
� ...
x (k) i = b i
a ii − 1 a ii
� a i1 x (k) 1 + a i2 x (k) 2 + ··· + a i(i−1) x i−1 (k) + ··· + a in x ( n k−1)
�
The process is continued till convergence is secured.
�
Example 3.11 Solve the system of equations 5x 1 + x 2 + 2x 3 = 10
−3x 1 + 9x 2 + 4x 3 = −14 x 1 + 2x 2 + 7x 3 = −33.
Using the Gauss-Seidel method correct to two decimal place using the initial approximation,
x (0) = [0 0 0] T .
�Solution: The system is diagonally dominant. Rearranging the equations we get x (k) 1 = 2 − 1
5
� x ( 2 k−1) + 2x ( 3 k−1) �
x (k) 2 = − 14 9 −
1 9
�
−x (k) 1 + 4x ( 3 k−1) � x (k) 3 = − 33
7 − 1 7
� x (k) 1 + 2x (k) 2 �
First approximation:
x (1) 1 = 2 − 1 5
� x (0) 2 + 2x (0) 3 �
= 2 x (1) 2 = − 14
9 − 1 9
�
−x (1) 1 + 4x (0) 3 �
= −0.8889 x (1) 3 = − 33
7 − 1 7
� x (1) 1 + 2x 2 (1) �
= −4.746 Second approximation:
x (2) 1 = 2 − 1 5
� x (1) 2 + 2x (1) 3 �
= 4.0762 x (2) 2 = − 14
9 − 1 9
�
−3x (2) 1 + 4x (1) 3 �
= 1.9125 x (2) 3 = − 33
7 − 1 7
� x (2) 1 + 2x 2 (2) �
= −5.843
Third approximation:
x (3) 1 = 2 − 1 5
� x (2) 2 + 2x (2) 3 �
= 3.9547 x (3) 2 = − 14
9 − 1 9
� −3x (3) 1 + 4x (2) 3 �
= 2.3596 x (3) 3 = − 33
7 − 1 7
� x (3) 1 + 2x 2 (3) �
= −5.9534 Fourth approximation:
x (4) 1 = 2 − 1 5
� x (3) 2 + 2x (3) 3 �
= 3.90944 x (4) 2 = − 14
9 − 1 9
� −3x (3) 1 + 4x (2) 3 �
= 2.3935 x (4) 3 = − 33
7 − 1 7
� x (3) 1 + 2x 2 (3) �
= −5.9566 Fifth approximation:
x (5) 1 = 2 − 1 5
� x (4) 2 + 2x (4) 3 �
= 3.903 x (5) 2 = − 14
9 − 1 9
�
−3x (5) 1 + 4x (4) 3 �
= 2.3928 x (5) 3 = − 33
7 − 1 7
� x (5) 1 + 2x 2 (5) �
= −5.9555 Sixth approximation:
x (6) 1 = 2 − 1 5
� x (5) 2 + 2x (5) 3 �
= 3.9036 x (6) 2 = − 14
9 − 1 9
� −3x (6) 1 + 4x (5) 3 �
= 2.3925 x (6) 3 = − 33
7 − 1 7
� x (6) 1 + 2x 2 (6) �
= −5.9555
Exercise 3.5 Solve the following system of linear equation using Jacobi’s method and Gauss Seidel method
1.
5x + 2y + z = 12 x + 4y + 2z = 15 x + 2y + 5z = 20 2.
8x − 3y + 2z = 20 6x + 3y + 12z = 35 4x + 11y − z = 33 3.
2x + 10y + z = 51 x + 2y + 10z = 61 10x + y + 2z = 44
�
3.4 EIGEN VALUE PROBLEMS 43 3.4 EIGEN VALUE PROBLEMS
Consider the eigen value problem
Ax = λx (3.4)
The eigen values of a matrix A are given by the roots of the characteristic equation
det(A − λI) = 0 (3.5)
If the matrix A is of order n, then expanding the determinant, we obtain the characteristic equation as
p( λ) = (−1) n λ n + a 1 λ n−1 + . . . + a n−1 λ + a n = 0 (3.6) A nonzero vector x i such that
Ax i = λ i x i
is called the eigenvector corresponding to the eigenvalue λ i .
�
Example 3.12 Find the eigen values and the corresponding eigenvectors for the matrix A =
� 3 1 2 2
� B =
1 2 1
6 −1 0
−1 −2 −1
�
Solution: a). The characteristic polynomial is
� �
� � 3 − λ 1 2 2 − λ
� �
� � = (3 − λ )(2 − λ ) − 2 = λ 2 − 5λ + 4 characterstics equation : λ 2 − 5λ + 4 = 0 Therefore, the eigenvalues are λ 1 = 4, λ 2 = 1
The eigenvector of A corresponding to λ 1 = 4, Ax = λx ⇒ x = (1 1) T The eigenvector of A corresponding to λ 2 = 1, Ax = λx ⇒ x = (1 −2) T 3.4.1 Power Method
Power method is used to determine the largest eigenvalue (in magnitude) and the corresponding eigenvector of the system
Ax = λx Let λ 1 ,λ 2 , . . . ,λ n are distinct eigen values such that
|λ 1 | > |λ 2 | > ... > |λ n |
and v 1 , v 2 , . . . , v n be the corresponding eigenvectors. The method is applicable if a complete system of n linearly independent eigen vectors exist, even though some of the eigen values λ 1 ,λ 2 , . . . , λ n , may not be distinct. Then, any eigenvector v in the space of eigenvectors v 1 ,v 2 , ..., v n can be written as a linear combination of these vectors. That is,
v = c 1 λ 1 v 1 + c 2 λ 2 v 2 + . . . + c n λ n v n (3.7)
Premultiplying by A and substituting Av 1 = λ 1 v 1 , Av 2 = λ 2 v 2 , ..., Av n = λ n v n , we get Av = c 1 λ 1 v 1 + c 2 λ 2 v 2 + ... + c n λ n v n
= λ 1
�
c 1 v 1 + c 2 �λ 2
λ 1
�
v 2 + ... +c n �λ n
λ 1
� v n
�
Premultiplying repeatedly by A and simplifying, we get A 2 v = λ 1 2
�
c 1 v 1 + c 2 �λ 2
λ 1
� 2
v 2 + ... + c n �λ n
λ 1
� 2
v n
�
...
A k v = λ 1 k
�
c 1 v 1 + c 2 �λ 2
λ 1
� k
v 2 + ... + c n �λ n
λ 1
� k
v n
�
(3.8)
A k+1 v = λ 1 k+1
�
c 1 v 1 + c 2 �λ 2
λ 1
� k+1
v 2 + ... + c n �λ n
λ 1
� k+1
v n
�
(3.9) As k → ∞, the right hand sides of (3.8) and (3.9) tend to λ 1 k c 1 v 1 and λ 1 k+1 c 1 v 1 , since |λ i /λ 1 | <
1,i = 2,3,...,n. Both the right hand side vectors in (3.8), (3.9)
�
c 1 v 1 + c 2 �λ 2
λ 1
� k
v 2 + ... + c n �λ n
λ 1
� k
v n
� , and �
c 1 v 1 + c 2 �λ 2
λ 1
� k+1
v 2 + ... +c n �λ n
λ 1
� k+1
v n
�
tend to c 1 v 1 , which is the eigen vector corresponding to λ 1 . The eigen value λ 1 is obtained as the ratio of the corresponding components of A k+1 v and A k v. That is,
λ 1 = lim
k→∞
(A k+1 v) r
(A k v) r , r = 1,2,3,...,n (3.10)
where the suffix r denotes the rth component of the vector. Therefore, we obtain n ratios, all of them tending to the same value, which is the largest eigen value in magnitude, |λ 1 |
As k → ∞ , premultiplication each time by A, may introduce round-off errors. In order to keep the round-off errors under control, we normalize the vector before premultiplying by A.
The normalization that we use is to make the largest element in magnitude as unity. If we use this normalization, a simple algorithm for the power method can be written as follows. Let v 0 be a nonzero arbitrary initial vector (non-orthogonal to v 1 ) and find
y k+1 = Av k , ⇒ v k+1 = y k+1
m k+1
where m k+1 is the largest element in magnitude of y k+1 . Now, the largest element in magnitude of v k+1 is one unit. Then (3.10) can be written as
λ 1 = lim
k→∞
(y k+1 ) r
( y k ) r , r = 1,2,3,...,n and v k+1 is the required eigen vector.
R The iterations are stopped when all the magnitudes of the differences of the ratios are less
than the given error tolerance.
3.4 EIGEN VALUE PROBLEMS 45
�
Example 3.13 Determine the numerically largest eigen value and the corresponding eigen vector of the matrix,
A =
3 2 3 2 2 5 1 1 1
using the power method correct to three decimal place.
�Solution: Take v 0 = [1 1 1] T we find
y 1 = Av 0 =
3 2 3 2 2 5 1 1 1
1 1 1
=
8 9 3
= 9
0.8889 1 0.3333
⇒ v 1 =
0.8889 1 0.3333
, λ 1 = 9
y 2 = Av 1 =
3 2 3 2 2 5 1 1 1
0.8889 1 0.3333
=
5.6667 5.4444 2.2222
= 5.6667
1
0.9608 0.3922
⇒ v 2 =
1 0.9608 0.3922
, λ 2 = 5.6667
y 3 = Av 2 =
3 2 3 2 2 5 1 1 1
1 0.9608 0.3922
=
6.0980 5.8823 2.3529
= 6.0980
1
0.9646 0.3859
⇒ v 3 =
1
0.9646 0.3859
, λ 3 = 6.0980
y 4 = Av 3 =
3 2 3 2 2 5 1 1 1
1 0.9646 0.3859
=
6.0868 5.8585 2.3505
= 6.0868
1
0.9625 0.3862
⇒ v 4 =
1
0.9625 0.3862
, λ 4 = 6.0868
y 5 = Av 4 =
3 2 3 2 2 5 1 1 1
1 0.9625 0.3862
=
6.0835 5.8558 2.3487
= 6.0835
1
0.9626 0.3861
⇒ v 5 =
1
0.9626 0.3861
, λ 5 = 6.0835
y 6 = Av 5 =
1 1 −2 4 −2 1 3 −1 3
1
0.9626 0.3861
=
6.0833 5.8555 2.3486
= 6.0833
1 0.9626 0.3861
⇒ v 5 =
1 0.9626 0.3861
, λ 5 = 6.0833
Since |λ 5 − λ 4 | = |6.0833 − 6.0835| = 0.0002 < 0.5 × 10 −3 , thus, λ = 6.0833 is the largest eigenvalue of A and v =
1
0.9626 0.3861
is the corresponding eigenvector correct to three decimal place.
Exercise 3.6 Determine the largest eigen value and the corresponding eigen vector of the matrix,
A =
2 −1 0
−1 2 −1
0 −1 2
B =
1 2 1
6 −1 0
−1 −2 −1
using the power method correct to three decimal place.
�3.4.2 Inverse Power Method
Inverse power method is used to find the smallest eigenvalue in magnitude and the corresponding eigenvector of a given matrix A.
If λ is an eigenvalue of A, then 1
λ is an eigenvalue of A −1 corresponding to the same eigenvector.
The smallest eigenvalue in magnitude of A is the largest eigenvalue in magnitude of A −1 . To find the smallest eigen value choose an arbitrary vector y 0 (non-orthogonal to x) and applying the power method on A −1 .
�
Example 3.14 Determine the smallest eigen value and the corresponding eigen vector of the matrix,
A =
3 1 2 1 2 3 2 2 3
using the invers power method correct to two decimal place.
�Solution: We have A −1 =
0.3750 −0.3750 0.1250
−0.3750 −0.6250 0.8750 0.1250 0.8750 −0.6250
Take v 0 = [1 1 1] T we find,
A −1 v 0 =
0.3750 −0.3750 0.1250
−0.3750 −0.6250 0.8750 0.1250 0.8750 −0.6250
1 1 1
=
0.1250
−0.1250 0.375
= 0.375
0.3333
−0.3333 1
⇒ v 1 =
0.3333
−0.3333 1
, µ 1 = 0.375
A −1 v 1 =
0.3750 −0.3750 0.1250
−0.3750 −0.6250 0.8750 0.1250 0.8750 −0.6250
0.3333
−0.3333 1
=
0.3750 0.9583
−0.8750
= 0.9583
0.3913 1
−0.9131
⇒ v 2 =
0.3913 1
−0.9131
, µ 2 = 0.9583
A −1 v 2 =
0.3750 −0.3750 0.1250
−0.3750 −0.6250 0.8750 0.1250 0.8750 −0.6250
0.3913 1
−0.9131
=
−0.3424
−1.5707 1.4946
= 1.5707
−0.2180 0.9516 −1
⇒ v 3 =
−0.2180 0.9516 −1
, µ 3 = 1.5707
A −1 v 3 =
0.3750 −0.3750 0.1250
−0.3750 −0.6250 0.8750 0.1250 0.8750 −0.6250
−0.2180 0.9516 −1
=
0.4122 1.5394
−1.4970
= 1.5394
0.2678 1
−0.9724
⇒ v 4 =
0.2678 1
−0.9724
, µ 4 = 1.5394
3.4 EIGEN VALUE PROBLEMS 47
A −1 v 4 =
0.3750 −0.3750 0.1250
−0.3750 −0.6250 0.8750 0.1250 0.8750 −0.6250
0.2678 1
−0.9724
=
−0.3961
−1.5763 1.5162
= 1.5763
−0.2513 0.9619 −1
⇒ v 5 =
−0.2513 0.9619 −1
, µ 5 = 1.5763
A −1 v 5 =
0.3750 −0.3750 0.1250
−0.3750 −0.6250 0.8750 0.1250 0.8750 −0.6250
−0.2513 0.9619 −1
=
0.4010 1.5609
−1.5076
= 1.5609
0.2569 1
−0.9658
⇒ v 6 =
0.2569 1
−0.9658
, µ 6 = 1.5609
A −1 v 6 =
−0.3994
−1.5664
−1.5076
= 1.5664
−0.2550 0.9645 −1
⇒ v 7 =
−0.2550 0.9645 −1
, µ 7 = 1.5664
A −1 v 7 =
0.4000 1.5646
−1.5097
= 1.5646
−0.2556 1
−0.9649
⇒ v 8 =
0.2556 1
−0.9649
, µ 8 = 1.5646
Since |µ 8 − µ 7 | = |1.5646 − 1.5664| = 0.0018 < 0.5 × 10 −2 , thus, µ = 1.5646 is the largest eigenvalue of A −1 and v = �
0.2556 1 −0.9649 � T
is the corresponding eigenvector correct to three decimal place.
Therefore, λ = 1
µ = 1
1.5646 = 0.6391 is the smallest eigenvalue of A. and v = �
0.2556 1 −0.9649 � T
is the corresponding eigenvector.
�
