Applied Linear Algebra
——–
Chapter 1: Solving Linear Systems and the Terminology of Vectors and Matrices
Section 4: Elementary matrices, permutation matrices, LU factorization
Ivan Contreras, Sergey Dyachenko and Robert G Muncaster University of Illinois at Urbana-Champaign
Echelon Form of a Matrix
Let us look back at
2u+v+w =5 4u−6v+0w = −2
−2u+7v+2w =9
We have defined matrix multiplication so that this sytem is Ax =b where
A= 2 1 1 4 −6 0 −2 7 2 , b= 5 −2 9 , x = u v w
After G-E we have an equivalent system (i.e. same solutions) Ux =c where U = 2 1 1 0 −8 −2 0 0 1 | {z }
row Echelon form of A
, c = 5 −12 2
U is called the row Echelon form of A (REF).
Gaussian Elimination with Elementary Matrices
Recall the steps of G-E for A:
Step 1: Row 2 + (-2) x Row 1 , equiv: mult by E =E12(−2)
Step 2: Row 3 + (1) x Row 1 , equiv: mult by F =E13(1)
Step 3: Row 3 + (1) x Row 2 , equiv: mult by G =E23(1)
Therefore Ax =b⇐⇒ GFEA | {z } U x =GFEb | {z } c i.e. U =GFEA, c =GFEb This is how U and c are related to A and b.
Introduction to Inverses
Can we find an elementary matrix that “undoes” what Eij(a) does to a
matrix (by multiplication)? Since Eij(a)adds a times row i to row j , we
can undo this by immediately after applying Eij(−a): add−a times row i
to row j . Thus
Eij(−a)Eij(a) =I
We call Eij(−a) the inverse of Eij(a) and denote it by Eij(a)−1. Thus
E−1 = 1 0 0 2 1 0 0 0 1 since E−1E = 1 0 0 2 1 0 0 0 1 1 0 0 −2 1 0 0 0 1 = 1 0 0 0 1 0 0 0 1
The A = LU Factorization
Similarly set F−1 = 1 0 0 0 1 0 −1 0 1 , G−1 = 1 0 0 0 1 0 0 −1 1 Then U =GFEA=⇒G−1U =G−1GFEA=IFEA=FEA F−1G−1U =F−1FEA=IEA=EA E−1F−1G−1U =E−1EA=IA=A
We can do a similar calculation beginning with c =GFEb. Therefore
Properties of L and U
A direct calculation gives
L= 1 0 0 2 1 0 −1 −1 1 , U = 2 1 1 0 −8 −2 0 0 1
U is the Echelon form of A and it is upper triangular with the pivots on the diagonal. L is always lower triangular with 1’s on the diagonal. L can be constructed directly from the multipliers in G-E. In general
E = 1 0 0 a 1 0 0 0 1 , F = 1 0 0 0 1 0 b 0 1 , G = 1 0 0 0 1 0 0 c 1 and so L= 1 0 0 −a 1 0 0 0 1 1 0 0 0 1 0 −b 0 1 1 0 0 0 1 0 0 −c 1 = 1 0 0 −a 1 0 −b −c 1
i.e. the negatives of the multipliers appear below the diagonal.
Applications of A = LU
So what is the value of the factorization A=LU? In essence Ax =b has been replaced with two new systems:
Ux =c and Lc =b
Start with b and solve the second system for c. Then solve the first system for x . Why is this better than just Ax =b? Look more closely:
Ux =c | {z } linear system for x
upper triangular back substitution
efficient
Lc =b | {z } linear system for c
lower triangular forward substitution
efficient
This is useful if one needs to solve Ax =b for a large collection of b’s but the same A.
An Example
Here is a factorization found by G-E:
1 −1 0 0 −1 2 −1 0 0 −1 2 −1 0 0 −1 2 | {z } A = 1 0 0 0 −1 1 0 0 0 −1 1 0 0 0 −1 1 | {z } L 1 −1 0 0 0 1 −1 0 0 0 1 −1 0 0 0 1 | {z } U
Then the two triangular systems are
x1−x2 =c1 c1=1 x2−x3 =c2 −c1+c2=2 x3−x4 =c3 −c2+c3=1 x4 =c4 −c3+c4=3 where b= 1 2 1 3
A quick calculation gives
c1=1, c2 =3, c3 =4, c4 =7, x4 =7, x3 =11, x2 =14, x1 =15.
Row Exchanges in Gaussian Elimination
So far G-E as we have defined it requires: same number of equations as knowns
only one operation: adding a multiple of one row to another pivots in positions 11, 22, 33, etc.
What do we do if the third item breaks down, i.e. a zero appears in one of the diagonal positions?
Example 0 2 3 4 2 −5 | {z } no pivot in 11 position ⇐⇒ 0u+2v =2 3u+4v = −5 ⇐⇒ 3u+4v = −5 0u+2v =2 ⇐⇒ 3 4 0 2 −5 2 | {z }
G-E can now proceed
Permutation Matrices
Permutation matrices P: A permutation matrix is any matrix obtained from I by permuting some of its rows. For example
P = Row 1 → Row 3 and Row 3 → Row 1 of I
= 0 0 1 0 1 0 1 0 0 Note that PA= 0 0 1 0 1 0 1 0 0 a11 a12 a13 a21 a22 a23 a31 a32 a33 = a31 a32 a33 a21 a22 a23 a11 a12 a13
that is, multiplying a matrix by a permutation matrix permutes its rows according to the exchanges for that permutation matrix.
More on Permutations
Another example:
P =Row 1 → Row 2, Row 2 → Row 3, Row 3 → Row 1 of I
= 0 0 1 1 0 0 0 1 0 Then PA= 0 0 1 1 0 0 0 1 0 1 2 3 4 5 6 7 8 9 = 7 8 9 1 2 3 4 5 6 | {z }
same row exchanges as in P
Conclusion: We can do row exchanges by multiplying by a permutation matrix
The PA = LU Factorization
Theorem: If A is non-singular, then there is a permutation matrix P such that PA=LU
i.e. G-E can be performed on PA
A non-singular =⇒G-E with row exchanges works
A singular=⇒ G-E with row exchanges fails (you lose a pivot) Example: 0 0 2 1 3 3 1 4 5 → 1 3 3 0 0 2 1 4 5 | {z } Row 1↔Row 2 → 1 3 3 0 0 2 0 1 2 → 1 3 3 0 1 2 0 0 2 | {z } Row 2↔Row 3 so
P = [Row 2 ↔ Row 3] [Row 1 ↔ Row 2]
= 1 0 0 0 0 1 0 1 0 0 1 0 1 0 0 0 0 1 = 0 1 0 0 0 1 1 0 0
