Matrix Calculations: Applications of Eigenvalues and Eigenvectors; Inner Products
H. Geuvers
Institute for Computing and Information Sciences – Intelligent Systems Radboud University Nijmegen
Version: spring 2015
Outline
Applications of Eigenvalues and Eigenvectors
Inner products
Political swingers re-re-revisited, part I
• Recall the political transisition matrix P =0.8 0.1
0.2 0.9
= 101 8 1 2 9
• Eigenvalues λ are obtained via det(P − λ I2) = 0:
(108 − λ)(109 − λ) −101 ·102 = λ2−1710λ +107 = 0
• Solutions via “abc”
1 2
17 10±
q
17 10
2
− 2810
= 12
17 10±q
289
100−280100
= 12
17 10±q
9 100
= 12
17 10±103
• Hence λ = 1 ·20 = 1or λ = 1 ·14 = 7 .
Political swingers re-re-revisited, part II
λ = 1 solve: −0.2x + 0.1y = 0
0.2x + −0.1y = 0 giving (1, 2) as eigenvector
• Indeed 0.8 0.1 0.2 0.9
·1 2
=0.8 + 0.2 0.2 + 1.8
=1 2
= 11 2
X
λ = 0.7 solve: 0.1x + 0.1y = 0
0.2x + 0.2y = 0 giving (1, −1) as eigenvector
• Check:
0.8 0.1 0.2 0.9
· 1
−1
=0.8 − 0.1 0.2 − 0.9
= 0.7
−0.7
= 0.7 1
−1
X
Political swingers re-re-revisited, part III
• The eigenvalues 1 and 0.7 aredifferent, and indeed the eigenvectors (1, 2) and (1, −1) are independent
• The coordinate-translation TV ⇒B from the eigenvector basis V = {(1, 2), (1, −1)} to the standard basis
B = {(1, 0), (0, 1)} consists of the eigenvectors:
TV ⇒B = 1 1 2 −1
• In the reverse direction:
TB⇒V = TV ⇒B−1
= −1−21 −1 −1
−2 1
= 131 1 2 −1
Political swingers re-re-revisited, part IV
We explicitly check thediagonalisation equation:
TV ⇒B· 1 0 0 0.7
!
· TB⇒V = 1 1 2 −1
!
· 1 0 0 0.7
!
·13 1 1 2 −1
!
= 13 1 0.7 2 −0.7
!
· 1 1 2 −1
!
= 13 2.4 0.3 0.6 2.7
!
= 0.8 0.1 0.2 0.9
!
= P, the original political transition matrix!
Political swingers re-re-revisited, part V
Thisdiagonalisation P = T ·1 0 0 0.7
· T−1 is useful foriteration
• P2 = T ·1 0 0 0.7
· T−1· T ·1 0 0 0.7
· T−1
= T ·1 0 0 0.7
·1 0 0 0.7
· T−1
= T ·12 0 0 (0.7)2
· T−1
• Pn = T ·(1)n 0 0 (0.7)n
· T−1
• lim
n→∞Pn = T ·1 0 0 0
· T−1 since lim
n→∞(0.7)n= 0
= 1 1 2 −1
·1 0 0 0
·131 1 2 −1
= 131 1 2 2
Political swingers re-re-revisited, part VI
• In an earlier lecture we wondered how to compute Pn·100 150
• We can now see that in the limit it goes to:
n→∞lim Pn·100 150
= 131 1 2 2
·100 150
= 13250 500
= 8313 16623
(This was already suggested earlier, but now we can calculate it!)
Recall the useful limit result
n→∞lim an= 0, for |a| < 1.
Rental car returns, part I
• Assume a car rental company with three locations, for picking up and returning cars, written as P,Q,R
• The weekly distribution historyshows:
Location P 60% stay at P 10% go to Q 30% go to R Location Q 10% go to P 80% stay at Q 10% go to R Location R 10% go to P 20% go to Q 70% stay at R
Rental car returns, part II
Two possible representations of these return distributions
1 As probabilistic transition system
?>=<
89:;P
0.6 0.1 ))
0.3 ..
GFED
@ABCQ
0.8
uu
oo 0.1
pp 0.1
?>=<
89:;R
0.7
JJ
eeJJJJJ 0.1
JJJJJJJJ
0.2
t99t tt tt tt tt tt t
Rental car returns, part III
2 As a transition matrix C =
0.6 0.1 0.1 0.1 0.8 0.2 0.3 0.1 0.7
= 101
6 1 1 1 8 2 3 1 7
This matrix C describes what is called aMarkov chain:
• all entries are in the unit interval [0, 1] of probabilities
• in each column, the entries add up to 1
Rental car returns, part IV
Task:
• Start from the following division of cars:
P = Q = R = 200 ie.
P Q R
=
200 200 200
• Determine the division of cars after two weeks
• Determine theequilibrium division, reached as the number of weeks goes to infinity
Rental car returns, part V
• Afterone week we have:
C ·
200 200 200
= 101
6 1 1 1 8 2 3 1 7
·
200 200 200
= 101
1200 + 200 + 200 200 + 1600 + 400 600 + 200 + 1400
=
160 220 220
• Aftertwo weeks we have:
C ·
160 220 220
= 101
6 1 1 1 8 2 3 1 7
·
160 220 220
= 101
960 + 220 + 220 160 + 1760 + 440 480 + 220 + 1540
=
140 236 224
Rental car returns, part VI
• For the equilibriumwe first compute eigenvalues and eigenvectorsof the transition matrix C
• The characteristic polynomial is:
0.6 − λ 0.1 0.1 0.1 0.8 − λ 0.2 0.3 0.1 0.7 − λ
= 1
1000
6 − 10λ 1 1
1 8 − 10λ 2
3 1 7 − 10λ
= 10001 h
(6 − 10λ)
(8 − 10λ)(7 − 10λ) − 2
−1
(7 − 10λ) − 1
+ 3
2 − 1(8 − 10λ)
i
= · · ·
= 10001 h
− 1000λ3+ 2100λ2− 1400λ + 300i
= −λ3+ 2.1λ2− 1.4λ + 0.3.
Rental car returns, part VII
• Next we solve −λ3+ 2.1λ2− 1.4λ + 0.3 = 0.
• We seek a trivial solution; again λ = 1 works!
• Now we can write
−λ3+ 2.1λ2− 1.4λ + 0.3 = (λ − 1)(−λ2+ 1.1λ − 0.3)
• We can apply the “abc” formula to the second part:
−1.1±√
(1.1)2−4·0.3
−2 = −1.1±
√ 1.21−1.2
−2
= −1.1±
√ 0.01
−2
= −1.1±0.1−2
• This yields additional eigenvalues: λ = 0.5 andλ = 0.6.
Rental car returns, part VIII
λ = 1 has eigenvector (4, 9, 7); indeed:
C ·
4 9 7
= 101
6 1 1 1 8 2 3 1 7
·
4 9 7
= 101
24 + 9 + 7 4 + 72 + 14 12 + 9 + 49
= 1
4 9 7
λ = 0.6 has eigenvector (0, −1, 1):
C ·
0
−1 1
= 101
6 1 1 1 8 2 3 1 7
·
0
−1 1
= 101
−1 + 1
−8 + 2
−1 + 7
= 0.6
0
−1 1
λ = 0.5 has eigenvector (−1, −1, 2):
C ·
−1
−1
= 101
6 1 1 1 8 2
·
−1
−1
= 101
−6 − 1 + 2
−1 − 8 + 4
= 0.5
−1
−1
Rental car returns, part IX
• Now: eigenvector base V = {(4, 9, 7), (0, −1, 1), (−1, −1, 2)}
and standard base as B = {(1, 0, 0), (0, 1, 0), (0, 0, 1)}.
• Then we can do change-of-coordinates back-and-forth:
TV ⇒B =
4 0 −1 9 −1 −1
7 1 2
TB⇒V = 201
1 1 1
25 −15 5
−16 4 4
• These translation matrices yield a diagonalisation:
C = TV ⇒B·
1 0 0
0 0.6 0 0 0 0.5
· TB⇒V
Rental car returns, part X
• Thus:
n→∞limCn = lim
n→∞TV ⇒B·
1n 0 0
0 (0.6)n 0 0 0 (0.5)n
· TB⇒V
= TV ⇒B·
1 0 0 0 0 0 0 0 0
· TB⇒V = 201
4 4 4 9 9 9 7 7 7
• Finally, the equilibriumstarting from P = Q = R = 200 is:
1 20
4 4 4 9 9 9 7 7 7
·
200 200 200
=
120 270 210
.
Length of a vector
• Intuitively, each vector v = (x1, . . . , xn) ∈ Rn has alength (aka. sizeor normor Euclidian length), written as kv k
• This kv k is a non-negative real number: kv k ∈ R, kv k ≥ 0
• Some special cases:
• n = 1: so v ∈ R, with kv k = |v |
• n = 2: so v = (x1, x2) ∈ R2 and with Pythagoras:
kv k2= x12+ x22 and thus kv k =q x12+ x22
• n = 3: so v = (x1, x2, x3) ∈ R3 and also with Pythagoras:
kv k2= x12+ x22+ x32 and thus kv k =q
x12+ x22+ x32
• In general, for v = (x1, . . . , xn) ∈ Rn, kv k =
q
x2+ x2+ · · · + x2
Distance between points
• Assume now we have two vectors v , w ∈ Rn, written as:
v = (x1, . . . , xn) w = (y1, . . . , yn)
• What is the distance between the endpoints?
• commonly written asd (v , w )
• again, d (v , w ) is a non-negative real
• For n = 2, d (v , w ) =
q
(x1− y1)2+ (x2− y2)2 = kv − w k = kw − v k
• This will be used also for other n, so:
d (v , w ) = kv − w k
Length is fundamental
• Distance can be obtained from length of vectors
• Interestingly, also anglescan be obtained from length!
• Both length of vectors and angles between vectors can de derived from the notion of inner product
Inner product definition
Definition
For vectors v = (x1, . . . , xn), w = (y1, . . . , yn) ∈ Rn define their inner productas the real number:
hv , w i = x1y1+ · · · + xnyn
= X
1≤i ≤n
xiyi
Note: Length kv k can be expressed via inner product:
kv k2 = x12+ · · · + xn2 = hv , v i, so kv k = phv, vi.
Inner products via matrix transpose
Recall matrix transposition
For an m × n matrix A, thetranspose AT is the n × m matrix A obtained by mirroring in the diagonal:
a11 · · · a1n ... am1 · · · amn
T
=
a11 · · · am1 ... a1n · · · amn
The inner product of v = (x1, . . . , xn), w = (y1, . . . , yn) ∈ Rn is then a matrix product:
hv , w i = x1y1+ · · · + xnyn= (x1 · · · xn) ·
y1
... yn
= vT · w .
Inner products and angles, part I
0 // //
h(1,0),(2,0)i=2
0
??//
h(1,0),(1,1)i=1
0 OO //
h(1,0),(0,1)i=0
0 __>>>
>>>>
//
h(1,0),(−1,1)i=−1
oo 0 //
h(1,0),(−1,0)i=−1
0
//
h(1,0),(−1,−1)i=−1
0
//
h(1,0),(0,−1)i=0
0
>
>>
>>
>> //
h(1,0),(1,−1)i=1
Reminder: cosine law
C • •
B A•
◦D
a c
c c c c c c
c c
b
γ
h d
c2= a2+ b2− 2ab cos(γ)
Proof: By Pythagoras b2 = h2+ d2 and c2= h2+ (a − d )2. Hence by subtraction:
b2−c2 = d2−a2+2ad −d2 = −a2+2ad and so c2 = a2+b2−2ad Recall cos(γ) = db, so by substituting d = b cos(γ) we are done. -
Inner products and angles, part II
0• v
w
◦ -
kv k
c c c c c c c
d (v , w ) = kv − w k
kw k* γ
Thecosine rulesays:
kv − w k2 = kv k2+ kw k2− 2kv k kw k cos(γ) That is:
cos(γ) = kv k2+ kw k2− kv − w k2 2kv k kw k
Let’s elaborate it . . .
Inner products and angles, part III
Starting from the cosine rule:
cos(γ) = kv k2+ kw k2− kv − w k2 2kv k kw k
= x12+ · · · + xn2+ y12+ · · · + yn2− (x1− y1)2− · · · − (xn− yn)2 2kv k kw k
= 2x1y1+ · · · + 2xnyn
2kv k kw k
= x1y1+ · · · + xnyn kv k kw k
= hv , w i
kv k kw k remember this: cos(γ) = hv , w i kv k kw k Thus,anglesbetween vectors are expressible via the inner product (since kv k =phv, vi).
Recall the cosine function
Linear algebra in gaming, part I
• Linear algebra plays an important role in game visualisation
• Here: simple illustration, borrowed from blog.wolfire.com (More precisely: http://blog.wolfire.com/2009/07/
linear-algebra-for-game-developers-part-2)
• Recall: cosine cosfunction is positive on angles between -90 and +90 degrees.
Linear algebra in gaming, part II
• Consider a guardG and heroH in:
• The guard is at position (1, 1), facing in direction D =1 1
, with a 180 degrees field of view
• The hero is at (3, 0). Is he within view?
Linear algebra in gaming, part III
• The direction vector V is: V = 3 0
−1 1
= 2
−1
• The angle γ between D and V must be between -90 and +90!
• Hence we must have: cos(γ) = kDk·kV khD,V i ≥ 0
• Since kDk ≥ 0 and kV k ≥ 0, it suffices to have: hD, V i ≥ 0
• Well, hD, V i = 1 · 2 + 1 · −1 = 1. Hence H is within sight!
Linear algebra in gaming, part IV
• Now what if the guard’s field of view is 60 degrees?
• Inbetween -30 and +30 degrees we have cos ≥ 12√
3 ∼ 0.87
• The cosine of the actual angle γ between D and V is:
cos(γ) = hD, V i
kDk · kV k = 1 · 2 + 1 · −1
√
12+ 12·p22+ (−1)2
= 1
√ 2 ·√
5 ∼ 0.31 < 0.87
Inner product for vector spaces in general
Definition
We say that a vector space V has aninner product if there is a special function:
V × V h−,−i //R satisfying the following five requirements.
1 hv , v i ≥ 0
2 hv , v i = 0 if and only if v = 0
3 hv , w i = hw , v i
4 hv + v0, w i = hv , w i + hv0, w i (similarly in w , by 3)
5 hav , w i = ahv , w i (and similarly in w , by 3) Given such inner product,define length, distance and angle:
kv k =phv, vi d (v , w ) = kv − w k cos(γ) = hv , w i kv k kw k.
Hilbert spaces
• The notion of inner product turns out to be very general and flexible
• It combines algebra (vectors) and geometry (distance)
• It forms the basis of Hilbert spaces (involving “completeness”)
• Our examples: inner product on Rn
• Many other examples exist, involving for instance distance between functions
• Important topic in abstract analysis and (quantum mechanics)
• Our main applications: projections andapproximations