CHAPTER I. Introduction

Introduction

1

In this lecture, we discuss theory, numerics and application of advanced prob- lems in linear algebra:

(II) matrix equations (example: solveAX ` XB “ C),

(III) matrix functions: computef pAqorf pAqb, whereA P C^nˆn,b P C^n, (IV) randomized algorithms.

The main focus is on problems defined by real matrices/vectors. In most chap- ters, we have to make the distinction between problems defined by

• dense matrices of small /moderate dimensions and

• large, sparse matrices, e.g. A P C^nˆn,n ą 10⁴or greater, but onlyOpnq nonzero entries, often from PDEs.

We first have to review two important standard problems in numerical linear algebra, namely solving linear systems of equations and eigenvalue problems.

I.1 Linear systems of equations

We consider the linear system

Ax “ b, (I.1)

withA P C^nˆnpR^nˆnq, b P CⁿpRⁿq. The linear system (I.1) admits a unique solution, if and only if

• there exists an inverseA^´1

• detpAq ‰ 0

• no eigenvalues/ singular values are equal to zero

• . . .

Numerical methods for small and dense A P C

Gaussian Elimination (LU-factorization):

We decomposeAsuch that A “ LU, L “

„

@

@

1

1

@@



, U “

„

@

@

@

 . We obtain, that

(I.1) ô LU x “ b ô x “ U^´1pL^´1bq.

Hence, we solve (I.1) in two steps:

1. SolveLy “ bvia backward substitution.

2. SolveU x “ yvia backward substitution.

This procedure is numerically more robust with pivotingP AQ “ LU, where P, Q P C^n,nare permutation matrices. This method has a complexity ofOpn³q and is, therefore, only feasible for small (moderate) dimensions.

QR-decomposition:

We decomposeAinto a product ofQandRwhere Qis an orthogonal matrix andR is an upper triangular matrix leading to the so-called Gram-Schmidt or the modified Gram-Schmidt algorithm. Numerically this can be done either with Givens rotations or with Householder transformations.

Methods for large and sparse A P C

Storing and computing dense LU-factors is infeasible for large dimensions n (Opn²q memory, Opn³q flops). One possibility are sparse direct solvers, i.e.

find permutation matrices P and Q, such that P AQ “ LU has sparse LU- factors (cheap forward/ backward substitution andOpnqmemory).

Example: We consider the LU-factorization of the following matrix

A “

«˚ . . . ˚ ..

. ..

.

˚ ˚

ff

“

„

@

@

1

1

@@

 „

@

@

@

 . With the help of permutation matricesP andQ, we can factorize

P AQ “

«˚ ˚

... .. .

˚ . . . ˚

ff

“

«˚ .. .

...

˚ ˚

ff «˚ . . . ˚ ...

˚

ff .

Algorithm 1 Arnoldi method Input: A P C^nˆn, b P Cⁿ

Output: Orthonormal basisQkof (I.2)

1: Setq1 “ _}b}^{b and}Qq:“ rq1s^.

2: forj “ 1, 2, . . . do

3: Setz “ Aqj.

4: Setw “ z ´ QjpQ^H_jzq.

5: Setq_j`1 “ _}w}^{w .}

6: SetQj`1“ rQj, qj`1s^.

7: end for

Finding suchP andQand still ensuring numerical robustness is difficult and based e.g. on graph theory.

In MATLAB, sparse-direct solvers are found in the "z"-command: x “ Azb or lupAq-routine. (Never useinvpAq!)

Iterative methods

Often an approximationx « xp is sufficient. Hence, we generate a sequence x₁, x₂, . . . , x_kby an iteration, such that

kÑ8lim xk“ x “ A^´1b

and each x_k, k ě 1 is generated efficiently (only Opnq computations). Of course, we wantxk« xfork ! n.

Idea: Search approximated solution in a low-dimensional subspaceQ_k Ă C^n, dimpQ_kq “ k. LetQ_kbe given asrangepQ_kq “ Qkfor a matrixQ_k P C^nˆk. A good choice of the subspace is the Krylow-subspace

Q_k “ KkpA, bq “ spantb, Ab, . . . , A^k´1bu. (I.2) It holds for z P KkpA, bq^{, that} z “ ppAqb for a polynomial of degree k ´ 1 p P Π_k´1. An orthonormal basis ofK_kpA, bqcan be constructed with the Arnoldi process presented in Algorithm1.

The Arnoldi process requires matrix-vector productsz “ Aq. These are cheap for sparseAand therefore feasible for large dimensions.

We find an approximationxkP x0` Qkby two common ways:

• Galerkin-approach:

Imposer “ b ´ Ax_kK rangepQkq ô pQ^H_kAQ_kqyk“ Q^H_kb. We have to solve ak-dimensional systemñ^{low costs.}

• Minimize the residual:

min

xkPrangepQkq}b ´ Axk}

in some norm. Ifx_kis not good enough, we expandQ_k.

There are many Krylov-subspace methods for linear systems. (Simplification forA “ A^H: Arnoldiù^Lanczos)

In practice: Convergence acceleration by preconditioning:

Ax “ b ô P^´1Ax “ P^´1b

for easily invertibleP P C^n,nandP^´1A"nicer" thanA(ùLiterature NLA I).

Another very important building block is the numerical solution of eigenvalue problems.

I.2 Eigenvalue problems (EVP)

For a matrixA P C^n,n we want to find the eigenvectors 0 ‰ x P C^{n and the} eigenvaluesλ P C^{such that}

Ax “ λx.

The set of eigenvaluesΛpAq “ tλ1, . . . , λnuis called the spectrum ofA.

Small, dense problems:

Computing the Jordan-Normal-Form (JNF)

X^´1AX “ J “ diagpJs1pλ1q, . . . , Jskpλkqq, Jsjpλjq :“

» –

λ_j 1 . .. 1

λj

fi fl to several eigenvalues and eigenvectors is numerically infeasible, unstable (NLA I).

Theorem I.1 (Schur): For all A P C^nˆn exists a unitary matrix Q P C^n,n (Q^HQ “ I), such that

Q^HAQ “ R “

»

— –

λ₁

˚

. ..

0

fi ffi fl loooooooomoooooooon

Schur form of A

withλi P ΛpAqin arbitrary order.

The Schur form can be numerically stable computed inOpn³q (NLA I) by the Francis-QR-algorithm. It is this basis for dense eigenvalue computations. In MATLAB we userQ, Rs “ schurpAq. Additionally, the routineeigspAquses the Schur form. In general, the columns ofQare no eigenvectors ofA, butQk “ Qp:, 1 : kqspans anA-invariant subspace for allk:

AQ_k“ QkR_k, for a matrix R_kP C^{kˆk with} ΛpR_kq Ď ΛpAq.

But because of theOpn³q complexity and Opn²q memory, the Schur form is infeasible for large and sparse matricesA.

Eigenvalue problems defined by large and sparse matrices A can again be treaded with the Arnoldi-process and projections on the Krylov-subspace K_kpA, bq “ rangepQ_kq^{. We obtain the} approximated eigenpair x_k “ Qky_k « x, µ « λ by using the Galerkin-condition on the residual of the eigenvalue problem:

r_k“ Axk´ µ xkK rangepQkq ô Q^H_kAQ_ky_k“ µ yk,

which means pµ, ykq are the eigenpairs of the k ˆ k-dimensional eigenvalue problem forQ^H_kAQk. This small eigenvalue problem is solvable by the Francis- QR-method. This is the basis of theeigspAqroutine in MATLAB for computing a few (! n) eigenpairs ofA.

Summary: Solving linear systems and eigenvalue problems is for small or large and sparse matricesAno problem!

Matrix Equations

7

II.1 Preliminaries

Up to now we know linear systems of equations Ax “ b,

whereA P R^nˆnandb P Rⁿare given andx P Rⁿhas to be found.

In this course we consider more general equations

F pXq “ C, (II.1)

whereF : R^qˆr Ñ R^pˆs, C P R^pˆs is given, andX P R^qˆr has to be found.

Equations of the form (II.1) are called algebraic matrix equations.

II.1.1 Examples of Algebraic Matrix Equations

1) F pXq “ AXB,i. e., (II.1) is

AXB “ C.

2) Sylvester equations:

AX ` XB “ C, 3) algebraic Lyapunov equations:

a) continuous time:

AX ` XA^T “ ´BB^T, X “ X^T, b) discrete time:

AXA^T ´ X “ ´BB^T, X “ X^T, 4) algebraic Riccati equations:

a) continuous time:

A^TX ` XA ´ XBR<sup>´1</sup>B<sup>T</sup>X ` C^TQC “ 0, X “ X^T, b) discrete time:

A^TXA ´ X ´ pA^TXBqpR ` B^TXBq^´1pB^TXAq

` C^TQC “ 0, X “ X^T.

c) non-symmetric

AX ` XM ´ XGX ` Q “ 0.

Examples 1) – 3) are linear matrix equations, since the mapF is linear. Equa- tions of the type 4) are called quadratic matrix equations. The goal of this lecture is to understand the solution theory as well as numerical algorithms for the above matrix equations. Our focus will be on the equations 2),3a) and 4a) since these are the equations mainly appearing in the applications.

The term continuous-/discrete-time in 3a,b), 4a,b) refers to applications in con- text of continuous-time dynamical systems

xptq “ Axptq,9 t P R or discrete-time dynamical systems

x_k`1 “ Axk, k P N,

respectively. More info in courses on control theory or model order reduction.

There are also variants of the above equations containingX^T orX^H – these will not play a prominent role here. Furthermore, there are matrix equations whereX “ Xptqis a matrix-valued function andFcontains derivative informa- tion ofX. Such equations are called differential matrix equations, for example the differential Lyapunov equation

Xptq ` Aptq9 <sup>T</sup>Xptq ` XptqAptq ` Qptq “ 0,

whereA, Q P Cprt₀, t_fs, R^nˆnq^{, and}X P C¹prt0, t_fs, R^nˆnq^withQptq “ Qptq^T ě 0andXptq “ Xptq^T for allt P rt₀, t_fsand the initial conditionXpt₀q “ X0.

II.2 Linear Matrix Equations

In this chapter we discuss the solution theory and the numerical solution of linear matrix equations as defined precisely below.

Definition II.1 (linear matrix equation): LetAi P C^pˆq, Bi P C^{rˆs, and} C P C^pˆs, i “ 1, . . . , kbe given. An equation of the form

k

ÿ

i“1

A_iXB_i“ C ^(II.2)

is called a linear matrix equation.

II.2.1 Solution Theory

To discuss solvability and uniqueness of solutions of (II.2) we need the following concepts.

Definition II.2 (vectorization operator and Kronecker product): For X “

“x₁ . . . xm

‰“

»

— –

x₁₁ . . . x_1m ... ... x_n1 . . . x_nm

fi ffi fl P C

nˆmandY P C^pˆq

a) the vectorization operatorvec : C^nˆmÑ C^nmis given by

vecpXq :“

»

— –

x1

... xm

fi ffi fl,

b) the Kronecker product is given by

X b Y “

»

— –

x11Y . . . x1mY ... ... xn1Y . . . xnmY

fi ffi fl P C

npˆmq.

Lemma II.3: ForT P C^nˆm,O P C^{mˆp, and}R P C^{pˆrit holds} vecpT ORq “`R^T b T˘ vecpOq

(Note that it has to be R^T in the above formula, even if all the matrices are complex.)

Proof. Exercise.

By this lemma, and the obvious linearity ofvecp¨q, we see that

k

ÿ

i“1

AiXBi “ C ô

k

ÿ

i“1

`B^T_i b Ai

˘ looooooomooooooon

A

vecpXq loomoon

X

“ vecpCq , looomooon

B

and we find that (II.2) has a unique solution if and only if the linear system of equationsAX “ Bhas one. Equivalently,Ahas to be nonsingular.

Theorem II.4: The linear matrix equation (II.2) withps “ qrhas a unique solu- tion iff all eigenvalues of the matrix

A “

k

ÿ

i“1

`B_i^T b Ai

˘

are non-zero.

In the following we will focus on the casek ď 2 and p “ s “ q “ r, since Lyapunov equationspk “ 2, A1 “ A, B1 “ A2 “ In, B2 “ A^Tqand Sylvester equationspk “ 2, A1 “ A, B2“ B, A2 “ In, B1 “ Imqare important special cases of interest in applications.

To check the above condition for unique solvability, we do not want to evaluate the Kronecker products. Therefore, we now derive easily checkable conditions based on the original matrices.

Lemma II.5: a) Let W, X, Y, Z be matrices such that the products W X and Y Z are defined. ThenpW b Y qpX b Zq “ pW Xq b pY Zq^.

b) Let S, G be nonsingular matrices. Then S b G is nonsingular, too, and pS b Gq^´1 “ S^´1b G^´1.

c) IfAandB, as well as,CandDare similar matrices thenA b CandB b D are similar (Asimilar toB ifDQnonsingular s.t. A “ Q^´1BQ).

d) LetX P C^{nˆn and}Y P C^mˆm be given. Then

ΛpX b Y q “ tλµ | λ P ΛpXq, µ P ΛpY qu.

Proof. Exercise.

Theorem II.6 (Theorem of Stephanos): Let A P C^{nˆn and} B P C^{mˆm with} ΛpAq “ tλ₁, . . . , λ_nu,ΛpBq “ tµ₁, . . . , µ_mube given. For a bivariate polyno- mialppx, yq “

řk i,j“0

c_ijxⁱy^j we define by

ppA, Bq :“

k

ÿ

i,j“0

cijpAⁱb B^jq

a polynomial of the two matrices. Then the spectrum ofppA, Bqis given by ΛpppA, Bqq “ tppλ_r, µ_sq | r “ 1, . . . , n, s “ 1, . . . , mu.

Proof. Use JNF or Schurforms ofA, B+ LemmaII.5.

Now we are ready to consider our preferred special cases of (II.2).

a) AXB “ C:

A “ B^T b Ainvertible ô λ ¨ µ ‰ 0 @λ P ΛpAqandµ P ΛpBq

ô λ ‰ 0^andµ ‰ 0 @λ P ΛpAq^andµ P ΛpBq ô ^bothAandB are nonsingular.

b) continuous-time Sylvester equation AX ` XB “ C, where A P C^nˆn, B P C^mˆm,C, X P C^nˆm:

A “ Imb A ` B<sup>T</sup> b Ininvertible ô λ ` µ ‰ 0 @λ P ΛpAq^andµ P ΛpBq ô ΛpAq X Λp´Bq “ H.

c) continuous-time Lyapunov equationAX `XA^H “ W, whereA, X P C^nˆn, W “ W^H P C^nˆn:

A “ Inb A ` A b Ininvertible ô ΛpAq X Λp´A^Hq “ H.

For example, this is the case whenAis asymptotically stable.

d) discrete-time Lyapunov equationsÑ^exercise.

The following result gives some useful results about the solution structure of Sylvester equations.

Theorem II.7: LetA P C^nˆn, B P C^{nˆn with}ΛpAq Ă C´,ΛpBq Ă C´. Then AX ` XB “ W has a (unique) solution

X “ ´ ż8

0

e^AtW e^Btdt

Proof. Exercise.

From now on

AX ` XA^˚ “ W, W “ W^˚. (II.3)

Definition II.8 (controllability): LetA P C^nˆnandB P C^{nˆm. We say}pA, Bqis controllable if rankrB, AB, . . . A^n´1Bs “ n.

Lemma II.9: The above controllability condition is equivalent to rankrA ´ λI, Bs “ n^{for all}λ P C

ðñ y^˚B ‰ 0 @y ‰ 0 : y^˚A “ y^˚λ pleft. eigenvecs ofq A

Proof. We first prove that rankrA ´ λI, Bs “ n @λ P Cis equivalent to Def- initionII.8. Assuming that rankrA ´ λI, Bs ă nfor aλ P Cthen there exists aw ‰ 0 such thatw^TrA ´ λI, Bs “ 0 which means thatw^TpA ´ λIq “ 0 andw^TB “ 0and that means that w^TrB, AB, . . . A^n´1Bs “ 0which means pA, Bqis not controllable. Assuming pA, Bq is not controllable and therefore rankrB, AB, . . . A^n´1Bs ă nwe define a matrix M contains a basis of the im- age ofrB, AB, . . . A^n´1Bs. Then there is a matrixM˜ such thatT “ rM, ˜M sis invertible and

A “ T˜ ^´1AT “

«

Ar₁₁ Ar₁₂ 0 Ar₂₂ ff

(II.4)

B “ T˜ ^´1B “

„ Br1

0



(II.5)

Letλbe an eigenvalue ofA˜₂₂andw₂₂a left eigenvector. Then w :“

„ 0

˜ w₂₂



T^´1 ‰ 0.

It also holds thatw^TA “ λw^T andw^TB “ 0and therefore rankrA ´ λi, Bs^not full. The proof of the equivalence is basically also done within this proof.

Theorem II.10: Consider Lyapunov equation (II.3) withW “ W^˚ “ ´BB^T ď 0,B P R^nˆm.

a) ForΛpAq Ă C´: pA, Bqcontrollableô Dunique sol.X “ X^˚ą 0^.

b) LetpA, Bqbe controllable and assume thereD unique sol. X “ X^˚ ą 0. ThenΛpAq Ă C´.

Proof. a) If the spectrum ofAis in the left half plane andW “ W^˚ then there exist a unique symmetric solution of the Lyapunov equation. What is left to prove is the equivalence ofpA, Bqbeing controllable and the solution being positive definite. The solution is given by

X “ ż8

0

e^AtBB^Te^A˚tdt

which is positive if and only ifpA, Bqare controllable.

b) Take an eigenvalueλ P ΛpAqand a corresponding left eigenvectory. Then 0 ą ´y^˚BB^Ty “ y^˚AXy ` y<sup>˚</sup>XA<sup>˚</sup>y “ pλ ` ¯λqy^˚Xy

SinceX “ X^˚ ą 0we must have thatλ ` ¯λ “ 2Reλ ă 0and sinceλwas arbitrary thatΛpAq Ă C´

II.2.2 Direct Numerical Solution

We have seen that linear matrix equations are equivalent to linear systems.

Why do we not just apply a linear solver? Consider a (real) Lyapunov equation where we obtain the system matrix A “ I_nb A ` A b In P R^{n2ˆn2. For} computing an LU-factorization of A and a forward/backwards substitution we need approximately ²₃pn²q³ “ ²₃n⁶FLOPS andn⁴memory. This is only feasible for smalln. If n Á 50, then this is already prohibitively expensive (even if we exploit the structure and symmetry).

Therefore, our first goal is to develop a basic algorithm with complexityOpn³q for moderately sized linear matrix equations.

The Bartels-Stewart Algorithm

The idea of this method is the transformation of the matrixAinto Schur form.

The Schur form can be computed in a numerically stable fashion by the QR algorithm and it is the backbone of many dense eigenvalue algorithms (MAT- LABschur).

Consider (II.3) with ΛpAq X Λp´A^Hq “ H ^{and let} Q^HAQ “ T with be the (complex) Schur form ofA.

Premultiplication of (II.3) byQ^H and postmultiplication byQleads to Q^HAXQ ` Q^HXA^TQ “ Q^HW Q ô Q^HAQ Q^HXQ

lo omo on

“: ˜X

`Q^HXQQ^HA^TQ “ Q^HW Q looomooon

“: ˜W

ô T ˜X ` ˜XT^H “ ˜W (II.6) We partition this in the form

„T₁ T₂ 0 T₃

 „ X₁ X₂ X₂^H X₃



`„ X₁ X₂ X₂^H X₃

 „T₁^H 0 T₂^H T₃^H



“

„W˜₁ W˜₂ W˜₂^H W˜₃

 , whereT₁ P Cpn´1qˆpn´1q, T₂P C^n´1, T₃ P C. Thus we get

$

’&

’%

T₁X₁` T2X<sub>2</sub><sup>H</sup>` X1T₁^H ` X2T<sub>2</sub><sup>H</sup> “ ˜W<sub>1</sub>, T<sub>1</sub>X<sub>2</sub>` T2X₃` X2T<sub>3</sub><sup>H</sup> “ ˜W<sub>2</sub>, T<sub>3</sub>X<sub>3</sub>` X3T₃^H “ ˜W₃,

ô

$

’&

’%

T1X1` X1T<sub>1</sub><sup>H</sup> “ ˜W1´ T2X<sub>2</sub><sup>H</sup> ´ X2T<sub>2</sub><sup>H</sup>, pn ´ 1q ˆ pn ´ 1q T1X2` X2T3“ ˜W2´ T2X3, pn ´ 1q ˆ 1

pT3` T3qX3“ ˜W₃. 1 ˆ 1

Algorithm 2 Bartels-Stewart algorithm (complex version) Input: A,W P C^nˆnwithW “ W^H.

Output: X “ X^H solving (II.3).

1: ComputeT “ Q^HAQwith the QR algorithm.

2: ifdiagpT q X diag`

´T^H˘

‰ Hthen

3: STOP (no unique solution)

4: end if

5: SetW :“ Q˜ ^HW Q.

6: Setk :“ n ´ 1.

7: whilek ą 1do

8: Solve (II.7a) with W˜₃ “ ˜W pk ` 1, k ` 1qand T₃ “ T pk ` 1, k ` 1q to obtainXpk ` 1, k ` 1q.

9: Solve (II.7b) withT1 “ T p1 : k, 1 : kq^,T2 “ T p1 : k, k ` 1q<sup>,</sup>W˜2 “ ˜W p1 : k, k ` 1q, andX₃“ Xpk ` 1, k ` 1qto obtainXp1 : k, k ` 1q.

10: SetW “ ˜˜ W p1 : k, 1 : kq ´ T2X₂^H ´ X2T₂^H

11: Setk :“ k ´ 1.

12: end while

13: Solve (II.7c) withT1 “ T p1, 1qandWˆ1“ ˜W p1, 1q.

14: SetX :“ QXQ^H.

Now we get

X₃ “ W˜3

T3` T3

, (II.7a)

whereT₃` T3‰ 0sinceT₃ P ΛpAq R iR.Next we obtain

T1X2` X2T3 “ ˜W2´ T2X3 “: ˆW2, (II.7b) which is a special Sylvester equation that is equivalent to the linear system

pT3In´1` T1qX2 “ ˆW2,

and can easily be solved by backward substitution. Its solution always exists sinceΛpT₁q X ´T3

( “ H. It remains to solve the smaller pn ´ 1q ˆ pn ´ 1q sized ’triangular’ Lyapunov equation

T₁X₁` X1T₁^H “ ˜W₁´ T2X₂^H ´ X2T₂^H “: ˆW₁, (II.7c) which is also solvable sinceΛpT₁q X Λp´T₁^Hq “ HandWˆ₁“ ˆW₁^H.This leads to the complex Bartels-Stewart algorithm, see Algorithm 2. As a convention we use MATLAB notation, i. e., we denote the section of a matrixA P C^nˆn consisting only of the rowsr₁ tor₂ and the columnsc₁ toc₂ byApr₁ : r₂, c₁ : c2q. If for example,r1“ r2, then we shortly writeApr1, c1 : c2q.

Remark: a) In total this algorithm needs approximately 32n³ « 25nloomoon³

Schur

` 3nloomo³on

premult.

` 3nloomo³on

postmult.

`loomon³on

while loop

complex floating point operations.

b) The algorithm uses only numerically backward stable parts and unitary transformations and thus it can be considered backward stable.

c) The method is implemented in the MATLAB routinelyapand in SLICOT in SB03MD(real version only).

d) The version for Sylvester equations works analogously (see exercise).

Major drawback: The algorithm uses complex arithmetic operations even if all data is real. Luckily, it can be reformulated to use real operations only.

Theorem II.11 (real Schur form): For everyA P R^nˆn there exists an orthogo- nal matrixQ P R^{nˆn such that}Ais transformed to real Schur form, i. e.

Q^TAQ “ T “

»

— –

T11 . . . T_1k . .. ...

T_kk fi ffi

fl, (II.8)

where for i “ 1, . . . , k, Tii P R^1ˆ1 (corresponding to a real eigenvalue of A) or T_ii “

” αi βi

´βiαi

ı

P R^2ˆ2 (corresponding to a pair of complex conjugate eigenvaluesαi˘ iβiofA).

Proof. See the course on “Numerical Linear Algebra”.

To this end, we replace the Schur form by the real Schur form (II.8). ThenT3

may be a2 ˆ 2block, i. e.,T₃ ““_t1t2

t3t4

‰. We obtain

„t₁ t₂ t3 t4

 „x₁ x₂ x2 x3



`„x₁ x₂ x2 x3

 „t₁ t₃ t2 t4



“„w₁ w₂ w2 w3

 .

This is equivalent to

$

’&

’%

w₁“ t1x₁` t2x<sub>2</sub>` t1x₁` t2x<sub>2</sub> “ 2pt1x<sub>1</sub>` t2x₂q,

w₂“ t1x₂` t2x<sub>3</sub>` t3x₁` t4x<sub>2</sub> “ t3x<sub>1</sub>` pt1` t4qx2` t2x₃, w₃“ t3x₂` t4x<sub>3</sub>` t3x₂` t4x<sub>3</sub> “ 2pt3x<sub>2</sub>` t4x₃q.

We can write this as a linear system of equations

» –

t₁ t₂ 0 t3 t1` t4 t2

0 t3 t4

fi fl

» –

x₁ x2

x3

fi fl“

» –

w1

2

w2 w3

2

fi fl.

Additionally, one can exploit the fact thatT₃ corresponds to a pair of complex conjugate eigenvaluesλ_1,2 “ a ˘ ibandT₃“„ a b

´b a



which leads to

» –

a b 0

´b 2a b 0 ´b a fi fl

» –

x₁ x2

x₃ fi fl“

» –

w1

2

w2 w3

2

fi fl. Now (II.7b) becomes

T1X2` X2T₃^T “ ˆW2 :“ ˜W2´ T2X3 P R^n´2ˆ2. (II.9) Consider the partitions corresponding to the quasi-triangular structure ofT₁:

X₂ “

»

— –

x1

... xk´1

fi ffi

fl, Wˆ₂ “

»

— –

ˆ w1

... ˆ wk´1

fi ffi fl,

In general we havexi, ˆwiP R^{niˆnk, where}ni, nkP t1, 2uandi “ 1, . . . , k ´ 1. We now computeX₂ block-wise by progressing upwards fromx_k´1 to x₁. It holds

Tjjxj` xjT₃^T “ ˆwj ´

k

ÿ

h“j`1

T_jhx_h“: ˜wj, j “ k ´ 1, k ´ 2, . . . , 1.

For the solution of this Sylvester equation four cases have to be considered:

a) nj “ nk“ 1: We obtain a scalar equation such thatxj “ ˜wj{pTjj` T3q. b) n_j “ 2, nk “ 1: We obtain a linear system inR² with unique solution given

by

pTjj` T3I2qxj “ ˜wj.

c) nj “ 1, nk “ 2: We obtain a linear system inR² with unique solution given by

pTjjI₂` T3q x^T_j “ ˜w^T_j.

d) n_j “ 2, nk “ 2: We obtain a linear system inR⁴ with unique solution given by

ppI2b Tjjq ` pT3b I2qq vecpxjq “ vecp ˜wjq .

Hence, we getX₂ and can set up a Lyap. eqn. forX₁ defined byT₁. Repeat whole process untilT₁ P R^orT₁P R^2ˆ2. Then back-transform the solution.

Remark II.12: The Sylvester equation (II.9) can be solved alternatively by solv- ing a linear system of the form

pT₁²` αT1` βIn´2qX2 “ ˜W2,

whereX2 “ rs, ts, ˜W2“ ry, zs P R^n´2ˆ2andα, β P R(see exercise).

Hammarling’s Method

Now we consider (II.3) with W “ ´BB^T. By Theorem II.10 we know that X “ X^T ą 0, provided thatΛpAq Ă C^´ and the pairpA, Bqis controllable.

Sometimes it is desirable to only compute a factorU of the solution, i. e.,X “ U U^H with some matrixU. Later we will see that many further algorithms such as projection methods for large scale matrix equations proceed with factors rather than Gramians themselves.

Assume that we have already computed and applied the Schur decomposition ofA “ Q^HT Q, analogously to (II.6). So our starting point is

T ˜X ` ˜XT^H “ ´ ˜B ˜B^H with X “ Q˜ ^HXQ, B “ Q˜ ^HB. (II.10) SinceX ą 0, we also haveX ą 0˜ by Sylvester’s law of inertia. Our goal is to compute upper triangular Cholesky factorsU˜ ofX “ ˜˜ U ˜U^H.

Partition U “˜

„

@

@

@



“„U₁ u 0 τ



, U₁ P C^n´1ˆn´1, u P C^n´1, 0 ă τ P R.

Hammarlings method computes (similar to B.S.) firstτ (scalar equation), then u(LS of sizen ´ 1), and finallyU1as Cholesky factor or an ´ 1 ˆ n ´ 1Lyap.

equation defined byT₁. As in B.S., repeat this untilT₁ P C, afterwards back- transform U Ð QU. Complexity, stability, real version analog to BS. Details here omitted.

Remark: Iterative methods for small, dense Matrix Equations: There are sev- eral, iterative methods computing sequencesX_k,k ě 0converging to the true solution, i.e., lim

kÑ8X_k “ X. For instance:

• Matrix sign function iteration

• Alternating directions implicit (ADI) iterationùlater for large problems.

II.2.3 Iterative Solutions of Large and Sparse Matrix Equations

AX ` XA^T “ ´BB^T, (II.11)

whereA P R^nˆnandnis ’large’, butAis sparse, i. e., only a few entries inA are non-zero. Therefore, multiplication withAcan be performed inOpnqrather thanOpn²qFLOPS. Also solves withAorA ` pIcan be performed efficiently.

However,X P R^nˆn is usually dense and thusX cannot be stored for largen since we would needOpn²qmemory.

Thus the question arises whether it is possible to store the solutionX more efficiently.

The Low-Rank Phenomenon

In practice we often haveB P R^{nˆm, where}m ! n, i. e., the right-hand side BB^T has a low rank. Recall that if pA, Bqis controllable thenX “ X^T ě 0 andrankpXq “ n.

It is a very common observation in practice that the eigenvalues ofX solving (II.11) decay very rapidly towards zero, and fall early below the machine preci- sion.

This gives the concept of the numerical rank ofX:

rankpX, τ q “ argminj“1,...,rankpXqtσjpXq ě τ u, e.g., τ “ _machσ₁pXq.

Can we also theoretically explain this eigenvalue decay?

Theorem II.13: LetA be diagonalizable, i. e., there exists an invertible matrix V P C^{nˆn such that}A “ V ΛV^´1. Then the eigenvalues ofX solving (II.11) withB P C^nˆmsatisfy

λ_km`1pXq

λ₁pXq ď }V }²₂›

›V^´1›

›

2

2ρpM_kq² for any choice of shift parameterspkused to construct

M_k “

k

ź

i“1

pA ´ pkIqpA ` p_kIq^´1

(in particular, the optimal ones).

In the Theorem above the spectral radiusρof a matrix is used:

ρpAq “ max

1ďiďn|λipAq|.

Remark II.14: • If the eigenvalues ofAcluster in the complex plane, only a fewp_kin the clusters suffice to get a smallρpM_kqand thusλ_ipXqdecay fast.

• IfAis normal, then}V }₂›

›V^´1›

›2 “ 1and the bound gives a good expla- nation for the decay. The nonnormal case is much harder to understand.

• This bound (and most others) does not precisely incorporate the eigen- vectors ofAas well as the precise influence ofB.

Consequence: If there is a fast decay ofλipXq, thenX can be well approxi- mated asX “ X^T « ZZ^{H, where}Z P C^{nˆr with}r ! nis a low-rank solution factor. Hence, onlynr memory is required. Thus, in the next subsection we consider algorithms for computing the factorZ without explicitly formingX.

Projection Methods

Now we consider projection-based methods for the solution of large and sparse Lyapunov equations

The main idea consists of representing the solution X by an approximation extracted from a low-dimensional subspaceQ_k “ im Qk withQ^T_kQ_k “ Ikm, i. e.,X « X_k“ QkY_kQ^T_k for someY_kP R^mkˆmk. Impose a Galerkin condition

RpX_kq :“ AXk` XkA<sup>T</sup> ` BB^T K Zk, where

Z_k:“

!

Q_kZQ^T_k P R^nˆn ˇ ˇ ˇ Q

TkQ_k“ Imk, im Q_k“ Qk, Z P R^kmˆkm ) and orthogonality is with respect to the trace inner product. Equivalently,Y_k solves the small-scale Lyapunov equation

H_kY_k` YkH<sub>k</sub><sup>T</sup> ` Q^T_kBB^TQ_k“ 0, H_k:“ Q^T_kAQ_k, (II.12) which can be solved by the Bartels-Stewart or Hammerling’s method.

In case that the residual norm}RpXkq}is not small enough, we increase the dimension ofQ_k by a clever expansion (orthogonally expand Q_k), otherwise we prolongate to obtainXk“ QkYkQ^T_k (never formed explicitly).

What are good choices forQ_k?

a) Standard block Krylov subspaces

Q_k “ KkpA, Bq :“ spantB, AB, . . . , A^k´1Bu :

A matrixQ_kwith orthonormal columns spanningQ_kcan be generated by a block Arnoldi process, i. e. in thekth iteration we haveQ_k ““V₁ . . . V_k‰ fulfilling (assuming there is no breakdown in the process)

AQ_k“ QkH_k` Vk`1H_k`1,kE_k^T, where

Hk“

»

—

—

—

—

—

—

— –

H₁₁ H₁₂ . . . H_1k H₂₁ H₂₂ . . . ^...

0 H32 H33 . . . ^... ... . .. . .. . .. ... 0 . . . 0 Hk,k´1 Hkk

fi ffi ffi ffi ffi ffi ffi ffi fl

is a block upper Hessenberg matrix andE_kis a matrix of the lastmcolumns ofI_km, and

Hk“ Q^T_kAQk.

The residual norm computation for this method is cheap as shown by the following theorem.

Theorem II.15: Suppose that k steps of the block Arnoldi process have been taken. Assume thatΛpH_kq X Λp´Hkq “ H. Then the following state- ments are satisfied:

a) It holds Q^T_kRpQ_kY Q^T_kqQk “ 0 if and only ifY “ Y_k, where Y_k solves the Lyapunov equation (II.12).

b) The residual norm is given by

›

›RpQkYkQ^T_kq›

›F “? 2›

›Hk`1,kE_k^TYk

›

›F.

Proof. Exercise.

Unfortunately, this method often converges only slowly. Therefore, one often chooses modified Krylov subspaces as follows.

b) Extended block Krylov subspaces

EK_qpA, Bq :“ KqpA, Bq Y KqpA^´1, A^´1Bq :

The resulting method is also known as EKSM (extended Krylov subspace method) or KPIK (Krylov plus inverted Krylov). We obtain a similar construc- tion formula as for the block Arnoldi method above and also the residual norm formula is similar. However, the approximation quality is often signif- icantly better than with K_qpA, Bqonly. On the other hand, the subspace dimension grows by2min each iteration step (untilnis reached).

c) Rational Krylov subspaces

RK_qpA, B, Sq (II.13)

:“spantps1In´ Aq^´1B, ps2In´ Aq^´1B, . . . , psqIn´ Aq^´1Bu, S “ts1, . . . , squ Ă C^`, si ‰ sj, i ‰ j : (shifts)

This choice often gives an even better approximation quality compared to EK_qpA, Bq, provided that good shifts S are known. Generating the basis requires solving LSpsiI ´ Aqv “ q_i, but this is usually efficiently possible (cf. Intro).

The shifts si are crucial for a fast convergence, but finding good ones is difficult. For one possible shift selection approach, let m “ 1. One can show

}Rk} „ max |ψkpzq| ^with ψ_kpzq “

k

ź

j“1

z ´ λ_j

z ` s_j, λ_j P ΛpHkq.

This leads to the following procedure for getting the next shift s_k`1 “ argmax_zPBD|ψkpzq|,

where BD is a discrete set of point taken from the convex hull of ΛpHkq (BD Ă convpΛpHkqq^).

For all choices of subspace a)-c): Is the reduced Lyapunov equation (II.12) always uniquely solvable?

For general matricesAthe answer is no. However, for strictly dissipative matri- ces, i. e., matricesAwithA ` A^T ă 0we have the following result.

Theorem II.16: Let A P R^nˆn be strictly dissipative and Qk P R^{nˆmk with} Q^T_kQ_k “ Ikm. ThenΛpH_kq Ă C^´and the reduced Lyapunov equation (II.12) is always uniquely solvable.

Proof. Since A ` A<sup>T</sup> is symmetric and negative definite, it holds x<sup>H</sup>`A ` A^T˘x ă 0for allx P Cⁿ. Then we have

z^H`H<sub>k</sub>` H_k^T˘z “ z^HpQ^T_kAQ_k` Q^T_kA^TQ_kqz

“ y^HpA ` A<sup>T</sup>qy ă 0, y :“ Qkz, @ z P C<sup>km</sup> ñ H<sub>k</sub>` H_k^T ă 0

Now letH_kx “ ˆˆ λˆxforx P Cˆ ^kmzt0u. Then we have ˆ

x^H`H<sub>k</sub>` H_k^T˘ ˆx “ ˆλˆx^Hx ` ˆˆ λˆx^Hx “ 2 Reˆ

´λˆ

¯ ˆ

x^Hx ă 0.ˆ

ThusΛpHkq Ă C^´and the reduced Lyapunov equation (II.12) is uniquely solv- able.

Low-rank ADI

Consider the discrete-time Lyapunov equations

X “ AXA^T ` W, A P R^nˆn, W “ W^T P R^nˆn. (II.14) The existence of a unique solution is ensured if|λ| ă 1for all λ P ΛpAq(see exercise). This motivates the basic iteration

X_k“ AXk´1A^T ` W, k ě 1, X0 P R^nˆn. (II.15) LetAbe diagonalizable, i.e., there exists a nonsingular matrixV P C^{nˆn such} thatA “ V ΛV^´1. LetρpAq :“ max_λPΛpAq|λ|denote the spectral radius ofA. Since

}Xk´ X}2 “ }ApXk´1´ XqA^T}2 “ . . . “ }A^kpX0´ XqpA^Tq^k}2

ď }A^k}²₂}X0´ X}2ď }V }²₂}V^´1}²₂ρpAq^2k}X0´ X}2, (II.16) this iteration converges becauseρpAq ă 1(fixed point argumentation).

For continuous-time Lyapunov equations, recall the result from the exercise:

Lemma II.17: The continuous-times Lyapunov equation AX ` XA^T “ W, ΛpAq Ă C^´ is equivalent to the discrete-time Lyapunov equation

X “CppqXCppq^H` ˜W ppq, Cppq :“ pA ´ pInqpA ` pInq^´1, W ppq :“ ´ 2 Reppq pA ` pI˜ nq<sup>´1</sup>W pA ` pInq^´H

(II.17)

forp P C^´.

Proof. Exercise.

We callCppqa Cayley transformations ofAwhich is the rational function φ_ppzq “ z ´ p

z ` p.

applied toA. Forz, p P C´ we have|φppzq| ă 1. It can be easily shown that (special case of spectral mapping theorem)

ΛpCppqq “ tφ_ppλq, λ P ΛpAqu

and thereforeρpCppqq ă 1. Applying (II.15) to (II.17) gives the Smith iteration X_k“ CppqXk´1Cppq^H ` ˜W ppq, k ě 1, X0 P R^nˆn. (II.18)

Similarly as in (II.16), we have

}Xk´ X}2ď }V }²₂}V^´1}²₂ρpCppqq^2k}X0´ X}2.

This means that we obtain fast convergence by choosingpsuch thatρpCppqq ă 1is as small as possible. We will discuss this later in more detail.

By varying the shifts pin (II.18) in every step, we obtain the ADI iteration for Lyapunov equations

X_k“CppkqXk´1Cpp_kq^H ` ˜W pp_kq, k ě 1, X₀ P R^nˆn, p_kP C^´. (II.19) Remark: The name alternating directions implicit comes from a different (his- torical) derivation of ADI for linear systems. To get the main idea for Lyapunov equations, consider the splitting of the Lyapunov operator

LpXq “ AX ` XA<sup>T</sup> “ L1pXq ` L2pXq, L₁pXq “ AX, L2pXq “ XA^T. Obviously, L₁p¨q ^and L₂p¨q are commuting linear operators. It is possible to formulate an iteration working alternately onL₁p¨qandL₂p¨q, carrying out “half”- iteration steps for each operator:

pA ` piInqX_i´¹

2 “ ´Xi´1pA^T ´ piInq ` W, pA ` piI_nqX_i^T “ ´X_i´^T 1

2

pA^T ´ piI_nq ` W.

Rewriting this into a single step leads to (II.19).

We address two issues of the ADI iteration:

1. ADI requires, similar to the rational Krylov projection method, shift pa- rameters that are crucial for a fast convergence. How to choose the shift parameterspi,i ě 1?

2. The iteration (II.19) is in its given form not feasible for large Lyapunov equations.

The ADI Shift Parameter Problem One can show, similarly to (II.16), that

}Xk´ X}2ď }V }²₂›

›V^´1›

›

2

2ρpMkq²}X0´ X}2, Mk:“

k

ź

i“1

Cppiq, ^(II.20) whereV is a transformation matrix diagonalizingA(assuming it is diagonaliz- able). The eigenvalues of the product of the Cayley transformationsM_kare

ΛpMkq “

# _k ź

i“1

λ ´ pi

λ ` pi

ˇ ˇ ˇ ˇ ˇ

λ P ΛpAq +

.

Good shifts p^˚₁, . . . , p^˚_k should makeρpM_kq ă 1 as small as possible. This motivates the ADI shift parameter problem

rp^˚₁, . . . , p^˚_ks “ argmin_piPC^´ max

λPΛpAq

ˇ ˇ ˇ ˇ ˇ

k

ź

i“1

λ ´ pi

λ ` pi

ˇ ˇ ˇ ˇ ˇ

. (II.21)

In general, this is very hard to solve. For instance, in general,ρpCppqq is not differentiable and the problem is very expensive, ifAis a large matrix. However, there are some procedures that work well in practice:

• Wachspress shifts: Embed ΛpAqin an elliptic function region that de- pends on the parameters

max

λPΛpAqRepλq , min

λPΛpAqRepλq , arctan max

λPΛpAq

ˇ ˇ ˇ ˇ

Impλq Repλq ˇ ˇ ˇ ˇ

(or approximations thereof). Then, (II.21) can be solved by employing an elliptic integral.

• Heuristic Penzl shifts: If A is a large and sparse matrix, ΛpAq is re- placed by a small number of approximate eigenvalues (e.g., Ritz values).

Then (II.21) is solved heuristically.

• Self-generating shifts: IfAis large and sparse, these shifts are based on projections ofAwith the data obtained by previous iterations. These shifts also make use of the right-hand sideW.

The Low-Rank ADI For a low-rank version of ADI computing low-rank solu- tion factors, consider one step of the dense iteration (II.19) and insert Xj “ ZjZ_j^H:

X_j “ CppjqXj´1Cpp_jq^H ` ˜W pp_jq

“ pA ´ p_jI_nqpA ` pjI<sub>n</sub>q<sup>´1</sup>Z<sub>j´1</sub>Z<sub>j´1</sub><sup>H</sup> pA ` pjI_nq^´HpA ´ p_jI_nq^H

´ 2 Reppjq pA ` pjInq<sup>´1</sup>BB<sup>T</sup>pA ` pjInq^´H. ñ Xj “ ZjZ_j^H, Zj ““a

´2 ReppjqpA ` pjInq<sup>´1</sup>B pA ´ p<sub>j</sub>InqpA ` pjInq^´1Zj´1‰ . With Z0 “ 0 we find a low rank variant the ADI iteration (II.19) forming Zj

successively (grows bymcolumns in each step).

The drawback is that all columns are processed in every step which leads to quickly growing costs (in totaljmlinear systems have to be solved to getZ_j).

However, there is a remedy to this problem. Obviously, Si “ pA ` piInq^´1andTj “ pA ´ p_jInq

commute for alli, jwith each other and themselves (proof it yourself).

Now considerZ_j being the iterate after iteration stepj

Z_j ““α_jS_jB pTjS_jqαj´1S_j´1B . . . pTjS_jq ¨ ¨ ¨ pT2S₂qα1S₁B‰ withα_i “ a

´2 Reppiq. The order of application of the shifts is not important, and we reverse their application to obtain the following alternative iterate

Z˜_j ““α₁S₁B α₂pT1S₁qS2B . . . α_jpT1S₁q ¨ ¨ ¨ pTj´1S_j´1qSjB‰

““α₁S₁B α₂pT1S₂qS1B . . . α_jpTj´1S_jqpTj´2S_j´1q ¨ ¨ ¨ pT1S₂qS1B‰

““α₁V1 α2V2 . . . αjVj‰ ,

V1“ S1B, Vi “ Ti´1SiVi´1, i “ 1, . . . , j.

We haveX_j “ ˜Z_jZ˜_j^H, but in this formulation only the new columns are pro- cessed. Even more structure is revealed by the Lyapunov residual.

Theorem II.18: The residual at stepjof (II.19), started withX₀ “ 0, is of rank at mostmand given by

R_j :“AZ_jZ_j^H ` ZjZ<sub>j</sub><sup>H</sup>A<sup>T</sup> ` BB^T “ WjW_j^H,

Wj “MjB “ CppjqWj´1 “ Wj´1´ 2 Reppjq Vj, W0 :“ B, whereMj :“ś_j

i“1Cppiq. Moreover, it holdsVj “ pA ` pjInq^´1Wj´1.

Proof. We have

R_j “ AXj` XjA<sup>T</sup> ` BB^T “ ApXj´ Xq ` pXj´ XqA^T p^{by (II.11)}q

“ AMjpX0´ XqM_j^H` MjpX0´ XqM_j^HA^T

“ ´MjAXM_j^H ´ MjXA^TM_j^H

“ ´MjpAX ` XA^TqMj “ MjBB^TMj. Moreover, it holds

Vj “ Tj´1SjVj´1“ Tj´1SjTj´2Sj´1Vj´2 “ . . . “

“ Sj

˜_j´1 ź

k“1

T_kS_k

¸

B “ SjMj´1B “ pA ` pjInq^´1Wj´1, (II.22)

and

W_j “ MjB “ S_jT_jW_j´1“ Wj´1´ 2 Reppjq SjW_j´1“ Wj´1´ 2 Reppjq Vj.

Algorithm 3 Low-rank ADI (LR-ADI) iteration for Lyapunov equations

Input: A, B from (II.11), shifts P “ tp₁, . . . , p_maxiteru Ă C^´, residual toler- ancetol.

Output: Z_ksuch thatX “ Z_kZ_k^H (approx.) solves (II.11).

1: Initializej “ 1,W₀ :“ B,Z₀:“ r s.

2: while}Wj´1}2 ě toldo

3: SetVj :“ pA ` pjInq^´1Wj´1.

4: SetW_j :“ W_j´1´ 2 Reppjq Vj.

5: SetZj :““ Zj´1

a´ ReppjqVj

‰.

6: Setj :“ j ` 1.

7: end while

Thank to the above theorem, the norm of the Lyapunov residual norm can be cheaply computed via}Rj}2 “ }WjW_j^H}2“ }Wj}²₂. All this leads to Algorithm 3. Again, the major work is solving the LSpA ` pjI_nqVj “ Wj´1in each step, which is efficiently possible for large,sparseA(cf. Introduction).

Algorithm3produces complex low-rank factors, if some of the shifts are com- plex, which might be required for problems with nonsymmetricA. Ensuring that Zj P R^nˆnjand limiting the number of complex operations can be achieved by assuming that for a complex shiftpiwe havepi`1“ pi(see handout)