and Y of x∗and y∗and a continuous function φ : Y → X , such that
f(x, y) = 0 ⇐⇒ x= φ (y), for all x∈ X, y ∈ Y. Furthermore, if f is Cp-smooth for p∈ N, then φ is Cp-smooth with gradient
∇φ (y) = −(∇xf(φ (y), y))−1∇yf(φ (y), y).
2.3
Linear algebra
For a matrix A ∈ Cn,n, denote by ai∈ Rnits ith row, AH its Hermitian and A∗its adjoint. A matrix is said to be Hermitian (resp. self-adjoint) if AH = A (resp. A∗= A).
We denote by ker A the kernel of A, i.e. the subspace of vectors x such that Ax = 0. Consider a Hermitian matrix A ∈ Cn,n. We say that A is positive-definite if
⟨x, Ax⟩ > 0 for all x ∈ Cn,
and positive-semidefinite if
⟨x, Ax⟩ ≥ 0 for all x ∈ Cn.
A positive-definite matrix is always nonsingular, i.e. it admits an inverse A−1∈ Cn,n, which is also positive-definite.
We denote by Inthe identity matrix in Rn,nand by 0nthe zero matrix. When the dimension
is unambiguous, we occasionally write I instead.
The (operator) norm of a matrix A ∈ Rm,n is defined as ∥A∥ := sup
x∈Sn−1
∥Ax∥,
while the Frobenius norm is defined as
∥A∥F := s n
∑
i, j=1 a2i, j, where ai, j= aij.The rank of a matrix A is the dimension of the column space (or equivalently row space) of the matrix, i.e. the number of linearly independent column vectors, and is denoted by rank(A).
Definition 2.4 (Spectrum). The spectrum of a square matrix A ∈ Cn,n is its set of eigenvalues,
σ (A) := {λ ∈ C : ∃x ∈ Cn\ {0} with λ x = Ax}.
For a positive-definite matrix A, its condition number is the ratio κA := λn/λ1≥ 1,
where λn and λ1 are respectively the matrix’ largest and smallest eigenvalues. A large
condition number means the matrix is ill-conditioned while a low condition number means it is well-conditioned.
Definition 2.5 (Spectral radius). The spectral radius of a square matrix A ∈ Cn,n is
ρ (A) := sup
λ ∈σ (A)
|λ |.
Gelfand’s formula denotes an important relationship between A and its spectral radius. Proposition 2.6 (Gelfand’s formula [124, Theorem 7.5.5]). For any square matrix A ∈ Cn,n, the following limit holds:
lim
k→∞ k
q
∥Ak∥ = ρ(A).
The following result is a variation on [179, Chapter 2, Theorem 1], a key ingredient for showing convergence of various iterative schemes, based on Gelfand’s formula.
Proposition 2.7. Let (Ak)k∈N⊂ Cn,n,(bk)k∈N∈ Cn, and for d0∈ Cn, define the iterates dk+1= Akdk+ bk, k∈ N.
If Ak→ A and bk→ b with ρ(A) < 1, then the iterates (dk)
k∈Nconverge linearly to the fixed
point(I − A)−1b.
Proof. Since ρ(A) < 1, it follows from Gelfland’s formula that the Neumann sequence ∑∞i=0Ai is well-defined and equal to (I − A)−1. Therefore, d∗:= (I − A)−1bis the unique
fixed point of the mapping d 7→ Ad + b.
Write yk= dk− d∗, rk= (Ak− A)yk, and sk= (A
k− A)d∗+ bk− b. Then
2.3 Linear algebra 21 Thus yk+1= Ak+1y0+ k
∑
i=1 Ak−iri+ k∑
i=1 Ak−isi,∥yk+1∥ ≤ ∥Ak+1∥∥y0∥ +
k
∑
i=1 ∥Ak−i∥∥ri∥ + k∑
i=1 ∥Ak−i∥∥si∥.By [179, Chapter 2, Lemma 1], for any ρ ∈ (ρ(A), 1) there is c > 0 such that ∥Ak∥ ≤ cρk for all k ∈ N, which implies
∥yk+1∥ ≤ cρk+1∥y0∥ + c k
∑
i=1 ρk−i∥ri∥ + c k∑
i=1 ρk−i∥si∥.Since ∥si∥ → 0, the third term vanishes as k → ∞. Finally, as rk= o(yk), the result follows.
In Chapter 7, we will repeatedly make use of the following result to simplify the analysis. Proposition 2.8. If A and B ∈ Cn,nare self-adjoint matrices, and B is positive-definite, then σ (AB) = σ (BA) ⊂ R.
Proof.By [124, Theorem 9.4.2], there is a self-adjoint, positive-definite square root of B,√B∈ Cn,n, such that√B2= B. One can verify that σ (AB) = σ (√BA√B). It remains to note that√BA√Bis self-adjoint, so by [124, Theorem 9.2.1], σ (√BA√B) ⊂ R.
To show the equality σ (AB) = σ (BA), note that for any square matrix A, the eigenvalues of AH equal the complex conjugates of the eigenvalues of A. Then the equality follows from the fact that (AB)H= BA and that σ (AB) ⊂ R.
Finally, we introduce the (Moore–Penrose) pseudoinverse which can be defined accord- ingly in finite-dimensional spaces.
Definition 2.9 (Moore–Penrose pseudoinverse). Let A ∈ Rm,n be a matrix. The Moore– Penrose pseudoinverse of A, A†∈ Rn,m, is the (unique) matrix which satisfies the following four conditions.
AA†A= A, A†AA†= A†, (AA†)∗= AA†, (A†A)∗= A†A.
For square, invertible matrices A, we have A† = A−1. However, pseudoinverses are also uniquely defined for non-square and singular matrices. For further details on the pseudoinverse, see [83, Section 2.1].