Linear Algebra & Properties of the Covariance Matrix
October 3rd, 2012
Estimation of ¯ r and C
Let rn1, rnt, . . . , rnT be the historical return rates on the nth asset.
rn=
rn1 rn2 ... rnT
n = 1, 2, . . . , N .
Estimation of ¯ r and C
The expected return is approximated by ˆ¯rn = 1
T
T
X
t=1
rnt
and the covariance is approximated by
ˆ
cmn= 1 T − 1
T
X
t=1
(rnt− ˆ¯rn)(rmt − ˆ¯rm) ,
or in matrix/vector notation
C =b 1 T − 1
T
X
t=1
(rt− ˆ¯r )(rt− ˆ¯r )> an outer product.
Importance of C
The expected return ¯r is often estimated exogenously (e.g. not statistically estimated). But
statistical estimation of C is an important topic, is used in practice,
and is the backbone of many finance strategies.
A major part of Markowitz theory was the assumption for C to be a covariance matrix it must be symmetric positive definite (SPD).
Real Matrices/Vectors
Let’s focus on real matrices, and only use complex numbers if necessary. A vector x ∈ RN and a matrix A ∈ RM×N are
x =
x1
x2 ... xN
and A =
a11 a12 . . . a1N a21 a22 . . . a2N ... . .. ... aM1 aM2 . . . aMN
Matrix/Vector Multiplication
The matrix A times a vector x is a new vector
Ax =
a11 a12 . . . a1N a21 a22 . . . a2N ... . .. ... aM1 aM2 . . . aMN
x1
x2 ... xN
=
a11x1+ a12x2+ · · · + a1NxN
a21x1+ a22x2+ · · · + a2NxN ...
aM1x1+ aM2x2+ · · · + aMNxN
∈ RM .
Matrix/matrix multiplication is an extension of this.
Matrix/Vector transpose
Their transposes are
x>=
x1 x2
... xN
>
= (x1, x2, . . . , xN)
and A>=
a11 a12 . . . a1N a21 a22 . . . a2N
... . .. ... aM1 aM2 . . . aMN
>
=
a11 a21 . . . aM1 a12 a22 . . . aM2
... . .. ... a1N a2N . . . aMN
.
Note that (Ax )>= x>A>.
Vector Inner/outer Product
The vector x ∈ RN inner product with another vector y ∈ RN is x>y = y>x = x1y1+ x2y2+ . . . xNyn .
There is also the outer product,
xy>=
x1y1 x1y2 . . . x1yN x2y1 x2y2 . . . x2yN ... . .. ... xNy1 xNy2 . . . XNYN
6= yx> .
Matrix/Vector Norm
The norm on the vector space RN is k · k, and for any x ∈ RN we have
kxk =√ x>x . Properties of the norm
kxk ≥ 0 for any x, kxk = 0 iff x = 0,
kx + y k ≤ kxk + ky k (triangle inequality),
|x>y | ≤ kx kky k with equality iff x = ay for some a ∈ R (Cauchy-Schwartz),
kx + y k2 = kx k2+ ky k2 if x>y = 0 (Pythagorean thm).
Linear Independence & Orthogonality
Two vectors x , y are linear independent if there is no scalar constant a ∈ R s.t.
y = ax . These two vectors are orthogonal if
x>y = 0 . Clearly, orthogonality ⇒ linear independent.
Span
A set of vectors x1, x2, . . . , xn is said to span a subspace V ⊂ RN if for any y ∈ V there are constants a1, a2, . . . , an such
y = a1x2+ a2x2+ · · · + anxn .
We write, span(x1, . . . , xn) = V . In particular, a set x1, x2, . . . , xN
will span RN iff they are linearly independent:
span(x1, x2, . . . , xN) = RN ⇔ xi linearly independent of xj for all i , j . If so, then x1, x2, . . . , XN are a basis for RN.
Orthogonal Basis
Definition
A basis of RN is orthogonal if all its elements are orthogonal, span(x1, x2, . . . , xN) = RN and xi>xj = 0 ∀i , j .
Invertibility
Definition
A square matrix A ∈ RN×N is invertible if for any b ∈ RN there exists x s.t.
Ax = b . If so, then x = A−1b.
A is invertible if any of the following hold:
A has rows that are linearly independent A has columns that are linearly independent det(A) 6= 0
there is no non-zero vector x s.t. Ax = 0.
In fact, these 4 statements are equivalent.
Eigenvalues
Let A be an N × N matrix. A scalar λ ∈ C is an eigenvalue of A if
Ax = λx for some vector x = re(x ) +√
−1 im(x) ∈ CN. If so x is an eigenvector.
By 4th statement of previous slide, A−1 exists ⇔ zero is not an eigenvalue.
Eigenvalue Diagonalization
Let A be an N × N matrix with N linearly independent
eigenvectors. There is an N × N matrix X and an N × N diagonal matrix Λ such that
A = X ΛX−1 ,
where Λii is an eigenvalue of A, and the columns of X = [x1, x2, . . . , xN] are eigenvectors,
Axi = Λiixi .
Real Symmetric Matrices
A matrix is symmetric if A>= A.
Proposition
A symmetric matrix has real eigenvalues.
pf: If Ax = λx then
λ2kxk2= λ2x∗x = x∗A2x = x∗A>Ax = kAx k2, which is real and nonnegative (x∗ = re(x )>−√
−1 im(x)> i.e the conjugate transpose). Hence, λ is cannot be imaginary or complex.
Real Symmetric Matrices
Proposition
Let A = A> be a real matrix. A’s eigenvectors form an orthogonal basis of RN, and the diagonalization is simplified:
A = X ΛX> , i.e. X−1 = X>.
Basic Idea of Proof: Suppose that Ax = λx and Ay = αy with α 6= λ. Then
λx>y = x>A>y = x>Ay = αx>y , and so it must be that x>y = 0.
Skew Symmetric Matrices
A matrix is skew symmetric if A>= −A.
Proposition
A skew symmetric matrix has purely imaginary eigenvalues.
pf: If Ax = λx then
λ2kxk2= λ2x∗x = x∗A2x = −x∗A>Ax = −kAx k2 which is less than zero. Hence, λ2 < 0 must be purely imaginary.
Positive Definitness
Definition
A matrix A ∈ RN×N is positive definite iff x>Ax > 0 ∀x ∈ RN . It is only positive semi-definite iff
x>Ax ≥ 0 ∀x ∈ RN .
Positive Definitness ⇒ Symmetric
Proposition
If A ∈ RN×N is positive definite, then A>= A, i.e. it is symmetric positive definite (SPD).
pf: For any x ∈ RN we have
0 < x>Ax = 1 2
x>(A + A>
| {z } sym.
)x + x>( A − A>
| {z } skew-sym.
)x
,
but if A − A>6= 0 then there exists eigenvector x s.t.
(A − A>)x = αx where α ∈ C \ R, which contradicts the inequality.
Eigenvalue of an SPD Matrix
Proposition
If A is SPD, then it has all positive eigenvalues.
pf: By definition for any y ∈ RN \ {0} it holds that y>Ay > 0 .
In particular, for any λ and x s.t. Ax = λx , we have λkx k2 = x>Ax > 0
and so λ > 0.
Consequence: for A SPD there’s no x s.t. Ax = 0, and so A−1 exists.
So Can/How Do We Invert C ?
The invertibility of the covariance matrix is very important:
we’ve seen a need for it in the efficient frontiers and Markowitz problem,
if it’s not invertible then there are redundant assets, we’ll need it to be invertible when we discuss multivariate Gaussian distributions.
If there is no risk-free, our covariance matrix is practically invertible by definition:
0 < var X
i
wiri
!
=X
i ,j
wiwjCij = w>Cw
where w ∈ RN\ {0} is any allocation vector (w may not sum to
So How Do We Know b C is Covariance Matrix?
Recall, bC = T −11 PT
t=1(rt− ¯r )(rt− ¯r )>. Each summand is symmetric,
(rt− ˆ¯r )(rt− ˆ¯r )> is symmetric ,
and sum of symmetric matrices is symmetric. But each summand is not invertible because there exists a vector y s.t.
y>(rt− ˆ¯r ) = 0, which means that zero is an eigenvalue, (rt− ˆ¯r ) (rt− ˆ¯r )>y
| {z }
=0
= 0 .
So How Do We Know b C is Covariance Matrix?
If
span(r1− ˆ¯r , r2− ˆ¯r , . . . , rT − ˆ¯r ) = RN then bC is invertible. So need T ≥ N.
In order for bC to be close C will need T N.
C is also SPD:b
y>C y = yb > 1 T − 1
T
X
t=1
(rt− ˆ¯r )(rt− ˆ¯r )>
! y
= 1
T − 1
T
X
t=1
y>(rt−ˆ¯r )(rt−ˆ¯r )>y = 1 T − 1
T
X
t=1
(y>(rt−ˆ¯r ))2> 0 ,
A Simple Covariance Matrix
C = (1 − ρ)I + ρU where I is the identity and Uij = 1 for all i , j . U1 = N1
Ux = 0 if x>1 = 0 where1 .
= (1, 1, . . . , 1)>. Hence, for any x we have Cx = (1 − ρ)Ix + ρUx = (1 − ρ)x + ρ(1>x )1 , which means x is an eigenvector iff1>x = 0 or x = a1. The eigenvalues of C are 1 − ρ + Nρ and 1 − ρ, and the eigenvectors are1 and the N − 1 vectors orthogonal to 1, respectively.
Positive Definite (Complex Hermitian Matrices)
Let C be a square matrix, i.e. C ∈ CN×N. C is positive definite if
z∗Cz > 0 ∀z ∈ CN\ {0}
where CN is N-dimensional complex vectors, and z∗ is conjugate transpose,
z∗ = (zr + izi)∗ = zr − izi . C is positive semi-definite if
z∗Cz ≥ 0 ∀z ∈ CN\ {0}
If C positive definite and real, then it must also be
Gershgorin Circles
Proposition
All eigenvalues of the matrix A lie in one of the Gershgorin circles,
|λ − Aii| ≤X
j 6=i
|Aij| for at least 1 i ≤ N,
where λ is an eigenvalue of A.
pf: Let λ be an eigenvalue and let x be an eigenvector. Choose i so that |xi| = maxj|xj|. We have P
jAijxj = λxj . W.L.O.G. we can take |xi| = 1 so thatP
j 6=iAijxj = λ − Aii, and taking absolute values yields the result,
|λ − Aii| =
X
j 6=i
Aijxj
≤X
j 6=i
|Aij| · |xj| ≤X
j 6=i
|Aij| .