SF2940: Probability theory Lecture 8: Multivariate Normal Distribution

SF2940: Probability theory

Lecture 8: Multivariate Normal Distribution

Timo Koski

24.09.2014

Learning outcomes

Random vectors, mean vector, covariance matrix, rules of transformation

Multivariate normal R.V., moment generating functions, characteristic function, rules of transformation

Density of a multivariate normal RV Joint PDF of bivariate normal RVs

Conditional distributions in a multivariate normal distribution

PART 1: Mean vector, Covariance matrix, MGF,

Characteristic function

Vector Notation: Random Vector

A random vector X is a column vector

X =



 

  X

X

.. . X



 

 

= ( X

, X

, . . . , X

)

Each X

is a random variable.

Sample Value Random Vector

A column vector

x =



 

  x

x

.. . x



 

 

= ( x

, x

, . . . , x

)

We can think of x

is an outcome of X

.

Joint CDF, Joint PDF

The joint CDF (=cumulative distribution function) of a continuous random vector X is

F

( x ) = F

( x

, . . . , x

) = P ( X ≤ ^x ) =

= P ( X

≤ ^x

, . . . , X

≤ ^x

) Joint probability density function (PDF)

f

( x ) = ^∂

n

∂x

. . . ∂x

F

( x

, . . . , x

)

Mean Vector

µ

= E [ X ] =



 

 

E [ X

] E [ X

]

.. . E [ X

]



 

  ,

a column vector of means (=expectations) of X.

Matrix, Scalar Product

If X

is the transposed column vector (=a row vector), then XX

is a n × ⁿ matrix, and

X

X =

n

∑

i=1

X

is a scalar product, a real valued R.V..

Covariance Matrix of A Random Vector

Covariance matrix

C

: = E h

( X − ^µ

) ( X − ^µ

)

ⁱ where the element ( i , j )

C

( i, j ) = E [( X

− ^µ

) ( X

− ^µ

)]

is the covariance of X

and X

.

A Quadratic Form

x

C

x =

n

∑

i=1 n

∑

j=1

x

x

C

( i, j ) . We see that

=

n

∑

i=1 n

∑

j=1

x

x

E [( X

− ^µ

) ( X

− ^µ

)]

= E

"

∑

i=1 n

∑

j=1

x

x

( X

− ^µ

) ( X

− ^µ

)

#

(∗)

Properties of a Covariance Matrix

Covariance matrix is nonnegative definite, i.e., for all x we have x

C

x ≥ ⁰

Hence

det C

≥ ^0.

The covariance matrix is symmetric

C

= C

Properties of a Covariance Matrix

The covariance matrix is symmetric C

= C

since

C

( i, j ) = E [( X

− ^µ

) ( X

− ^µ

)]

= E [( X

− ^µ

) ( X

− ^µ

)] = C

( j , i )

Properties of a Covariance Matrix

A covariance matrix is positive definite, x

C

x > ₀ for all x 6= ^{0 iff}

det C

> ₀

(i.e. C

is invertible).

Properties of a Covariance Matrix

Proposition

x

C

x ≥ ⁰ Pf: By (∗) ^above

x

C

x = x

E h

( X − ^µ

) ( X − ^µ

)

ⁱ x

= E h

x

( X − ^µ

) ( X − ^µ

)

x i

= E h

x

w · ^w

^{x i}

where we have set w = ( X − ^µ

) . Then by linear algebra x

w = w

x

= ∑

w

x

. Hence

E h

x

ww

x i

= E





n

∑

=

w

x

!



 ≥ ^0.

Properties of a Covariance Matrix

In terms of the entries c

of a covariance matrix C = ( c

)

there are the following necessary properties.

1

c

= c

(symmetry).

2

c

= Var ( X

) = σ

≥ 0 (the elements in the main diagonal are the variances, and thus all elements in the main diagonal are

nonnegative).

3

c

≤ ^c

· ^c

(Cauchy-Schwartz’ inequality).

Coefficient of Correlation

The Coefficient of Correlation ρ of X and Y is defined as ρ : = ρ

: = ^Cov ( X , Y )

p Var ( X ) · ^Var ( Y ) ^,

where Cov ( X , Y ) = E [( X − ^µ

) ( Y − ^µ

)] . This is normalized

− ¹ ≤ ^ρ

≤ ¹ For random variables X and Y ,

Cov ( X , Y ) = ρ

= 0 does not always mean that X , Y are

independent.

Special case: Covariance Matrix of A Bivariate Vector

X = ( X

, X

)

.

C

=

 σ

ρσ

σ

ρσ

σ

σ

 ,

where ρ is the coefficient of correlation of X

and X

, and σ

= Var ( X

) , σ

= Var ( X

) . C

is invertible iff ρ

6= 1, for proof we note that

det C

= σ

σ

1 − ^ρ

Special case: Covariance Matrix of A Bivariate Vector

Λ =

 σ

ρσ

σ

ρσ

σ

σ

 , if ρ

6= 1, the inverse exists and

Λ

= ¹

σ

σ

( 1 − ^ρ

)

 σ

− ^ρσ

σ

− ^ρσ

^σ

^σ



,

Y = _BX + b

Proposition

X is a random vector with mean vector µ

and covariance matrix C

. B is a m × n matrix. If Y = BX + b, then

E Y = B µ

+ b C

= BC

B

Pf: For simplicity of writing, take b = µ = 0. Then C

= E YY

= EB X ( B X )

=

= EBXX

B

= BE h XX

i

B

= BC

B

Moment Generating and Characteristic Functions

Definition

Moment generating function of X is defined as

ψ

( t )

= Ee

= Ee

Definition

Characteristic function of X is defined as

ϕ

( t )

= Ee

= Ee

Special cases: take t

= 1, t

= t

= . . . = t

= 0, then

ϕ

( t ) = ϕ

( t

) .

PART 2: Def I of a multivariate normal distribution

We recall first some of the properties of univariate normal distribution

Normal (Gaussian) One-dimensional RVs

X is a normal random variable if f

( x ) = ¹

σ √

2π e

where µ is real and σ > _0.

Notation: X ∈ ^N ( µ, σ

)

Properties: E ( X ) = µ, Var = σ

Normal (Gaussian) One-dimensional RVs

−2 0 2 4 6

0 0.2 0.4 0.6 0.8

x

f X(x)

−2 0 2 4 6

0 0.2 0.4 0.6 0.8

x

fX(x)

(a)

µ = 2, σ = 1/2 , (b) µ = 2, σ = 2

Linear Transformation

X ∈ ^N ( µ

, σ

) ⇒ ^Y = aX + b is N ( aµ

+ b, a

σ

) Thus Z =

X

∈ ^N ( 0, 1 ) and

P ( X ≤ ^x ) = P  X − ^µ

σ

≤ ^x − ^µ

σ



or

F

( x ) = P



Z ≤ ^x − ^µ

σ



= ^{Φ  x} − ^µ

σ



Normal (Gaussian) One-dimensional RVs

X ∈ ^N ( µ, σ

) then the moment generating function is ψ

( t ) = E h

e

i

= e

, and the characteristic function is

ϕ

( t ) = E h e

i

= e

as found in previous Lectures.

Multivariate Normal Def. I

Definition

An n × 1 random vector X has a normal distribution iff for every n × 1-vector a the one-dimensional random vector a

X has a normal distribution.

We write X ∈ ^N ( µ, Λ ) , when µ is the mean vector and Λ is the

covariance matrix.

Consequences of Def. I (1)

An n × ^{1 vector X} ∈ ^N ( µ, Λ ) iff the one-dimensional random vector a

X has a normal distribution for every n-vector a .

Now we know that (take B = a

in the preceding) Ea

X = a

µ, Var h

a

X i

= a

Λa

Consequences of Def. I (2)

Hence, if Y = a

X, then Y ∈ ^{N a}

^{µ, a}

^Λa
and the moment generating function of Y is

ψ

( t ) = E h e

i

= e

. Therefore

ψ

( a ) = Ee

= ψ

( 1 ) = e

.

Consequences of Def. I (3)

Hence we have shown that if X ∈ ^N ( µ, Λ ) , then

ψ

( t ) = Ee

= e

.

is the moment generating function of X.

Consequences of Def. I (4)

In the same way we can find that

ϕ

( t ) = Ee

= e

.

is the characteristic function of X ∈ ^N ( µ, Λ ) .

Consequences of Def. I (5)

Let Λ be a diagonal covariance matrix with λ

s on the main diagonal, i.e.,

Λ =



 

 

 

λ

0 0 . . . 0 0 λ

0 . . . 0 0 0 λ

. . . 0 0 . .. ... . . . 0 0 0 0 . . . λ



 

 

  ,

Proposition

If X ∈ ^N ( µ, Λ ) , then X

, X

, . . . , X

are independent normal variables.

Consequences of Def. I (6)

Pf: Λ is diagonal, the quadratic form becomes a single sum of squares.

ϕ

( t ) = e

=

= e

= e

e

· · · ^e

is the product of the characteristic functions of X

∈ ^{N µ}

^{, λ}

, which are thus seen to be independent N µ

, λ

.

Kac’s theorem: Thm 8.1.3. in LN

Theorem

X = ( X

, X

, · · · ^{, X}

)

. The components X

, X

, · · · ^{, X}

^are independent if and only if

φ

( s ) = ^{E h e}

ⁱ =

n

∏

i=1

φ

( s

) ,

where φ

( s

) is the characteristic function for X

.

Further properties of the multivariate normal

X ∈ ^N ( µ, Λ )

Every component X

is one-dimensional normal. To prove this we take a = ( 0, 0, . . . , 1

|{z}

position k

, 0, . . . , 0 )

and the conclusion follows by Def. I.

X

+ X

+ · · · ^X

is one-dimensional normal. Note: The terms in the

sum need not be independent.

Properties of multivariate normal

X ∈ ^N ( µ, Λ )

Every marginal distribution of k variables ( 1 ≤ ^k < _n is normal. To

prove this we consider any k variables X

, X

. . . X

and then take a

such that a

= 0 for j 6= ⁱ

^{, . . . i}

and then apply Def. I.

Properties of multivariate normal

Proposition

X ∈ ^N ( µ, Λ ) and Y = BX + b. Then

Y ∈ ^{N  B µ} + b, BΛB

 . Pf:

ψ

( s ) = E h e

i

= E h

e

i

=

= e

E h e

i

= e

E



e (

)

^

E



e (

)

^ = ψ

 B

s 

.

Properties of multivariate normal

X ∈ ^N ( µ, Λ ) ψ

 B

s 

= e (

)

(

)

(

) _.

 B

s 

µ = s

B µ,

 B

s 

Λ  B

s 

= s

BΛB

s,

e (

)

(

)

(

) = e

Properties of multivariate normal

ψ

 B

s 

= e

.

ψ

( s ) = e

ψ

 B

s 

= e

e

ψ

( s ) = e

,

which proves the claim as asserted.

PART 3: Multivariate normal, Def. II: characteristic

function, DEF III: density

Multivariate normal, Def. II: char. fnctn

Definition

A random vector X with mean vector µ and a covariance matrix Λ is N ( µ, Λ ) if its characteristic function is

ϕ

( t ) = ^Ee

= e

.

Multivariate normal, Def. II implies Def. I

We need to show that the one-dimensional random vector Y = a

X has a normal distribution.

ϕ

( t ) = E h e

i

= E h

e

i

=

= E h e

i

= ϕ

( ta ) =

= e

and this is the characteristic function of N a

µ, a

Λa

.

Multivariate normal, Def. III: joint PDF

Definition

A random vector X with mean vector µ and an invertible covariance matrix Λ is N ( µ, Λ ) , if the density is

f

( x ) = ¹

( 2π )

^p det ( Λ ) ^e

−¹₂(x−µ)^TΛ⁻¹(x−µ)

Multivariate normal

It can be checked by a computation that e

=

Z

Rⁿ

e

1

( 2π )

^p det ( Λ ) ^e

−¹₂(x−µ)^TΛ⁻¹(x−µ)

d x

(complete the square) Hence Def. III implies the property in Def. II. The

three definitions are equivalent, in the case inverse of the covariance

matrix exists.

PART 4: Bivariate normal with density

Multivariate Normal: the bivariate case

As soon as ρ

6= 1, the matrix Λ =

 σ

ρσ

σ

ρσ

σ

σ

 , is invertible, and the inverse is

Λ

= ¹

σ

σ

( 1 − ^ρ

)

 σ

− ^ρσ

^σ

− ^ρσ

σ

σ



,

Multivariate Normal: the bivariate case

ρ

6= ^{1, and X} = ( X

, X

)

, then f

( x ) = ¹

2π √

det Λ e

= ¹

2πσ

σ

p 1 − ^ρ

^e

−¹₂^Q(x1,x2)

Multivariate Normal: the bivariate case

where

Q ( x

, x

) = 1

( 1 − ^ρ

) ·

" x

− ^µ

σ



− ^2ρ ( x

− ^µ

)( x

− ^µ

)

σ

σ

+ ^{ x}

− ^µ

σ



#

For this, invert the matrix Λ and expand the quadratic form !

ρ = 0

0 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0

0

3 2 1 0 -1 -2 -3

0

3 2

1 0

-1 -2

-3

ρ = 0.9

0 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0

0

3 2 1 0 -1 -2 -3

0

3 2

1 0

-1 -2

-3

ρ = − ^0.9

0 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0

0

3 2 1 0 -1 -2 -3

0

3 2

1 0

-1 -2

-3

Conditional densities for the bivariate normal

Complete the square of the exponent to write f

( x, y ) = f

( x ) f

( y ) where

f

( x ) = ¹ σ

√ 2π e

1 2σ21

(x−^µ1)²

f

( y ) = ¹

˜σ

√ 2π e

1 2 ˜σ22

(y−^˜µ2(x))²

˜

µ

( x ) = µ

+ ρ σ

σ

( x − ^µ

) , ˜σ

= σ

q

1 − ^ρ

Bivariate normal properties

E ( X ) = µ

Given X = x, Y is Gaussian

Conditional mean of Y given X = x:

˜

µ

( x ) = µ

+ ρ σ

σ

( x − ^µ

) = E ( Y | ^X = x ) Conditional variance of Y given X = x:

Var ( Y | ^X = x ) = σ

1 − ^ρ

Bivariate normal properties

Conditional mean of Y given X = x:

˜

µ

( x ) = µ

+ ρ σ

σ

( x − ^µ

) = E ( Y | ^X = x ) Conditional variance of Y given X = x:

Var ( Y | ^X = x ) = σ

1 − ^ρ

Check Section 3.7.3. and Exercise 3.8.4.6. By this is seen that the conditional mean of Y given X variable in a bivariate normal

distribution is also the best LINEAR predictor of Y based on X , and

the conditional variance is the variance of the estimation error.

Marginal PDFs

Proof of conditional pdf

Consider

f

( x, y ) f

( x ) = σ

√

2π 2πσ

σ

p 1 − ^ρ

^e

−¹₂^Q(x,y)+ ¹

2σ21

(x−µ1)²

Proof of conditional pdf

− ¹ ₂ ^Q ( x, y ) + ¹

2σ

( x − ^µ

)

= − ¹ ₂ ^H ( x, y ) ,

Proof of conditional pdfs

H ( x, y ) = 1

( 1 − ^ρ

) ·

"

x − ^µ

σ



− ^2ρ ( x − ^µ

)( y − ^µ

)

σ

σ

+ ^{ y} − ^µ

σ



#

− ^{ x} − ^µ

σ



Proof of conditional pdf

H ( x, y ) = ρ

( 1 − ^ρ

)

( x − ^µ

)

σ

− ^2ρ ( x − ^µ

)( y − ^µ

)

σ

σ

( 1 − ^ρ

) + ( y − ^µ

)

σ

( 1 − ^ρ

)

Proof of conditional pdf

H ( x, y ) =



y − ^µ

− ^ρ

( x − ^µ

) ^

σ

( 1 − ^ρ

)

Conditional pdf

f

( x, y ) f

( x ) = 1

p 1 − ^ρ

^σ

√ 2π e



−¹₂

(

)

σ22(1−^ρ2)





This establishes the bivariate normal properties claimed above.

Bivariate normal properties : ρ

Proposition

( X , Y ) bivariate normal ⇒ ^ρ = ρ

Proof:

E [( X − ^µ

)( Y − ^µ

)]

= E ( E ([( X − ^µ

)( Y − ^µ

)] | ^X ))

= E (( X − ^µ

) E [ ^Y − ^µ

] | ^X ))

Bivariate normal properties : ρ

= E (( X − ^µ

) E [( Y − ^µ

)] | ^X ))

= E ( X − ^µ

) [ E ( Y | ^X ) − ^µ

]

= E (( X − ^µ

)



µ

+ ρ σ

σ

( X − ^µ

) − ^µ



= ρ σ

σ

E ( X − ^µ

)(( X − ^µ

))

Bivariate normal properties : ρ

= ρ σ

σ

E ( X − ^µ

)( X − ^µ

)

= ρ σ

σ

E ( X − ^µ

)

= ρ σ

σ

σ

= ρσ

σ

Bivariate normal properties : ρ

In other words we have checked that

ρ = ^E [( ^X − ^µ

)( Y − ^µ

)]

σ

σ

ρ = 0 ⇔ bivariate normal X , Y are independent.

PART 5: Generating a multivariate normal variable

Standard Normal Vector: definition

Z ∈ ^N ( 0, I ) is a standard normal vector.

I is the n × ⁿ identity matrix.

f

( z ) = ¹

( 2π )

^p det ( I ) ^e

−¹₂(z−0)^TI⁻¹(z−0)

= ¹

( 2π )

^e

−¹2z^Tz

Distribution of X = _AZ + b

X = AZ + b, Z is standard Gaussian, then X = N 

b, AA



(follows by a rule in the preceding)

Multivariate Normal: the bivariate case

If

Λ =

 σ

ρσ

σ

ρσ

σ

σ

 , then Λ = AA

, where

A =

 σ

0 ρσ

σ

p 1 − ^ρ



,

Standard Normal Vector

X ∈ ^N ( µ

, Λ ) , and A is such that Λ = AA

(An invertible matrix A with this property exists always, if Λ is positive definite (we need the symmetry of Λ, too.) Then

Z = A

( X − ^µ

) is a standard Gaussian vector.

Proof: We give the first idea of his proof, a rule of transformation.

Rule of transformation

If X has density f

( x ) , Y = AX + b, A is invertible, then

f

( y ) = ¹

| ^{det A} | ^f

^A

−1

( y − ^b )

Note that if Λ = AA

, then

det Λ = det A · ^{det A}

= det A · ^{det A} = det A

, so that | ^{det A} | = √

det Λ.

Johann Carl Friedrich Gauss (30 April 1777 23 February

1855)

Diagonalizable Matrices

An n × ⁿ matrix A is orthogonally diagonalizable, if there is an orthogonal matrix P (i.e., P

P = PP

= I ) such that

P

AP = Λ,

where Λ is a diagonal matrix.

Diagonalizable Matrices

Theorem

If A is an n × ⁿ matrix, then the following are equivalent:

(i) A is orthogonally diagonalizable.

(ii) A has an orthonormal set of eigenvectors.

(iii) A is symmetric.

Since covariance matrices are symmetric, we have by the theorem above

that all covariance matrices are orthogonally diagonalizable.

Diagonalizable Matrices

Theorem

If A is a symmetric matrix, then

(i) Eigenvalues of A are all real numbers.

(ii) Eigenvectors from different eigenspaces are orthogonal.

That is, all eigenvalues of a covariance matrix are real.

Diagonalizable Matrices

Hence we have for any covariance matrix the spectral decomposition C =

n

∑

i=1

λ

e

e

, (1) where Ce

= λ

e

. Since C is nonnegative definite, and its eigenvectors are orthonormal,

0 ≤ ^e

Ce

= λ

e

e

= λ

,

and thus the eigenvalues of a covariance matrix are nonnegative.

Diagonalizable Matrices

Let now P be an orthogonal matrix such that P

C

P = Λ,

and X ∈ ^N ( 0, C

) , i.e., C

is a covariance matrix and Λ is diagonal (with the eigenvalues of C

on the main diagonal). Then if Y = P

X, we have that

Y ∈ ^N ( 0, Λ ) .

In other words, Y is a Gaussian vector and has independent components.

This method of producing independent Gaussians has several important

applications. One of these is the principal component analysis.