• minimum mean-square estimation (MMSE)

EE363 Winter 2005-06

Lecture 6 Estimation

• Gaussian random vectors

• minimum mean-square estimation (MMSE)

• MMSE with linear measurements

• relation to least-squares, pseudo-inverse

Gaussian random vectors

random vector x ∈ R

is Gaussian if it has density p

(v) = (2π)

(det Σ)

exp



− 1

2 (v − ¯x)

Σ

(v − ¯x)

 ,

for some Σ = Σ

> 0, ¯ x ∈ R

• denoted x ∼ N (¯x, Σ)

• ¯x ∈ R

is the mean or expected value of x, i.e.,

¯

x = E x = Z

vp

(v)dv

• Σ = Σ

> 0 is the covariance matrix of x, i.e.,

Σ = E(x − ¯x)(x − ¯x)

= E xx

− ¯x¯x

= Z

(v − ¯x)(v − ¯x)

p

(v)dv

density for x ∼ N (0, 1):

0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45

PSfrag replacements

p

(v ) =

e

• mean and variance of scalar random variable x

are E x

= ¯ x

, E (x

− ¯x

)

= Σ

hence standard deviation of x

is √

Σ

• covariance between x

and x

is E(x

− ¯x

)(x

− ¯x

) = Σ

• correlation coefficient between x

and x

is ρ

= Σ

pΣ

Σ

• mean (norm) square deviation of x from ¯x is

E kx − ¯xk

= E Tr(x − ¯x)(x − ¯x)

= Tr Σ =

n

X

i=1

Σ

(using Tr AB = Tr BA)

example: x ∼ N (0, I) means x

are independent identically distributed

(IID) N (0, 1) random variables

Confidence ellipsoids

p

(v) is constant for (v − ¯x)

Σ

(v − ¯x) = α, i.e., on the surface of ellipsoid

E

= {v | (v − ¯x)

Σ

(v − ¯x) ≤ α}

thus ¯ x and Σ determine shape of density

can interpret E

as confidence ellipsoid for x:

the nonnegative random variable (x − ¯x)

Σ

(x − ¯x) has a χ

distribution, so Prob(x ∈ E

) = F

n

(α) where F

n

is the CDF some good approximations:

• E gives about 50% probability

geometrically:

• mean ¯x gives center of ellipsoid

• semiaxes are √

αλ

u

, where u

are (orthonormal) eigenvectors of Σ

with eigenvalues λ

example: x ∼ N (¯x, Σ) with ¯x =

 2 1



, Σ =

 2 1 1 1



• x

has mean 2, std. dev. √ 2

• x

has mean 1, std. dev. 1

• correlation coefficient between x

and x

is ρ = 1/ √ 2

• E kx − ¯xk

= 3

90% confidence ellipsoid corresponds to α = 4.6:

−4

−2 0 2 4 6 8

PSfrag replacements

x

Affine transformation

suppose x ∼ N (¯x, Σ

)

consider affine transformation of x:

z = Ax + b, where A ∈ R

, b ∈ R

then z is Gaussian, with mean

E z = E(Ax + b) = A E x + b = A¯ x + b and covariance

Σ

= E(z − ¯z)(z − ¯z)

= E A(x − ¯x)(x − ¯x)

A

= AΣ

A

examples:

• if w ∼ N (0, I) then x = Σ

w + ¯ x is N (¯x, Σ)

useful for simulating vectors with given mean and covariance

• conversely, if x ∼ N (¯x, Σ) then z = Σ

(x − ¯x) is N (0, I)

(normalizes & decorrelates)

suppose x ∼ N (¯x, Σ) and c ∈ R

scalar c

x has mean c

x and variance c ¯

Σc

thus (unit length) direction of minimum variability for x is u, where Σu = λ

u, kuk = 1

standard deviation of u

x is √

λ

(similarly for maximum variability)

Degenerate Gaussian vectors

it is convenient to allow Σ to be singular (but still Σ = Σ

≥ 0) (in this case density formula obviously does not hold)

meaning: in some directions x is not random at all write Σ as

Σ = [Q

Q

]

 Σ

0

0 0



[Q

Q

]

where Q = [Q

Q

] is orthogonal, Σ

> 0

• columns of Q

are orthonormal basis for N (Σ)

• columns of Q

are orthonormal basis for range(Σ)

then Q

x = [z

w

]

, where

• z ∼ N (Q

x, Σ ¯

) is (nondegenerate) Gaussian (hence, density formula holds)

• w = Q

x ¯ ∈ R

is not random

(Q

x is called deterministic component of x)

Linear measurements

linear measurements with noise:

y = Ax + v

• x ∈ R

is what we want to measure or estimate

• y ∈ R

is measurement

• A ∈ R

characterizes sensors or measurements

• v is sensor noise

common assumptions:

• x ∼ N (¯x, Σ

)

• v ∼ N (¯v, Σ

)

• x and v are independent

• N (¯x, Σ

) is the prior distribution of x (describes initial uncertainty about x)

• ¯v is noise bias or offset (and is usually 0)

• Σ

is noise covariance

thus  x v



∼ N

 x ¯

¯ v

 ,

 Σ

0 0 Σ



using

 x y



=

 I 0 A I

  x v



we can write

E

 x y



=

 x ¯ A¯ x + ¯ v

 and

E

 x − ¯x y − ¯y

  x − ¯x y − ¯y



=

 I 0 A I

  Σ

0 0 Σ

  I 0 A I



 Σ

Σ

A



covariance of measurement y is AΣ

A

+ Σ

• AΣ

A

is ‘signal covariance’

• Σ

is ‘noise covariance’

Minimum mean-square estimation

suppose x ∈ R

and y ∈ R

are random vectors (not necessarily Gaussian) we seek to estimate x given y

thus we seek a function φ : R

→ R

such that ˆ x = φ(y) is near x one common measure of nearness: mean-square error,

E kφ(y) − xk

minimum mean-square estimator (MMSE) φ

minimizes this quantity

general solution: φ

(y) = E(x |y), i.e., the conditional expectation of x

MMSE for Gaussian vectors

now suppose x ∈ R

and y ∈ R

are jointly Gaussian:

 x y



∼ N

 

¯ x

¯ y

 ,

 Σ

Σ

Σ

Σ

 

(after alot of algebra) the conditional density is p

(v |y) = (2π)

(det Λ)

exp



− 1

2 (v − w)

Λ

(v − w)

 , where

Λ = Σ

− Σ

Σ

Σ

, w = ¯ x + Σ

Σ

(y − ¯y) hence MMSE estimator (i.e., conditional expectation) is

ˆ

x = φ

(y) = E(x |y) = ¯x + Σ

Σ

(y − ¯y)

φ

is an affine function

MMSE estimation error, ˆ x − x, is a Gaussian random vector ˆ

x − x ∼ N (0, Σ

− Σ

Σ

Σ

)

note that

Σ

− Σ

Σ

Σ

≤ Σ

i.e., covariance of estimation error is always less than prior covariance of x

Best linear unbiased estimator

estimator

ˆ

x = φ

(y) = ¯ x + Σ

Σ

(y − ¯y) makes sense when x, y aren’t jointly Gaussian

this estimator

• is unbiased, i.e., E ˆx = E x

• often works well

• is widely used

• has minimum mean square error among all affine estimators

sometimes called best linear unbiased estimator

MMSE with linear measurements

consider specific case

y = Ax + v, x ∼ N (¯x, Σ

), v ∼ N (¯v, Σ

), x, v independent

MMSE of x given y is affine function ˆ

x = ¯ x + B(y − ¯y) where B = Σ

A

(AΣ

A

+ Σ

)

, ¯ y = A¯ x + ¯ v intepretation:

• ¯x is our best prior guess of x (before measurement)

• estimator modifies prior guess by B times this discrepancy

• estimator blends prior information with measurement

• B gives gain from observed discrepancy to estimate

• B is small if noise term Σ

in ‘denominator’ is large

MMSE error with linear measurements

MMSE estimation error, ˜ x = ˆ x − x, is Gaussian with zero mean and covariance

Σ

= Σ

− Σ

A

(AΣ

A

+ Σ

)

AΣ

• Σ

≤ Σ

, i.e., measurement always decreases uncertainty about x

• difference Σ

− Σ

gives value of measurement y in estimating x

• e.g., (Σ

/Σ

)

gives fractional decrease in uncertainty of x

due to measurement

note: error covariance Σ

can be determined before measurement y is

made!

to evaluate Σ

, only need to know

• A (which characterizes sensors)

• prior covariance of x (i.e., Σ

)

• noise covariance (i.e., Σ

)

you do not need to know the measurement y (or the means ¯ x, ¯ v)

useful for experiment design or sensor selection

Information matrix formulas

we can write estimator gain matrix as

B = Σ

A

(AΣ

A

+ Σ

)

= A

Σ

A + Σ

A

Σ

• n × n inverse instead of m × m

• Σ

, Σ

sometimes called information matrices corresponding formula for estimator error covariance:

Σ = Σ − Σ A

(AΣ A

+ Σ )

AΣ

can interpret Σ

= Σ

+ A

Σ

A as:

posterior information matrix (Σ

)

= prior information matrix (Σ

)

+ information added by measurement (A

Σ

A)

proof: multiply

Σ

A

(AΣ

A

+ Σ

)

= A

Σ

A + Σ

A

Σ

on left by (A

Σ

A + Σ

) and on right by (AΣ

A

+ Σ

) to get

(A

Σ

A + Σ

)Σ

A

= A

Σ

(AΣ

A

+ Σ

)

which is true

Relation to regularized least-squares

suppose ¯ x = 0, ¯ v = 0, Σ

= α

I, Σ

= β

I estimator is ˆ x = By where

B = A

Σ

A + Σ

A

Σ

= (A

A + (β/α)

I)

A

. . . which corresponds to regularized least-squares MMSE estimate ˆ x minimizes

kAz − yk

+ (β/α)

kzk

over z

Example

navigation using range measurements to distant beacons

y = Ax + v

• x ∈ R

is location

• y

is range measurement to ith beacon

• v

is range measurement error, IID N (0, 1)

• ith row of A is unit vector in direction of ith beacon prior distribution:

x ∼ N (¯x, Σ ), x = ¯

 1 

, Σ =

 2

0 

90% confidence ellipsoid for prior distribution { x | (x − ¯x)

Σ

(x − ¯x) ≤ 4.6 }:

−6 −4 −2 0 2 4 6

−5

−4

−3

−2

−1 0 1 2 3 4 5

PSfrag replacements

x

x

Case 1: one measurement, with beacon at angle 30

fewer measurements than variables, so combining prior information with measurement is critical

resulting estimation error covariance:

Σ

=

 1.046 −0.107

−0.107 0.246



90% confidence ellipsoid for estimate ˆ x: (and 90% confidence ellipsoid for x)

−6 −4 −2 0 2 4 6

−5

−4

−3

−2

−1 0 1 2 3 4 5

PSfrag replacements

x

x

interpretation: measurement

• yields essentially no reduction in uncertainty in x

• reduces uncertainty in x

by a factor about two

Case 2: 4 measurements, with beacon angles 80

, 85

, 90

, 95

resulting estimation error covariance:

Σ

=

 3.429 −0.074

−0.074 0.127



90% confidence ellipsoid for estimate ˆ x: (and 90% confidence ellipsoid for x)

−6 −4 −2 0 2 4 6

−5

−4

−3

−2

−1 0 1 2 3 4 5

PSfrag replacements

x

x

interpretation: measurement yields

• little reduction in uncertainty in x

• small reduction in uncertainty in x