EXAM ESTIMATION - DETECTION SOLUTIONS 13/11/2013. Exercise 1: Estimation. 1. Maximum likelihood estimation. (a) = 0 = λ = n 2

EXAMESTIMATION - DETECTION 13/11/2013

SOLUTIONS

Exercise 1: Estimation 1. Maximum likelihood estimation

(a)

f (x1, · · · , xn; λ) =

n

Y

i=1

λ exp (−λxi) = λⁿexp −λ

n

X

i=1

xi

!

ln f (x₁, · · · , x_n; λ) = n ln(λ) − λ

n

X

i=1

x_i

∂ ln f (x1, · · · , xn; λ)

∂λ = n

λ−

n

X

i=1

xi

= 0! =⇒ λ = n Pn

i=1xi

∂²ln f (x1, · · · , xn; λ)

∂λ² = −n

λ² < 0 =⇒ λˆM L = n Pn

i=1xi

(b) Since ϕX(t) = Ee^itX = (1 − it/λ)⁻¹, it follows that

ϕ_{Y =}^Pn

m=1Xm(t) = E h

e^itPnm=1Xm i

=

n

Y

m=1

Ee^itXm = (1 − it/λ)⁻ⁿ and hence T ∼ G(n, 1/λ).

(c) Since ˆλ_{M L} = _Yⁿ with Y ∼ G(n, 1/λ) and E[g(y)] =R g(y)f_Y(y)dy, we find

EhˆλM L

i

= n Z 1

y 1

Γ(n)1/λⁿyⁿ⁻¹exp



− y 1/λ

 dy

= nλΓ(n − 1) Γ(n)

Z 1

Γ(n − 1)1/λⁿ⁻¹yⁿ⁻²exp



− y 1/λ

 dy

| {z }

R G(n−1,1/λ)=1

= n

n − 1λ

=⇒ biased: b(ˆλM L) = 1 n − 1λ.

Ehˆλ²_{M L} i

= n² Z 1

y² 1

Γ(n)1/λⁿyⁿ⁻¹exp



− y 1/λ

 dy

= n²λ²Γ(n − 2) Γ(n)

Z 1

Γ(n − 2)1/λⁿ⁻²yⁿ⁻³exp



− y 1/λ

 dy

| {z }

R G(n−2,1/λ)=1

= n²

(n − 1)(n − 2)λ²

VarhˆλM L

i

= n²

(n − 1)(n − 2)λ²+ n²

(n − 1)²λ² = n²λ²

(n − 1)²(n − 2) =⇒ convergent (CONTROL: the random variable ˆλM L/n has an inverse Gamma distribution IG(n, λ) and hence mean and variance _n−1ⁿ λ and _(n−1)ⁿ²2^λ(n−2)² , respectively.)

(d)

I(λ) = E



−∂²ln f (x₁, · · · , x_n; λ)

∂λ²



= n λ² bias: db(ˆλM L)

dλ = 1

n − 1 CRB(λ) = (1 − b⁰(λ))²

I(λ) = nλ² (n − 1)².

=⇒ ˆλM Lis not the efficient estimator for λ since 1) it is biased and 2) its variance exceeds the CRB

2. Estimator ˆλ2 = ^Pn−1n i=1xi

(a) It follows immediately from 1.(c) that

E[λˆ2] = λ and Var[ˆλ2] = λ² n − 2,

and from 1.(d) that the Cramer-Rao bound is given by CRB(λ) = ^λ_n², and hence that ˆλ₂ is unbiased but not the efficient estimator for λ.

(b) Clearly, Var[ˆλ₂] < Var[ˆλ_{M L}], and since ˆλ₂ is also unbiased, it is preferable. The same conclusion is obviously drawn by considering the mean squared errors of the two estimators.

3. ML estimator for β = ¹_λ (a)

f (x1, · · · , xn; β) =

n

Y

i=1

1 βexp



−x_i β



= 1

βⁿexp −1 β

n

X

i=1

xi

!

ln f (x₁, · · · , x_n; β) = −n ln(β) − 1 β

n

X

i=1

x_i

∂ ln f (x₁, · · · , x_n; β)

∂β = −n

β + 1 β²

n

X

i=1

xi

= 0! =⇒ β = 1 n

n

X

i=1

xi

∂²ln f (x₁, · · · , x_n; β)

∂β² = n

β² − 2 Pn

i=1x_i β³

β>0

∝ n β − 2

n

X

i=1

x_i    

β=¹_nPn i=1xi

= −

n

X

i=1

x_i < 0

=⇒ βˆ_{M L}= 1 n

n

X

i=1

x_i (b) From 1.(b), we have thatPn

i=1xi ∼ G(n, β) therefore E[βˆM L] = 1

nnβ = β =⇒ unbiased Var[ ˆβM L] = 1

n²nβ² = β² n . (c)

E ∂²ln f (x1, · · · , xn; β)

∂β²



= n β² − 2

Pn

i=1E[xi]

β³ = −n β²

=⇒ CRB(β) = β² n

=⇒ βˆ_{M L}is efficient.

4. Bayesian estimation with Jeffrey’s prior for λ

(a) From 1.(d), we immediately have p(λ) ∝pI(λ) ∝ ^√_λⁿ ∝ ¹_λ. (b)

f (λ|x) ∝ 1

λλⁿexp −λ

n

X

i=1

x_i

!

∝ G



n, 1

Pn i=1x_i



=⇒ ˆλ^J_{M M SE} = n Pn

i=1x_i = ˆλ_{M L}. ln f (λ|x) ∝ (n − 1) ln(λ) − λ

n

X

i=1

xi

∂ ln f (λ|x)

∂λ ∝ n − 1

λ −

n

X

i=1

x_i= 0^! =⇒ ˆλ^J_{M AP} = n − 1 Pn

i=1x_i = ˆλ₂.

∂²ln f (λ|x)

∂λ² ∝ −n − 1 λ² < 0

5. Bayesian estimation with Gamma prior G(k, θ) for λ: λ ∼ G(k, θ) (a)

f (λ|x, k, θ) ∝ f (x₁, · · · , x_n; λ)p_λ(λ; k, θ)

∝ λⁿexp −λ

n

X

i=1

xi

! 1

Γ(k)θ^kλ^k−1exp(−λ/k)

∝ G n + k, 1 P_n

i=1x_i+ ¹_θ

! .

(b) Since the posterior law of λ is G



n + k,Pn ¹ i=1xi+¹_θ



, we immediately have λˆM M SE= n + k

Pn

i=1xi+¹_θ.

ln f (λ|x, k, θ) ∝ (n + k − 1) ln(λ) − λ

n

X

i=1

xi+1 θ

!

∂ ln f (λ|x, k, θ)

∂λ ∝ n + k − 1

λ −

n

X

i=1

x_i−1 θ

= 0!

=⇒ ˆλM AP = n + k − 1 P_n

i=1x_i+¹_θ (One verifies that ^∂2ln f (λ|x,k,θ)

∂λ²

λ=ˆλM AP

< 0.) SincePn

i=1x_i  ¹_θ and n  k the estimators become equivalent as as n → ∞.

Exercise 2: Detection

1. We want to test the hypotheses H0 : θ = θ0and H1 : θ = θ1with θ1> θ0. (a)

reject H0if f (x1, · · · , xn; k, θ|H1) f (x1, · · · , xn; k, θ|H0) > Kα

Qn

i=1x^k−1_{i θ}^k

1 exp

−^x_θ^ki

1

 Qn

i=1x^k−1_{i θ}^k

0 exp



−^x_θ^ki

0

 > Kα

 θ₀ θ₁

n

exp −

n

X

i=1

x^k_i  1 θ₁ − 1

θ₀

!

> K_α

n(ln θ0− ln θ₁) +

n

X

i=1

x^k_i  1 θ0

− 1 θ1



| {z }

>0

> ln Kα

T =

n

X

i=1

x^k_i > t_α = θ₀θ₁

θ₁− θ₀ (ln K_α+ n(ln θ₁− ln θ₀)) (b) i. U = T^kand T = U^k1 and ^dT_dU = _k¹U^1k−1and hence

g(u) = f (t(u))    

dT dU    

= k

θu^k−1k exp



−u θ

1

ku^1k−1 = 1 θexp



−u θ



and hence U ∼ G(1, θ).

ii. Since ϕU(t) = Ee^itU = (1 − itθ)^−βwith β = 1, it follows that

ϕ_{T =}^Pn

m=1Um(t) = E h

e^itPnm=1Um i

=

n

Y

m=1

Ee^itUm = (1 − itθ)ⁿ and hence T ∼ G(n, θ).

Therefore, under H0, T ∼ G(n, θ0) and under H₁, T ∼ G(n, θ1).

(c)

α = P [reject H₀|H₀true] = P [T > tα|T ∼ G(n, θ₀)]

= 1 − Z tα

0

xⁿ⁻¹

Γ(n)θ₀ⁿexp(−x/θ₀)dx = 1 − F₀(t_α)

=⇒ tα= F₀⁻¹(1 − α) (d)

π = P [reject H₀|H₁true] = P [T > tα|T ∼ G(n, θ₁)]

= 1 − Z tα

0

xⁿ⁻¹

Γ(n)θⁿ₁ exp(−x/θ₁)dx = 1 − F₁(t_α)

= 1 − F1(F₀⁻¹(1 − α))

2. We first need to determine the maximum likelihood estimator for θ, f (x1, · · · , xn; β, θ) =

n

Y

i=1

kx^k−1_i θ exp



−x^k_i θ



= kⁿ θⁿexp



− Pn

i=1x^k_i θ

 n

Y

i=1

x^k_i

ln f (x1, · · · , xn; β, θ) = n ln(k) − n ln(θ) − 1 θ

n

X

i=1

x^k_i + ln

n

Y

i=1

x^k_i

!

∂ ln f (x₁, · · · , x_n; β, θ)

∂θ = −n

θ + P_n

i=1x^k_i θ²

= 0! =⇒ θ = 1 n

n

X

i=1

x^k_i

∂²ln f (x1, · · · , xn; β, θ)

∂θ²

   θ=_n¹Pn

i=1x^k_i

= −

n

X

i=1

x^k_i < 0 =⇒ θˆM L= 1 n

n

X

i=1

x^k_i.

The generalized likelihood ratio test is given by

reject H0if sup_θ∈Ω1f (x1, · · · , xn|θ)

sup_θ∈Ω0f (x1, · · · , xn|θ) = f (x₁, · · · , x_n|θ = ˆθ_{M L(1)}) f (x1, · · · , xn|θ = θ₀) > Kα

Qn i=1k ^x

k−1 i

θˆM L(1)

exp



−_ˆ ^xki

θM L(1)



Q_n

i=1k^x

k−1 i

θ0 exp

−^x_θ^ki

0

 > K_α

 nθ₀ Pn

i=1x^k_i

n

exp −

 n

Pn

i=1x^k_i − 1 θ₀

 n

X

i=1

x^k_i

!

> K_α

n ln(nθ₀) − n ln

n

X

i=1

x^k_i

!

−

 n

Pn

i=1x^k_i − 1 θ₀

 n

X

i=1

x^k_i > ln K_α

T = − ln

n

X

i=1

x^k_i

! + 1

nθ₀

n

X

i=1

x^k_i > t_α= ln Kα

n + 1 − ln(n) − ln(θ₀).

3. (a) Since the parameter θ is unknown, the Kolmogorov test can not be applied. The χ² test is appropriate for this problem.

(b)

f (t1, · · · , tM; n, θ) = QM

m=1tⁿ⁻¹_m

Γ(n)^Mθ^nM exp − PM

m=1t_m θ

!

ln f (t₁, · · · , t_M; n, θ) ∝ −nM ln(θ) − 1 θ

M

X

m=1

t_m

∂ ln f (t₁, · · · , t_M; n, θ)

∂θ = −nM

θ + P_M

m=1t_m θ²

= 0! =⇒ ˆθ_{M L}= P_M

m=1t_m nM

and hence

θˆ_{M L}= PM

m=1tm

nM = 1250

625 = 2

The mean is given by µ = nˆθM L= 50 and the standard deviation by σ = q

nˆθ²_{M L}= 10.

(c) With a = F⁻¹(3/4), the classes for testing for the standard Normal distribution N (0, 1) are given by

(−∞, −a], (−a, 0] (0, a], (a, ∞)

and the classes for N (µ = 50, σ² = 10²) are therefore

C₁ = (−∞, µ − σa] = (−∞, 43.25]

C2 = (µ − σa, µ] = (43.25, 50]

C3 = (µ, µ + σa] = (50, 56.75]

C4 = (µ + σa, ∞] = (56.75, ∞]

(d) The test statistic is given by

φ =

K

X

k=1

(n_k− M p_k)² M pk

with K = 4, pk= 1/4 and (n1, n2, n3, n4) = (7, 5, 6, 7). Hence, φ = 4

25

 9 16 +25

16 + 1 16 + 9

16



= 11

25 = 0.44.

Since the unknown parameter θ was replaced with its maximum likelihood estimate, the critical value of the test is given by the quantile of the χ²distribution with N = K − 1 − 1 degrees of freedom, tα = 4.61 > 0.44. The test with significance α = 0.1 therefore does not reject the hypothesis that the sample is Normal.