Modelling Missing Data

Modelling Missing Data

Complete-data model:

◦ independent and identically distributed (iid) draws Y ₁ , . . . , Y _n

◦ from multivariate distribution P _θ , Y _i = (Y _i1 , . . . , Y _ip ) ^T ∼ P _θ .

◦ Data matrix:

Y = (Y ₁ , . . . , Y _n ) ^T = (Y _ij ) i=1,...,n;j=1,...,p

1 2 ...

n

Y _i1 Y _i2 · · ·

· · · Y _ip Units

Inference: estimate θ

◦ likelihood approach maximize likelihood function

◦ Bayesian approach consider posterior distribution Likelihood function of the complete data:

f _Y (y|θ) =

n

Q

i=1

f _Y

(y _i |θ)

where f _Y

(·|θ) is the density of P _θ

Modelling Missing Data

Problem: Some data Y ij may be missing

Example: Cholesterol levels of heart-attack patients Data:

◦ Serum-cholesterol levels for n = 28 patients treated for heart attacks.

◦ Cholesterol levels were measured for all patients 2 and 4 days after the attack.

◦ For 19 of the 28 patients, an additional measurement was taken 14 days after the attack.

◦ See also Schafer, sections 5.3.6 and 5.4.3.

Id Y ₁ Y ₂ Y ₃ Id Y ₁ Y ₂ Y ₃ 1 270 218 156 15 294 240 264

2 236 234 — 16 282 294 —

3 210 214 242 17 234 220 264

4 142 116 — 18 224 200 —

5 280 200 — 19 276 220 188

6 272 276 256 20 282 186 182

7 160 146 142 21 360 352 294

8 220 182 216 22 310 202 214

9 226 238 248 23 280 218 —

10 242 288 — 24 278 248 198

11 186 190 168 25 288 278 —

12 266 236 236 26 288 248 256

13 206 244 — 27 244 270 280

14 318 258 200 28 236 242 204

Modelling Missing Data

Idea: describe missingness of Y ij by indicator variable R ij

R _ij =  1 Y _ij has been observed 0 Y _ij is missing

Multivariate dataset with missing values:

Y _i1 Y _i2 · · · Y _ip R _i1 R _i2 · · · R _ip

Units 1, . . . , n 1 1 1 1

. . . , n ₂ 0 1 1

. . . , n ₃ 1 0 1

. . . , n ₄ 1 1 0

. . . , n ₅ 0 0 1

. . . , n 6 0 1 0

. . . , n ₇ 0 1 0

. . . , n ₈ 0 0 0

Observed-data model:

◦ The observed information consists of

 observed values y _ij

 the values r _ij indicating which values are missing

◦ A statistical model for the observations should make use of all available information

Notation:

Y = (Y _obs , Y _mis ) where

◦ Y _obs are the observed variables

◦ Y _mis are the missing variables

Missing Data Mechanism

Statistical model for missing data:

P(R = r|Y = y) = f R|Y (r|y, ξ) with parameters ξ ∈ Ξ.

Definition Missing completely at random (MCAR)

The observations are missing completely at random if P(R = r|Y = y) = P(R = r)

or equivalently

f _R|Y (r|y, ξ) = f _R (r, ξ),

that is, R and Y are independent.

Definition Missing at random (MAR)

The observations are missing at random if P(R = r|Y = y) = P(R = r|Y ^obs ) or equivalently

f _R|Y (r|y, ξ) = f _R|Y

(r|y _obs , ξ),

that is, knowledge about Y _mis does not provide any additional infor- mation about R if Y obs is already known (R and Y mis are conditionally independent given Y _obs ).

Definition Not missing at random (NMAR)

Observations are not missing at random if the above assumption for

MAR is not fulfilled.

Missing Completely at Random (MCAR)

Example: Bernoulli selection

◦ Complete data: Y ₁ , . . . , Y _n

◦ Response indicator: R ₁ , . . . , R _n

◦ Missingness mechanism: Each unit is observed with probability ξ f _R (r|ξ) =

n

Q

i=1

ξ ^r

(1 − ξ) ^1−r

Example: Simple random sample (SRS)

◦ Suppose that only m observations are available, n − m values are miss- ing.

◦ Responding units are simple random sample of all units:

f _R (r|ξ) = ( _n

m

−1

if P n

i=1 r _i = m

0 otherwise

◦ Note that ξ = m is a parameter of the distribution of R

Missing at Random (MAR)

Example: Hypertension trials

Data: Serial measurements of blood pressure during long-term trials of drugs which reduce blood pressure

Problem: Patients are withdrawn during the course of the study because of inadequate blood pressure control.

Example: from: Murray and Findlay (1988), Correcting for the bias caused by drop- outs in hypertension trials, Statistics in Medicine 7, 941-946.

◦ International double-blind randomized study with 429 patients

◦ 218 receive β-blocker metoprolol

◦ 211 receive serotonergic S ₂ -receptor blocking agent ketanserin

◦ Treatment phase lasted 12 weeks

◦ Scheduled clinic visits for weeks 0, 2, 4, 8, and 12

◦ Patients with diastolic blood pressure exceeding 110 mmHg in week 4 or week 8 “jump” to an open follow-up phase.

◦ Missing data pattern for the two groups:

Metoprolol group

0 2 4 8 12 Number of patients

1 1 1 1 1 162

1 1 1 1 0 14 (11 jumps)

1 1 1 0 1 2

1 1 1 0 0 30 (28 jumps)

1 1 0 1 1 1

1 1 0 0 0 2

1 0 1 1 1 2

1 0 1 0 1 1

1 0 1 0 0 1

1 0 0 1 1 1

1 0 0 0 1 1

1 0 0 0 0 1

Ketanserin group

0 2 4 8 12 Number of patients

1 1 1 1 1 136

1 1 1 1 0 19 (18 jumps)

1 1 1 0 1 1

1 1 1 0 0 41 (37 jumps)

1 1 0 1 1 2

1 1 0 0 0 4

1 0 1 1 1 2

1 0 1 1 0 1

1 0 1 0 1 1

1 0 1 0 0 1

1 0 0 1 1 2

1 0 0 1 0 1

1 0 0 0 0 1

◦ “Jumps” (majority of missing observations) lead to missing data which

are missing at random

Missing at Random (MAR)

Mean diastolic blood pressure based

a. on all available data at each time point b. on patients with complete data

c. on all available data, assuming missing data to be missing at random

Weeks on Treatment

Mean DBP (mmHg)

0 4 8 12

90 95 100 105 110

Ketanserin Metoprolol

(a)

Weeks on Treatment

Mean DBP (mmHg)

0 4 8 12

90 95 100 105 110

Ketanserin Metoprolol

(b)

Weeks on Treatment

Mean DBP (mmHg)

0 4 8 12

90 95 100 105 110

Ketanserin Metoprolol

(c)

Missing at Random (MAR)

Examples for data missing at random:

◦ Double sampling: sample survey:

 observe variables Y _i1 , . . . , Y _ik for all individuals i = 1, . . . , n

 observe variables Y _i,k+1 , . . . , Y _ip only for subsample

◦ Sampling with nonresponse followup: sample survey with nonresponses

 intensive followup effort for all units not feasible

 take random sample of all nonresponding units

◦ Randomized experiment with unequal numbers of cases per treatment group:

 suppose original design was balanced

 missingness mechanism is deterministic, thus data are MAR

◦ Matrix sampling for questionnaire or test items

 divide test or questionnaire into sections

 groups of sections are administered to subjects in a randomized

fashion

Not Missing at Random (NMAR)

Example: Medical treatment of blood pressure n individuals are treated for high blood pressure

◦ Observations: binary responses

Y _ij = blood pressure of ith individual on jth day

◦ Missing observations: (not every individual shows up every day) R _ij =  1 ith individual appears for measurement on jth day

0 otherwise

◦ Statistical model for Y and R:

Y _ij ^iid ∼ N (µ _i , σ ² ) R _ij ^iid ∼ Bin 

1, 1 − ^{exp(Y ij )}

1 + exp(Y _ij )



that is, individuals with high blood pressure are less likely to turn

up for a measurement

Not Missing at Random (NMAR)

Example: Randomly censored data

◦ Objective: Analysis of data (e.g. survival times) T ₁ , . . . , T _n ^iid ∼ f (·|θ)

◦ Survivor function S(t): Probability of surviving beyond time t S(t) = P(T 1 > t)

◦ Censoring times associated with T _i ’s:

C ₁ , . . . , C _n ^iid ∼ g(·|ξ)

(e.g. due to loss to follow up, drop out, or termination of the study)

◦ Observations: (Y ₁ , R ₁ ), . . . , (Y _n , R _n ) where Y _i = min(T _i , C _i )

and

R _i =  1 if T i ≤ C _i

0 otherwise

Observed-Data Likelihood

Likelihood of complete observations (Y obs , R) L _n (θ, ξ|Y _obs , R) = f _Y

_,R (Y _obs , R|θ, ξ)

= Z

f _Y,R (Y _obs , y _mis , R|θ, ξ) dy _mis

= Z

f _R|Y (R|Y _obs , y _mis |ξ)f _Y (Y _obs , y _mis |θ) dy _mis

If the missing observations are missing at random (MAR), the first factor does not depend on y _mis and thus is not affected by integration over y _mis :

L n (θ, ξ|Y obs , R) = Z

f _R|Y (R|Y obs |ξ)f _Y (Y obs , y mis |θ) dy _mis

= f _R|Y (R|Y _obs |ξ) Z

f _Y (Y _obs , y _mis |θ) dy _mis

= f _R|Y (R|Y obs |ξ)f _Y

(Y obs |θ)

= f _R|Y (R|Y _obs |ξ)L _n (θ|Y _obs )

If the parameters ξ do not depend on θ, we get θ = argmax ˆ

θ∈Θ

L _n (θ, ξ|Y _obs , R) ⇔ ˆ θ = argmax

θ∈Θ

L _n (θ|Y _obs ),

that is, we can obtain the maximum likelihood estimator by minimizing L _n (θ|Y _obs ) with respect to θ.

Definition Observed-data likelihood The likelihood function

L _n (θ|Y _obs ) = f _Y

(Y _obs |θ)

is called the likelihood ignoring the missing-data mechanism or short

observed-data likelihood.

Ignorability

Definition

A missing-data mechanism is ignorable for likelihood inference if

◦ observations are missing at random (MAR) and

◦ the parameters ξ (missingness-mechanism) and θ θ (data model) are distinct, in the sense that the joint parameter space of (θ, ξ) is the product of the parameter spaces Ξ and Θ.

Example: Non-distinct parameters Suppose that

◦ Y _i ^iid ∼ Bin(1, θ)

◦ R _i ^iid ∼ Bin(1, θ)

◦ Y _obs = (Y ₁ , . . . , Y _m ) ^T (after reordering) The joint likelihood of Y _obs and R is

L n (θ|Y obs , R) =

n

Q

i=1

θ ^r

(1 − θ) ^1−r

m

Q

i=1

θ ^y

(1 − θ) ^1−y

= θ ^m+Y (1 − θ) ^n−Y

This is the likelihood of a Bernoulli experiment with n + m observations.

Consequently θ ˆ _ML = m + Y

m + n .

Ignoring the missing-data mechanism: The observed-data likelihood is L _n (θ|Y _obs ) =

m

Q

i=1

θ ^y

(1 − θ) ^1−y

which leads to the ML estimator

θ ˆ ML = Y

m .

Ignorability

The first condition (MAR) is typically regarded as the more important condition:

◦ If MAR does not hold, then the maximum likelihood estimator based on the observed-data likelihood can be seriously biased.

◦ If the data are MAR but distinctness does not hold, inference based on the observed-data likelihood L _n (θ|Y _obs ) is still valid from the frequentist perspective, but not fully efficient.

Example:

Suppose

◦ Y _i ^iid ∼ N (µ, σ ² )

◦ R _i ^iid ∼ Bin 1, e ^µ /(1 + e ^µ )
parameter are not distinct

Likelihood

µ

−0.4 −0.2 0.0 0.2 0.4

Ln(µ,1|Yobs) Ln(µ,1|R) L_n(µ,1|R,Yobs)

MLE based on Ln

(

)

µ^_obs

Frequency

−0.4 −0.2 0.0 0.2 0.4

0 30 60 90 120 150 180

MLE based on Ln

(

)

^

Frequency

−0.4 −0.2 0.0 0.2 0.4

0 30 60 90 120 150 180

Frequentist interpretation:

◦ Parameter θ ₀ = (µ 0 , σ ₀ ² ) is fixed.

◦ If R _i ^iid ∼ Bin(1, ¹ ₂ ) then:

missingness mechanism ignorable ˆ

µ _obs MLE for θ

◦ If µ ₀ = 0 then R _i ^iid ∼ Bin(1, ¹ ₂ ) Properties of ˆ µ _obs unchanged.

◦ However, ˆ µ _compl is more efficient.

Univariate Data with Missing Observations

Example: Incomplete univariate data

Unit

Y R

1 2 ...

m ...

n

1 1 ...

1 0 ...

0 Suppose that

◦ Y ₁ , . . . , Y _n is an iid random sample

◦ only Y _obs = (Y ₁ , . . . , Y _m ) ^T have been observed

◦ Y _mis = (Y _m+1 , . . . , Y _n ) ^T are missing at random The observed-data likelihood is

L _n (θ|Y _obs ) = Z

f _Y (Y _obs , y _mis |θ) dy _mis

= Z

· · · Z _m

Q

i=1

f _Y

(Y _i |θ)

n

Q

i=m+1

f _Y

(y _i |θ) dy _m+1 · · · dy _n

=

m

Q

i=1

f Y

(Y i |θ) Z

· · ·

Z _n

Q

i=m+1

f Y

(y i |θ) dy _m+1 · · · dy _n

=

m

Q

i=1

f _Y

(Y _i |θ)

Thus the observed-data likelihood is a complete-data likelihood based on

the reduced sample (Y ₁ , . . . , Y _m ).

Multivariate Normal Distribution

Let Y = (Y 1 , . . . , Y p ) ^T ∼ N (µ, Σ).

Properties

◦ If p = 2

Σ = σ ₁ ² σ ₁₂ σ 12 σ ₂ ²

!

= σ ₁ ² ρσ ₁ σ ₂ ρσ 1 σ 2 σ ₂ ²

!

where

ρ = σ ₁₂ σ ₁ σ ₂

is the correlation between Y ₁ and Y ₂ .

◦ Density of Y

f _Y (y|µ, Σ) = (2π) ⁻

(det(Σ)) ⁻

exp − ¹ ₂ (y − µ) ^T Σ ⁻¹ (y − µ)
,

◦ Linear transformations

AY + b ∼ N (Aµ + b, AΣA ^T ).

◦ Marginal distribution of Y ₁ Y ₁ ∼ N (µ ₁ , σ ₁ ² )

◦ Conditional distribution of Y ₂ given Y ₁ Y ₂ |Y ₁ ∼ N β ₀ + β ₁ Y ₁ , σ _2|1 ²
,

with parameters β ₀ = µ ₂ − β ₁ µ ₁ β ₁ = σ ₁₂

σ ₁ ²

σ _2|1 ² = σ ² ₂ − σ ² ₁₂

σ ₁ ²

Bivariate Normal Data

Example: Bivariate data with one variable subject to nonresponse

1 2 ...

m ...

n

1 1 ...

1 0 ...

0 Y ₁ Y ₂ R ₂ Units

Suppose that

Y _i = (Y _i1 , Y _i2 ) ^{T iid} ∼ N (µ, Σ)

with mean µ = (µ ₁ , µ ₂ ) ^T and covariance matrix Σ = σ ₁ ² σ ₁₂

σ 12 σ ₂ ²

! .

The observed-data likelihood is given by L _n (θ|Y _obs ) =

m

Q

i=1

f _Y

(Y _i |µ, Σ)

n

Q

i=m+1

f _Y

(Y _i1 |µ ₁ , σ ₁ ² )

=

m

Q

i=1

f _Y

_|Y

(Y _i2 |Y _i1 , β ₀ , β ₁ , σ _2|1 ² )

n

Q

i=1

f _Y

(Y _i1 |µ ₁ , σ ₁ ² ).

Bivariate Normal Data

Example: Bivariate data with one variable subject to nonresponse Data: Y _i = (Y _i1 , Y _i2 ) ^{T iid} ∼ N (µ, Σ)

◦ Parameter: µ ₁ = µ ₂ = 0, σ ₁ ² = σ ² ₂ = 1, ρ

◦ Y _i1 completely observed

◦ Y _i2 has missing values

Missing data mechanism: Proportion of missing data 1 − p = 20%

◦ Missing completely at random (MCAR):

R _i2 ^iid ∼ Bin(1, p)

◦ Missing at random (MAR):

R _i2 = 0 if Y _i1 > z _p

◦ Not missing at random (NMAR):

R _i2 = 0 if Y _i2 > z _p Simulation study:

◦ 1000 repetitions for ρ = 0, 0.5, 0.9

◦ Simulated means (standard variations) for ˆ µ _CC and ˆ µ _{M L}

MCAR MAR NMAR

ˆ

µ _CC µ ˆ _{M L} µ ˆ _CC µ ˆ _{M L} µ ˆ _CC µ ˆ _{M L} ρ = 0 0.00046 0.00048 -0.00095 -0.00143 -0.34585 -0.34587

(0.10762) (0.10797) (0.10876) (0.12250) (0.08760) (0.08776) ρ = 0.5 0.00557 0.00534 -0.17073 0.00459 -0.34919 -0.29226

(0.11294) (0.11229) (0.10585) (0.11850) (0.08406) (0.08171) ρ = 0.9 -0.00218 -0.00208 -0.31229 -0.00130 -0.34964 -0.10023

(0.11012) (0.10093) (0.09034) (0.10361) (0.08460) (0.08877)

Bivariate Normal Data

Missing Values in Y

(MCAR)

Y

Y

−3 −2 −1 0 1 2 3

−3

−2

−1 0 1 2 3

observed missing

Missing Values in Y

(MAR)

Y

Y

−3 −2 −1 0 1 2 3

−3

−2

−1 0 1 2 3

observed missing

Missing Values in Y

(NMAR)

Y

Y

−3 −2 −1 0 1 2 3

−3

−2

−1 0 1 2 3

observed

missing

Bivariate Normal Data

Complete−case (MCAR)

µ ^

Frequency

−0.8 0 −0.4 0.0 0.4

50 100 150 200

250 ρ = 0

Complete−case (MAR)

µ ^

Frequency

−0.8 0 −0.4 0.0 0.4

50 100 150 200

250 ρ = 0

Complete−case (NMAR)

µ ^

Frequency

−0.8 0 −0.4 0.0 0.4

50 100 150 200

250 ρ = 0

MLE (MCAR)

µ ^

Frequency

−0.8 0 −0.4 0.0 0.4

50 100 150 200

250 ρ = 0

MLE (MAR)

µ ^

Frequency

−0.8 0 −0.4 0.0 0.4

50 100 150 200

250 ρ = 0

MLE (NMAR)

µ ^

Frequency

−0.8 0 −0.4 0.0 0.4

50 100 150 200

250 ρ = 0

Bivariate Normal Data

Complete−case (MCAR)

µ ^

Frequency

−0.8 0 −0.4 0.0 0.4

50 100 150 200

250 ρ = 0.5

Complete−case (MAR)

µ ^

Frequency

−0.8 0 −0.4 0.0 0.4

50 100 150 200

250 ρ = 0.5

Complete−case (NMAR)

µ ^

Frequency

−0.8 0 −0.4 0.0 0.4

50 100 150 200

250 ρ = 0.5

MLE (MCAR)

µ ^

Frequency

−0.8 0 −0.4 0.0 0.4

50 100 150 200

250 ρ = 0.5

MLE (MAR)

µ ^

Frequency

−0.8 0 −0.4 0.0 0.4

50 100 150 200

250 ρ = 0.5

MLE (NMAR)

µ ^

Frequency

−0.8 0 −0.4 0.0 0.4

50 100 150 200

250 ρ = 0.5

Bivariate Normal Data

Complete−case (MCAR)

µ ^

Frequency

−0.8 0 −0.4 0.0 0.4

50 100 150 200

250 ρ = 0.9

Complete−case (MAR)

µ ^

Frequency

−0.8 0 −0.4 0.0 0.4

50 100 150 200

250 ρ = 0.9

Complete−case (NMAR)

µ ^

Frequency

−0.8 0 −0.4 0.0 0.4

50 100 150 200

250 ρ = 0.9

MLE (MCAR)

µ ^

Frequency

−0.8 0 −0.4 0.0 0.4

50 100 150 200

250 ρ = 0.9

MLE (MAR)

µ ^

Frequency

−0.8 0 −0.4 0.0 0.4

50 100 150 200

250 ρ = 0.9

MLE (NMAR)

µ ^

Frequency

−0.8 0 −0.4 0.0 0.4

50 100 150 200

250 ρ = 0.9

Bivariate Normal Data

Example: Bivariate data with both variables subject to nonresponse

1 ...

l l + 1

...

m m + 1

...

n

1 ...

1 0 ...

0 1 ...

1 1

...

1 1 ...

1 0 ...

0 Y ₁ Y ₂ R ₁ R ₂ Units

Data: Y _i = (Y _i1 , Y _i2 ) ^{T iid} ∼ N (µ, Σ)

◦ Both variables subject to nonresponse The observed-data likelihood is given by

L n (θ|Y obs ) =

l

Q

i=1

f Y

(Y i |µ, Σ)

m

Q

i=l+1

f Y

(Y i1 |µ ₁ , σ ₁ ² )

n

Q

i=m+1

f Y

(Y i2 |µ ₂ , σ ₂ ² )

In this case, no appropriate parametrization which leads to a factorization

into likelihoods with distinct parameters can be found.

Censored Variables

Example: Randomly censored data

◦ Objective: Analysis of data Y ₁ , . . . , Y _n ^iid ∼ N (µ, σ ² )

◦ Fixed censoring at c:

R _i =  1 Y _i ≤ c 0 otherwise

The likelihood of the complete observations is L n (θ|Y obs , R) =

m

Q

i=1

f Y

(Y i |µ, σ ² )

n

Q

i=m+1

Z ∞ c

f Y

(y i |µ, σ ² ) dy i

=

m

Q

i=1

f _Y

(Y _i |µ, σ ² )

n

Q

i=m+1

S _Y

(c)

=

m

Q

i=1

1

σ ϕ  Y i − µ σ

 _n

Q

i=m+1



1 − Φ  c − µ σ



where

S _Y

(y) = 1 − Φ  y − µ σ



is the survivor function of Y _i . Notation:

◦ ϕ(y) is the density of the standard normal distribution, ϕ(y) = 1

√ 2π exp

 − 1 2 y ²

 .

◦ Φ(y) is the corresponding cumulative distribution function (CDF), Φ(y) =

Z y

−∞

ϕ(z) dz.

Censored Variables

Example: Randomly censored data Derivation of the likelihood:

We have for the missing data mechanism

P(R ⁱ = r _i |Y ) = P(R ⁱ = r _i |Y _i ) = 1 _(−∞,c] (Y _i ) ^r

(1 − 1 _(−∞,c] (Y _i )) ^1−r

Hence the likelihood of the complete data (including R) is

L n (θ|Y, R) =

n

Q

i=1

f Y

(Y i |θ)

n

Q

i=1

1 _(−∞,c] (Y i ) ^R

(1 − 1 _(−∞,c] (Y i )) ^1−R

Since R _i = 1 and thus 1 _(−∞,c] (Y _i ) ^R

= 1 for uncensored observations Y _i , we have

L _n (θ|Y _obs , R)

= Z _m

Q

i=1

f _Y

(Y _i |θ)

n

Q

i=m+1

f _Y

(y _i |θ)1 _(−∞,c] (y _i ) ^R

1 _(c,∞) (y _i ) ^1−R

dy _m+1 · · · dy _n

=

m

Q

i=1

f _Y

(Y _i |θ)

Z _n

Q

i=m+1

f _Y

(y _i |θ)1 _(c,∞) (Y _i ) dy _m+1 · · · dy _n

=

m

Q

i=1

f _Y

(Y _i |θ) Z ∞

c

n

Q

i=m+1

f _Y

(y _i |θ) dy _m+1 · · · dy _n