Modelling Missing Data Complete-data model:
◦ independent and identically distributed (iid) draws Y 1 , . . . , Y n
◦ from multivariate distribution P θ , Y i = (Y i1 , . . . , Y ip ) T ∼ P θ .
◦ Data matrix:
Y = (Y 1 , . . . , Y n ) T = (Y ij ) i=1,...,n;j=1,...,p
1 2 ...
n
Y i1 Y i2 · · ·
· · · Y ip
Units
Inference: estimate θ
◦ likelihood approach à maximize likelihood function
◦ Bayesian approach à consider posterior distribution Likelihood function of the complete data:
f Y (y|θ) = Q n
i=1 f Y i (y i |θ) where f Y i ( ·|θ) is the density of P θ
Maximum Likelihood with Missing Data, Mar 30, 2004 - 1 -
Modelling Missing Data
Problem: Some data Y ij may be missing
Example: Cholesterol levels of heart-attack patients Data:
◦ Serum-cholesterol levels for n = 28 patients treated for heart attacks.
◦ Cholesterol levels were measured for all patients 2 and 4 days after the attack.
◦ For 19 of the 28 patients, an additional measurement was taken 14 days after the attack.
◦ See also Schafer, sections 5.3.6 and 5.4.3.
Id Y 1 Y 2 Y 3 Id Y 1 Y 2 Y 3
1 270 218 156 15 294 240 264 2 236 234 — 16 282 294 — 3 210 214 242 17 234 220 264 4 142 116 — 18 224 200 — 5 280 200 — 19 276 220 188 6 272 276 256 20 282 186 182 7 160 146 142 21 360 352 294 8 220 182 216 22 310 202 214 9 226 238 248 23 280 218 — 10 242 288 — 24 278 248 198 11 186 190 168 25 288 278 — 12 266 236 236 26 288 248 256 13 206 244 — 27 244 270 280 14 318 258 200 28 236 242 204
Maximum Likelihood with Missing Data, Mar 30, 2004 - 2 -
Modelling Missing Data Idea: describe missingness of Y ij by indicator variable R ij
R ij = ½ 1 Y ij has been observed 0 Y ij is missing Multivariate dataset with missing values:
Y i1 Y i2 · · · Y ip R i1 R i2 · · · R ip
Units 1, . . . , n 1 1 1 1
. . . , n 2 0 1 1
. . . , n 3 1 0 1
. . . , n 4 1 1 0
. . . , n 5 0 0 1
. . . , n 6 0 1 0
. . . , n 7 0 1 0
. . . , n 8 0 0 0
Observed-data model:
◦ The observed information consists of
¦ observed values y ij
¦ the values r ij indicating which values are missing
◦ A statistical model for the observations should make use of all available information
Notation:
Y = (Y obs , Y mis ) where
◦ Y obs are the observed variables
◦ Y mis are the missing variables
Missing Data Mechanism
Statistical model for missing data:
(R = r |Y = y) = f R|Y (r |y, ξ) with parameters ξ ∈ Ξ.
Definition Missing completely at random (MCAR) The observations are missing completely at random if
(R = r |Y = y) = (R = r) or equivalently
f R|Y (r |y, ξ) = f R (r, ξ), that is, R and Y are independent.
Definition Missing at random (MAR) The observations are missing at random if
(R = r |Y = y) = (R = r|Y obs ) or equivalently
f R |Y (r|y, ξ) = f R |Y obs (r|y obs , ξ),
that is, knowledge about Y mis does not provide any additional infor- mation about R if Y obs is already known (R and Y mis are conditionally independent given Y obs ).
Definition Not missing at random (NMAR)
Observations are not missing at random if the above assumption for
MAR is not fulfilled.
Missing Completely at Random (MCAR)
Example: Bernoulli selection
◦ Complete data: Y 1 , . . . , Y n
◦ Response indicator: R 1 , . . . , R n
◦ Missingness mechanism: Each unit is observed with probability ξ f R (r |ξ) =
n
Q
i=1
ξ r i (1 − ξ) 1 −r i
Example: Simple random sample (SRS)
◦ Suppose that only m observations are available, n − m values are miss- ing.
◦ Responding units are simple random sample of all units:
f R (r |ξ) = (¡ n
m
¢ −1 if P n i=1 r i = m
0 otherwise
◦ Note that ξ = m is a parameter of the distribution of R
Maximum Likelihood with Missing Data, Mar 30, 2004 - 5 -
Missing at Random (MAR)
Example: Hypertension trials
Data: Serial measurements of blood pressure during long-term trials of drugs which reduce blood pressure
Problem: Patients are withdrawn during the course of the study because of inadequate blood pressure control.
Example: from: Murray and Findlay (1988), Correcting for the bias caused by drop- outs in hypertension trials, Statistics in Medicine 7, 941-946.
◦ International double-blind randomized study with 429 patients
◦ 218 receive β-blocker metoprolol
◦ 211 receive serotonergic S 2 -receptor blocking agent ketanserin
◦ Treatment phase lasted 12 weeks
◦ Scheduled clinic visits for weeks 0, 2, 4, 8, and 12
◦ Patients with diastolic blood pressure exceeding 110 mmHg in week 4 or week 8 “jump” to an open follow-up phase.
◦ Missing data pattern for the two groups:
Metoprolol group 0 2 4 8 12 Number of patients
1 1 1 1 1 162
1 1 1 1 0 14 (11 jumps)
1 1 1 0 1 2
1 1 1 0 0 30 (28 jumps)
1 1 0 1 1 1
1 1 0 0 0 2
1 0 1 1 1 2
1 0 1 0 1 1
1 0 1 0 0 1
1 0 0 1 1 1
1 0 0 0 1 1
1 0 0 0 0 1
Ketanserin group 0 2 4 8 12 Number of patients 1 1 1 1 1 136 1 1 1 1 0 19 (18 jumps)
1 1 1 0 1 1
1 1 1 0 0 41 (37 jumps)
1 1 0 1 1 2
1 1 0 0 0 4
1 0 1 1 1 2
1 0 1 1 0 1
1 0 1 0 1 1
1 0 1 0 0 1
1 0 0 1 1 2
1 0 0 1 0 1
1 0 0 0 0 1
◦ “Jumps” (majority of missing observations) lead to missing data which are missing at random
Maximum Likelihood with Missing Data, Mar 30, 2004 - 6 -
Missing at Random (MAR) Mean diastolic blood pressure based
a. on all available data at each time point b. on patients with complete data
c. on all available data, assuming missing data to be missing at random
Weeks on Treatment
Mean DBP (mmHg)
0 4 8 12
90 95 100 105 110
Ketanserin Metoprolol
(a)
Weeks on Treatment
Mean DBP (mmHg)
0 4 8 12
90 95 100 105 110
Ketanserin Metoprolol
(b)
Weeks on Treatment
Mean DBP (mmHg)
0 4 8 12
90 95 100 105 110
Ketanserin Metoprolol
(c)
Missing at Random (MAR) Examples for data missing at random:
◦ Double sampling: sample survey:
¦ observe variables Y i1 , . . . , Y ik for all individuals i = 1, . . . , n
¦ observe variables Y i,k+1 , . . . , Y ip only for subsample
◦ Sampling with nonresponse followup: sample survey with nonresponses
¦ intensive followup effort for all units not feasible
¦ take random sample of all nonresponding units
◦ Randomized experiment with unequal numbers of cases per treatment group:
¦ suppose original design was balanced
¦ missingness mechanism is deterministic, thus data are MAR
◦ Matrix sampling for questionnaire or test items
¦ divide test or questionnaire into sections
¦ groups of sections are administered to subjects in a randomized
fashion
Not Missing at Random (NMAR) Example: Medical treatment of blood pressure
n individuals are treated for high blood pressure
◦ Observations: binary responses
Y ij = blood pressure of ith individual on jth day
◦ Missing observations: (not every individual shows up every day)
R ij = ½ 1 ith individual appears for measurement on jth day 0 otherwise
◦ Statistical model for Y and R:
Y ij
iid ∼ N (µ i , σ 2 ) R ij
iid ∼ Bin ³
1, 1 − 1 + exp(Y exp(Y ij )
ij )
´
à that is, individuals with high blood pressure are less likely to turn up for a measurement
Maximum Likelihood with Missing Data, Mar 30, 2004 - 9 -
Not Missing at Random (NMAR)
Example: Randomly censored data
◦ Objective: Analysis of data (e.g. survival times) T 1 , . . . , T n
iid ∼ f(·|θ)
◦ Survivor function S(t): Probability of surviving beyond time t S(t) = (T 1 > t)
◦ Censoring times associated with T i ’s:
C 1 , . . . , C n iid ∼ g(·|ξ)
(e.g. due to loss to follow up, drop out, or termination of the study)
◦ Observations: (Y 1 , R 1 ), . . . , (Y n , R n ) where Y i = min(T i , C i )
and
R i = ½ 1 if T i ≤ C i
0 otherwise
Maximum Likelihood with Missing Data, Mar 30, 2004 - 10 -
Observed-Data Likelihood
Likelihood of complete observations (Y obs , R) L n (θ, ξ |Y obs , R) = f Y obs ,R (Y obs , R |θ, ξ)
= Z
f Y,R (Y obs , y mis , R |θ, ξ) dy mis
= Z
f R|Y (R |Y obs , y mis |ξ)f Y (Y obs , y mis |θ) dy mis If the missing observations are missing at random (MAR), the first factor does not depend on y mis and thus is not affected by integration over y mis :
L n (θ, ξ |Y obs , R) = Z
f R|Y (R |Y obs |ξ)f Y (Y obs , y mis |θ) dy mis
= f R|Y (R |Y obs |ξ) Z
f Y (Y obs , y mis |θ) dy mis
= f R |Y (R |Y obs |ξ)f Y obs (Y obs |θ)
= f R |Y (R |Y obs |ξ)L n (θ |Y obs )
If the parameters ξ do not depend on θ, we get θ = argmax ˆ
θ ∈Θ
L n (θ, ξ |Y obs , R) ⇔ ˆθ = argmax
θ ∈Θ
L n (θ |Y obs ),
that is, we can obtain the maximum likelihood estimator by minimizing L n (θ |Y obs ) with respect to θ.
Definition Observed-data likelihood The likelihood function
L n (θ |Y obs ) = f Y obs (Y obs |θ)
is called the likelihood ignoring the missing-data mechanism or short observed-data likelihood.
Ignorability
Definition
A missing-data mechanism is ignorable for likelihood inference if
◦ observations are missing at random (MAR) and
◦ the parameters ξ (missingness-mechanism) and θ θ (data model) are distinct, in the sense that the joint parameter space of (θ, ξ) is the product of the parameter spaces Ξ and Θ.
Example: Non-distinct parameters Suppose that
◦ Y i iid ∼ Bin(1, θ)
◦ R i iid ∼ Bin(1, θ)
◦ Y obs = (Y 1 , . . . , Y m ) T (after reordering) The joint likelihood of Y obs and R is
L n (θ |Y obs , R) = Q n
i=1
θ r i (1 − θ) 1 −r i Q m
i=1
θ y i (1 − θ) 1 −y i
= θ m+Y (1 − θ) n −Y
This is the likelihood of a Bernoulli experiment with n + m observations.
Consequently θ ˆ ML = m + Y
m + n .
Ignoring the missing-data mechanism: The observed-data likelihood is L n (θ|Y obs ) =
m
Q
i=1
θ y i (1 − θ) 1 −y i which leads to the ML estimator
θ ˆ ML = Y
m .
Ignorability
The first condition (MAR) is typically regarded as the more important condition:
◦ If MAR does not hold, then the maximum likelihood estimator based on the observed-data likelihood can be seriously biased.
◦ If the data are MAR but distinctness does not hold, inference based on the observed-data likelihood L n (θ |Y obs ) is still valid from the frequentist perspective, but not fully efficient.
Example:
Suppose
◦ Y i
iid ∼ N (µ, σ 2 )
◦ R i
iid ∼ Bin ¡1, e µ /(1 + e µ ) ¢ Ã parameter are not distinct
Likelihood
µ
−0.4 −0.2 0.0 0.2 0.4
Ln