PLANNED MISSING DATA DESIGNS

(1)

SRCD

Developmental

Methodology

Conference

Feb 9 2012

PLANNED MISSING

DATA DESIGNS

(2)



In planned missing data designs, participants are

randomly assigned to conditions in which they do

not respond to all items, all measures, and/or all

measurement occasions



Why would you want to do this?

1. Long assessments can reduce data quality

2. Repeated assessments can induce practice effects

3. Collecting data can be time- and cost-intensive

4. Less taxing assessments may reduce unplanned

missingness

(3)



Cross-Sectional Designs



Multiple Matrix Sampling



Three-Form Design (and Variations)



Two-Method Measurement



Longitudinal Designs



Developmental Time-Lag



Wave- to Age-based designs



Monotonic Sample Reduction



Growth-Curve Planned Missing

PLANNED MISSING DATA DESIGNS

(4)



Shoemaker (1971)

MULTIPLE MATRIX SAMPLING

Test Items 1 2 3 4 5… K P ar ti ci pant s 1 1 0 1 1 0 2 0 1 1 1 1 3 1 0 1 1 0 4 0 0 0 0 1 …

(5)

MULTIPLE MATRIX SAMPLING

5 Test Items 1 2 3 4 5… K P ar ti ci pant s 1 1 0 1 1 0 2 0 1 1 1 1 3 1 0 1 1 0 4 0 0 0 0 1 … N

(6)

MULTIPLE MATRIX SAMPLING

Test Items 1 2 3 4 5… K P ar ti ci pant s 1 1 0 1 1 0 2 0 1 1 1 1 3 1 0 1 1 0 4 0 0 0 0 1 …

(7)



Assumptions



The K items are a random sample from a population

of items (just as N participants are a random sample

from a population)



Limitations



Properties of individual items or relations between

items are not of interest



Questions?

MULTIPLE MATRIX SAMPLING

(8)

THREE-FORM DESIGN

Form

Common Set X Variable Set A Variable Set B Variable Set C

1 ¼ of items

¼ of items

missing

2 ¼ of items

¼ of items

missing

¼ of items

3 ¼ of items

missing

¼ of items

 Graham et al. (2006)

 Raghunathan & Grizzle (1995) “split questionnaire design”

 Wacholder et al. (1994) “par tial questionnaire design”

(9)



What goes in the Common Set?

THREE-FORM DESIGN

Form

Common Set X Variable Set A Variable Set B Variable Set C

1 ¼ of items

¼ of items

missing

2 ¼ of items

¼ of items

missing

¼ of items

3 ¼ of items

missing

¼ of items

(10)



What goes in the split sets?

THREE-FORM DESIGN

Form

Common Set X Variable Set A Variable Set B Variable Set C

1 ¼ of items

¼ of items

missing

2 ¼ of items

¼ of items

missing

¼ of items

(11)

THREE-FORM DESIGN: EXAMPLE

11

Subtest Item

Demographics How old are you?

Are you male or female? What is your occupation?

Musical Taste What is your favorite genre

of music?

Do you like to listen to music while you work? Do you prefer music played

loud or softly?

Openness I have a rich vocabulary.

I have excellent ideas. I have a vivid imagination.

Subtest Item

Extraversion I start conversations.

I am the life of the party. I am comfortable around

people.

Neuroticism I get stressed out easily.

I get irritated easily. I have frequent mood

swings.

Conscientiousness I am always prepared.

I like order.

I pay attention to details.

Agreeableness I am interested in people.

I have a soft heart.

I take time out for others.

(12)

THREE-FORM DESIGN: EXAMPLE

Subtest Item

of music?

loud or softly?

Subtest Item

people.

swings.

I like order.

(13)

THREE-FORM DESIGN: EXAMPLE

13

Subtest Item

of music?

loud or softly?

Subtest Item

people.

swings.

I like order.

(14)

THREE-FORM DESIGN: EXAMPLE

Subtest Item

of music?

loud or softly?

Subtest Item

people.

swings.

I like order.



Set A

I have a rich vocabulary.

I start conversations.

I get stressed out easily.

I am always prepared.

(15)

THREE-FORM DESIGN: EXAMPLE

15

Subtest Item

of music?

loud or softly?

Subtest Item

people.

swings.

I like order.



Set B

I have excellent ideas.

I am the life of the party.

I get irritated easily.

I like order.

(16)

THREE-FORM DESIGN: EXAMPLE

Subtest Item

of music?

loud or softly?

Subtest Item

people.

I get irritated easily. I have frequent mood swings.

I like order.



Set C

I have a vivid imagination.

I am comfortable around people.

I have frequent mood swings.

(17)

17

Form 1 (XAB) Form 2 (XAC) Form 3 (XBC)

How old are you?

Are you male or female? What is your occupation? What is your favorite genre of

music?

Do you like to listen to music while you work?

Do you prefer music played loud or softly?

What is your favorite genre of music?

I have a rich vocabulary. I have excellent ideas.

I have a rich vocabulary. I have a vivid imagination.

I have excellent ideas. I have a vivid imagination. I start conversations.

I start conversations.

I am comfortable around people.

I am comfortable around people. I get stressed out easily.

I get stressed out easily.

I have frequent mood swings.

I have frequent mood swings. I am always prepared.

I like order.

I am always prepared. I pay attention to details.

I like order.

I pay attention to details. I am interested in people.

I am interested in people. I take time out for others.

(18)

Pa rtic ipa nt Fo rm A ge _Sex O cc up ati on G en re W or k M usi c Vo lum e O pe n1 O pe n2 O pe n3 Ex tr a1 Ex tr a2 Ex tr a3 N eu ro 1 N eu ro 2 N eu ro 3 C on sc 1 C on sc 2 C on sc 3 A gree 1 A gree 2 A gree 3

1 1 17 F professor Classical N loud 4 4 -- 1 5 -- 1 2 -- 4 2 -- 3 2 --2 1 12 F musician Funk N soft 1 3 -- 2 2 -- 5 3 -- 4 1 -- 2 1 --3 1 17 M student Jazz N soft 2 4 -- 5 5 -- 2 4 -- 5 1 -- 4 2 --4 1 29 M server Metal N soft 1 3 -- 5 2 -- 2 1 -- 1 1 -- 4 2 --5 1 17 M chef Rock N soft 1 4 -- 5 1 -- 2 2 -- 5 3 -- 2 2 --6 2 11 F painter Pop Y loud 4 -- 4 2 -- 1 1 -- 5 1 -- 5 5 -- 3 7 2 19 F librarian Alt N loud 1 -- 4 4 -- 3 4 -- 3 4 -- 2 4 -- 3 8 2 22 F server Ska N soft 4 -- 2 3 -- 3 3 -- 3 1 -- 2 5 -- 5 9 2 18 M doctor Punk N loud 1 -- 3 2 -- 2 2 -- 4 4 -- 1 3 -- 2 10 2 19 F statistician Pop N loud 4 -- 5 3 -- 4 5 -- 4 3 -- 2 3 -- 1 11 3 28 F chef Rock Y loud -- 3 3 -- 5 5 -- 5 4 -- 3 3 -- 2 5 12 3 25 M nurse Rock N soft -- 4 5 -- 2 2 -- 2 5 -- 4 5 -- 3 5 13 3 19 M lawyer Jazz Y soft -- 3 4 -- 3 2 -- 4 5 -- 4 5 -- 1 2 14 3 18 F accountant Metal N soft -- 3 1 -- 1 2 -- 3 3 -- 4 4 -- 5 4 15 3 21 F secretary Alt N loud -- 4 4 -- 1 2 -- 1 1 -- 5 3 -- 4 5

(19)

19 Pa rtic ipa nt Fo rm A ge _Sex O cc up ati on G en re W or k M usi c Vo lum e O pe n1 O pe n2 O pe n3 Ex tr a1 Ex tr a2 Ex tr a3 N eu ro 1 N eu ro 2 N eu ro 3 C on sc 1 C on sc 2 C on sc 3 A gree 1 A gree 2 A gree 3

(20)

(21)

21 Pa rtic ipa nt Fo rm A ge _Sex O cc up ati on G en re W or k M usi c Vo lum e O pe n1 O pe n2 O pe n3 Ex tr a1 Ex tr a2 Ex tr a3 N eu ro 1 N eu ro 2 N eu ro 3 C on sc 1 C on sc 2 C on sc 3 A gree 1 A gree 2 A gree 3

(22)

1 1 17 F professor Classical N loud 4 4 4 1 5 3- 1 2 2 4 2 1 3 2 5 2 1 12 F musician Funk N soft 1 3 -- 2 2 -- 5 3 -- 4 1 -- 2 1 --3 1 17 M student Jazz N soft 2 4 -- 5 5 -- 2 4 -- 5 1 -- 4 2 --4 1 29 M server Metal N soft 1 3 -- 5 2 -- 2 1 -- 1 1 -- 4 2 --5 1 17 M chef Rock N soft 1 4 -- 5 1 -- 2 2 -- 5 3 -- 2 2 --6 2 11 F painter Pop Y loud 4 -- 4 2 -- 1 1 -- 5 1 -- 5 5 -- 3 7 2 19 F librarian Alt N loud 1 -- 4 4 -- 3 4 -- 3 4 -- 2 4 -- 3 8 2 22 F server Ska N soft 4 -- 2 3 -- 3 3 -- 3 1 -- 2 5 -- 5 9 2 18 M doctor Punk N loud 1 -- 3 2 -- 2 2 -- 4 4 -- 1 3 -- 2 10 2 19 F statistician Pop N loud 4 -- 5 3 -- 4 5 -- 4 3 -- 2 3 -- 1 11 3 28 F chef Rock Y loud -- 3 3 -- 5 5 -- 5 4 -- 3 3 -- 2 5 12 3 25 M nurse Rock N soft -- 4 5 -- 2 2 -- 2 5 -- 4 5 -- 3 5 13 3 19 M lawyer Jazz Y soft -- 3 4 -- 3 2 -- 4 5 -- 4 5 -- 1 2 14 3 18 F accountant Metal N soft -- 3 1 -- 1 2 -- 3 3 -- 4 4 -- 5 4 15 3 21 F secretary Alt N loud -- 4 4 -- 1 2 -- 1 1 -- 5 3 -- 4 5

(23)

THREE-FORM DESIGN



Things to consider



number of forms



order of items

(24)



Summary



Shorter questionnaire = higher-quality data



Shorter questionnaire = less unplanned missing



High correlations between item sets = high

efficiency relative to a complete data design



Questions?

(25)

TWO-METHOD MEASUREMENT



Measure 1



Gold standard– highly valid (unbiased) measure

of the construct under investigation



Problem: Measure 1 is time-consuming and/or

costly to collect, so it is not feasible to collect

from a large sample



Measure 2



Practical– inexpensive and/or quick to collect on

a large sample



Problem: Measure 2 is systematically biased so

(26)



e.g., measuring stress



Measure 1 = collect spit samples, measure cortisol



Measure 2 = survey querying stressful thoughts



e.g., measuring intelligence



Measure 1 = WAIS IQ scale



Measure 2 = multiple choice IQ test



e.g., measuring smoking



Measure 1 = carbon monoxide measure



Measure 2 = self-report

(27)



How it works



ALL participants receive Measure 2 (the cheap

one)



A subset of participants

also

receive Measure

1 (the gold standard)



Using both measures (on a subset of

participants) enables us to estimate and

remove the bias from the inexpensive

measure (for all participants) using a latent

variable model

TWO-METHOD MEASUREMENT

(28)



Example



Does child’s level of classroom attention in

Grade 1 predict math ability in Grade 3?



Attention Measures



1) Direct Classroom Assessment (2 items, N

= 60)



2) Teacher Report (2 items, N = 200)



Math Ability Measure, 1 item (test score, N =

200)

(29)

Teacher Report 1 (N = 200) Teacher Report 2 (N = 200)

1

2 bias



Direct Assessment 1 (N = 60) Direct Assessment 2 (N = 60)

Attention

(Grade 1)

Teacher

Bias

1

1 TR



_TR₂



_DA₁



_DA₂



Math Score

(Grade 3)

Math Score (Grade 3) (N = 200)

1

29

(30)

1

2



Attention

(Grade 1)

Teacher

Bias

1

1 TR



_TR₂



_DA₁



_DA₂



Math Score

(Grade 3)

1

(31)

1

2 bias



Attention

(Grade 1)

Teacher

Bias

1

1 TR



_TR₂



_DA₁



_DA₂



Math Score

(Grade 3)

1

31

(32)

1

2



Attention

(Grade 1)

Teacher

Bias

1

1 TR



_TR₂



_DA₁



_DA₂



Math Score

(Grade 3)

1

(33)

1

2 bias



Attention

(Grade 1)

Teacher

Bias

1

1 TR



_TR₂



_DA₁



_DA₂



Math Score

(Grade 3)

1

33

(34)

1

2



Attention

(Grade 1)

Teacher

Bias

1

1 TR



_TR₂



_DA₁



_DA₂



Math Score

(Grade 3)

1

(35)



Assumptions:



expensive measure is unbiased (i.e., valid)



inexpensive measure is systematically biased



both measures access

the same construct

TWO-METHOD MEASUREMENT

(36)



Holding cost

constant, as

N

_total

increases,

N

_expensive

decreases



As

N

_total

increases,

SEs begin to decrease

(power increases); as

N

_total

continues to

increase, SEs

increase again

TWO-METHOD MEASUREMENT

36

(37)



Find the sweet spot!

TWO-METHOD MEASUREMENT

37

true-score

reliability

(expensive)

true-score

reliability

(cheap)

bias

.25

cheap only

.49

.25

cheap only

.25

.49

cheap only

.49

cheap only

.49

.25

neither

(38)



Summary



Rather than face a choice between high

power/low validity vs. low power/high validity,

2-method measurement can result in high

power/high validity



This design is only possible when there are

two suitable measures for the construct of

interest



Questions?

(39)



Use 2-time point data with variable time-lags

to measure a growth trajectory + practice

effects (McArdle & Woodcock, 1997)

DEVELOPMENTAL TIME-LAG MODEL

(40)

T1 T2

Age

1

student

2 3 4 5 6 7 8 0 1 2

Time

3 4 5 6 5;6 5;3 4;9 4;6 4;11 5;7 5;2 5;4 5;7 5;8 4;11 5;0 5;4 5;10 5;3 5;8

(41)

T0

T1

T2

T3

T4

T5

T6

(42)

T0

T1

1 T2

T3

T4

T5

T6

Intercept

1

1 t

t

Y



I



B G



A P

(43)

T0

T1

1 T2

T3

T4

T5

T6

43

1 Intercept

1

2 _{3 4}

5

6 Growth

0

1 t

t

Y



I



B G



A P

Linear growth

(44)

T0

T1

1 T2

T3

T4

T5

T6

1

1 Intercept

1

2 _{3 4}

5

6 ₁

1

1 Practice

0 1

₁

Growth

0

1 t

t

Y



I



B G



A P

(45)

45

T0

Y



I

T1

Y

  

I

G

P

2 T2

Y

 

I

G



P

3 T3

Y

 

I

G



P

(46)

T0

T1

1 T2

T3

T4

T5

T6

1

1 Intercept

1

2 _{3 4}

5

6 Practice

Growth

0 Linear growth; Exponential practice decline

.67

.55

.45

.37

0 1 .82

(47)



Summary



2 measured time points are formatted according to

time-lag



This formatting allows a growth-curve to be fit,

measuring growth and practice effects



Questions?

DEVELOPMENTAL TIME-LAG MODEL

(48)



The idea of reformatting data to answer a

different question is not limited to time-lag

designs



Wave-based data collection (e.g., data

collected at Grade 1-3) can be transformed

into age-based data with missingness

(49)

49 K 1 2

grade

1

student

2 3 4 5 6 7 8 4;6- 4;11 5;0- 5;5 5;6- 5;11

age

6;0- 6;5 6;6- 6;11 7;0- 7;5 7;6- 7;11 5;6 5;3 4;9 4;6 4;11 5;7 5;2 5;4 6;7 6;0 5;11 5;5 5;9 6;7 6;1 6;5 7;4 6;10 7;3 6;10 7;5 6;4 7;3 7;6

(50)

4;6- 4;11 5;0- 5;5 5;6- 5;11

age

6;0- 6;5 6;6- 6;11 7;0- 7;5 7;6- 7;11 5;6 5;3 4;9 4;6 4;11 5;7 5;2 5;4 6;7 6;0 5;11 5;5 5;9 6;7 6;1 6;5 7;4 6;10 7;3 6;10 7;5 6;4 7;3 7;6



Out of 3 waves,

create 7 waves of

data with high

missingness



Allows for more

fine-tuned

age-specific growth

modeling



Even high

amounts of

missing data are

not typically a

problem for

estimation

(51)



Sometimes used in large datasets (e.g., Early

Childhood Longitudinal Study) to reduce costs



At each wave, a randomly -selected subgroup of the

original sample is observed again



The remainder of the original participants do not

need to be kept track of, dramatically reducing costs

MONOTONIC SAMPLE REDUCTION

51

Group

Time 1

Time 2

Time 3

Time 4

Time 5

1 x

x

2 x

x

--

3 x

x

--

4 x

x

--

(52)



Advantages:



Cost reduction



A lot of power to estimate effects at earlier waves



Disadvantages:



Very little power to estimate effects dependent on

the last wave of data, e.g., growth curve models (may

be missing 80% of data)



It is important to be able to estimate attrition rates

before beginning data collection

(53)

GROWTH-CURVE PLANNED MISSING



With a particular analysis in mind,

missingness may be tailored to maximize

power



In growth-curve designs, the most important

parameters are the growth parameters (e.g.,

estimate the steepness and the shape of the curve)



Estimation precision depends heavily on the first and

last time points



A planned missing design can take advantage of this

by putting missingness in the middle

(54)

GROWTH-CURVE PLANNED MISSING

Group

Time 1

Time 2

Time 3

Time 4

Time 5

1 x

x

2 x

x

--

3 x

x

--

x

4 x

x

--

x

5 x

--

x

6 --

x



Participants are randomly assigned to be missing

one measurement occasion (Graham, Taylor &

Cumsille, 2001)

(55)

GROWTH-CURVE MODELS

Group

Time 1

Time 2

Time 3

Time 4

Time 5

1 x

x

2 x

x

--

3 x

x

--

x

--

4 x

--

x

--

5 --

x

--

6 x

x

--

x

7 x

--

x

--

x

8 --

x

--

x

9 x

--

x

10 --

x

--

x

11 --

--

x

55

(56)

GROWTH-CURVE MODELS

Group

Time 1

Time 2

Time 3

Time 4

Time 5

1 x

x

2 x

x

--

3 x

x

--

x

--

4 x

--

x

--

5 --

x

--

6 x

x

--

x

7 x

--

x

--

x

8 --

x

--

x

9 x

--

x

10 --

x

--

x

11 --

--

x

(57)

(58)



Purposeful missing data can address several issue in

study design



Cost of data collection



Participant burden/fatigue



Practice effects



Participant dropout



Rearranging data can turn one complete design into

a more nuanced missing data design



Developmental time-lag designs



Wave-missing into age-missing

(59)