Population Research Seminar Series

Jack Fowler, PhD

Center for Survey Research UMass Boston

Population Research Seminar Series

Session 4: Standardized Survey Interviewing

Part 1: Controlling Interviewer Related Error

What Interviewers Are Supposed to Do in Order to Be Consistent (Standardized)

• Read questions as worded

• Use nondirective probes when questions are not answered adequately

– Among other things, those probes should not

materially change the content of the questions that other respondents are answering so that all

respondents are answering as close to the same question as possible

Directive Probes

Any probe that can be answered with a ―yes‖ or ―no‖ is a directive probe

Why Does It Matter?

• Error 1: The more interviewers innovate (eg. when they probe) the

more they affect the data

• Error 2: Small changes in question wording sometimes have big effects on answers

Why Are There Problems?

• Bad question design

• Inadequate interviewer training

• Inadequate interviewer monitoring • Interviewer fatigue

Question Design and Wording

• Complex terms and concepts • Multi-barrelled questions

Three Barrels

• This question is about automobile injuries, including injuries from crashes, burns, and any other kind of accidents. Were you ever injured in a car accident because of your driving?

Poor Sequencing

• Have you been the victim of a crime in the last 12 months?

Poor Sequencing II

• Do you do any kind of activity for exercise on a regular basis?

• What are the different kinds of things you do for exercise?

Poor Sequencing III

• Including any days spent in a hospital, between

[START DATE] and [END DATE], did illness or injury keep

[PERSON] from work more than half of the day, not

counting work around the house?

• What were the dates that [PERSON] missed more

than half a day from work between [START DATE] and

[END DATE]?

• How many actual days was that between [FIRST DATE] and [END DATE]?

Unclear How to Respond

• When did you first vote in a Presidential election?

– While I was in college – During the Vietnam war – About 10 years ago

– In 1972

• In what year did you first vote in a Presidential election?

Unclear How to Respond II

• What kind of museums have you visited in the past year? *(field coded)

– Art museums

– Science museums – History museums – Other

Unclear How to Respond III

• How did you get to the game yesterday?

– I drove

– I got a ride with friends – By car

– We circled around on Elm Street to avoid the traffic

– I stayed up the night before to get my homework done

Cognitive Testing

• Identify ambiguous terms and define them • Identify complex questions and break

them up

• Identify questions for which it is not clear how to answer and build that information into the question

Behavior Coding

• During pretests, identify questions

– That interviewers have trouble reading as written

– That respondents have difficulty answering without additional probing

– For which respondents frequently ask for clarification

Intraclass correlation

• This is a measure of how much

interviewers affect the answers they obtain.

• Obviously, the ideal would be if the value is 0.

Biggest correlates of

interviewer-related error

• The rate at which questions have to be probed, which in turn is related to:

– Lack of clarity of what the question is asking – Lack of clarity about how to answer the

Train Interviewers I

% Who read questions acceptably by length of training

Training Days Read Acceptably

< 1 40%

2 62%

5 79%

Train Interviewers II

Percent who probe open ended questions acceptably

Training Days Probe Acceptably

< 1 13%

2 31%

5 29%

Train Interviewers III

• Percent who probe closed questions acceptably

Training Days Probe Acceptably

< 1 48%

2 67%

5 72%

Why Monitoring Matters

• Compared interviewers who were tape recorded with those whose interviewing was not reviewed by a supervisor

Interviewer Effects On Answers

Monitoring and Bias

What Are We Monitoring For?

• Reading questions • Good probing

• The way interviewers interact with respondents

Interviewers Set Standards

For Respondents

• The way an interviewer behaves has an important effect on what respondents do and how well they perform

Experience Can Be A Negative

(NSDUH)

• Reporting illicit drug use in lifetime

Percent “Yes”

Experienced Interviewer

33%

Experience Can Be A Negative II

• Percentage of known visits to doctors reported by week of data collection

Interviewer Week Percent NOT Reported 1 18% 2 21% 3 23% 4 27% 5 29%

Interviewers Communicate

Expectations

Interviewer concern about efficiency

Respondent said interviewer wanted Exact Answers General Ideas TOTAL Low 77% 23% 100% Medium 62% 38% 100% High 66% 34% 100%

Interviewers Communicate

Expectations II

Interviewer Expectations Affect

Data

• Please take your time and think carefully about your answers. It is important that you answers be as accurate as possible

Reported Media Use by Interview

Procedure (Miller)

Experimental Control

Watched TV previous day 86% 66%

Listened to radio previous day 67% 65%

Read newspaper previous day

83% 77%

Seen an X-rated movie 61% 51%

Mean number of books read

Monitoring Matters

• To show we care about reading as worded and probing

• To help experienced interviewers not get sloppy and tired

• To make sure interviewers are

communicating that interviewing is a serious data collection effort

How about recruitment?

• Other than the obvious skills—following directions, understandable speaking,

computer skills as needed—there are essentially no guidelines for selecting good interviewers

Matching

• If the obvious characteristics of

interviewers are related to answers, may see an effect

• For most questions, demographic

characteristics unrelated to answers if interviewers are well trained

How to minimize

interviewer-related error

1. Write good questions and test them 2. Train the interviewers in good

standardized practices

3. Script interviewers to emphasize data quality

4. Monitor how the data collection process is carried out