Questionnaire Design in Telephone Surveys: Interviewers and Research Call Center Managers Experience

Questionnaire Design in Telephone

Surveys: Interviewers’ and Research

Call Center Managers’ Experience

Wojciech Jablonski

University of Lodz | Poland

Introduction

Method

Results

Conclusions

Introduction

Method

Results

Conclusions

There is no visual contact between the interviewer and

the respondent – its has an impact on the quality of

the obtained data

CATI CHARACTERISTICS

• satisficing

• CATI interviewers are much more constrained than

f2f interviewers as far as using visual aids is

concerned

-CATI CHARACTERISTICS

SPECIAL TECHNIQUES

keyword summary

unfolding

split questions

branching

end-point only scales

/

with midpoint

RESEARCH QUESTION

What are the CATI interviewers’ and call center

managers’ opinions and attitudes towards different

response scales formats used by survey organizations?

We consider the interviewers’ opinions of an interview to

be reliable indicators of the quality of the conversation

Introduction

Method

Results

Conclusion

• Data collection:

2009–2010

&

2013 (follow-up)

•

12 major Polish survey research organizations

•

V

•

846 interviewer questionnaires

&

32 IDIs with well-experienced

interviewers

+

IDIs with CATI fieldwork managers

Introduction

Method

Results

Conclusion

The job of a telephone interviewer, as any other job, has both disadvantages and advantages. Below there are several features of CATI interviewer’s job listed. For each of them, say to what extent you consider it an advantage or a disadvantage.

Use the scale of 1 to 5 shown below, where 1 means this feature is a disadvantage and 5 means that it is an advantage. The other values on the scale may be used to

express opinions in-between the two extremes.

Communication with the respondent is verbal only – there is no

eye contact

14% 35% 48%

RESULTS

ADVANTAGES/DISADVANTAGES

Below is a list of different elements, which make up the general training organized for interviewer candidates. For each of them, state to what extent, in your opinion, is

it useful for participants. Use the scale of 1 to 5 shown below, where 1 means this issue is completely redundant (and so there is no point in discussing it) and 5 means

it is very useful (and so discussing it is necessary). The other values on the scale may be used to express opinions in-between the two extremes.

Information about the types of questions used in CATI scripts

5% 16% 78%

RESULTS

GENERAL TRAINING

RESULTS

GENERAL TRAINING

ELEMENT NECESSARY

(points 4-5)

Information about the types of questions used in CATI

scripts 78%

Information about the research company (outline of the

history, scope of operation) 47%

Information about the ESOMAR code regulations 55%

Information about the Data Protection Act 65%

Information about the methods of sampling in telephone

research 66%

Information about the methods of selecting respondent within

Situation considered to be particularly troublesome:

handling respondents who have difficulty in

understanding the questions

According to the interviewers…

…the questions used in CATI scripts are often formulated

using complicated vocabulary and syntax. The questions

have not been adapted to suit the intellectual skills of an

average respondent.

RESULTS

RESULTS

PROBLEMS

Sometimes questions are so long that when I finish reading them I don’t understand what

the beginning is. The respondent – without written text – all the more.

Some questions are just too clever. Even when I look through them I have to really

get into them. (…) The questions are not designed for ordinary people but for the

ones better educated.

Imagine a question in which you have 39 items, an 11 point scale and you must read it 4 times – for different companies. It’s a nightmare. When you conduct such interviews, you can notice a moment when the respondents let their minds wander,

A majority of interviewers claim that they sometimes

deviate from interviewing rules

•

Unfolding questions with semantic (fully-labeled) scales

•

Transforming fully-labeled scales into end-point only

scales

•

Transforming check-all-that-apply questions into

item-based yes-or-no questions

RESULTS

CATI call center managers partially share interviewers’ opinion

and consider CATI script design to be one of the major factors

influencing the difficulty of the interview.

Reasons:

•

Sometimes survey organization is responsible only for

fieldwork – has no influence on the content of the

questionnaire design

•

Pretesting questionnaires for more understandability is seldom

used in market research + There are researchers who only have

hazy idea of the reality of fieldwork in (CATI) research

•

No elements of conversational interviewing are used in CATI

scripts

RESULTS

Introduction

Method

Results

Conclusion

Interviewers demonstrate good methodological intuition

– they recognize questions that – according to

professional guidelines – should and should not be used

in CATI scripts

The design of some questions used in CATI scripts do

not follow these regulations. As a result, interactional

problems between the interviewers and the respondents

occur. The interviewers sometimes solve such problems

by destandardizing the interview protocol

• Courser, M.W., Lavrakas, P.J. (2012). Item Nonresponse and the 10-Point Response Scale in Telephone Surveys. Survey Practice 5(4): 1-5.

• de Leeuw, E.D. (2002). Data Quality in Mail, Telephone and Face to Face Surveys.

Amsterdam: TT-Publikaties.

• Dillman, D.A. (1978). Mail and Telephone Surveys. The Total Design Method. New

York: John Wiley & Sons.

• Dillman, D.A., Smyth, J.D., Christian, L.M. (2009). Internet, Mail and Mixed-Mode

Surveys. The Tailored Design Method. Hoboken: John Wiley & Sons.

• Frey, J.H. (1989). Survey Research by Telephone. Newbury Park: Sage Publications.

• Groves, R.M. (1979). Actors and Questions in Telephone and Personal Interview

Surveys. Public Opinion Quarterly 43(2): 190-205.

• Krosnick, J.A. (1999). Survey Research. Annual Review of Psychology 50: 537-567.

• Malhotra, N., Krosnick, J.A., Thomas, R.K. (2009). Optimal Design of Branching

Questions to Measure Bipolar Constructs. Public Opinion Quarterly 73(2): 304-324.

THANK YOU!

Wojciech Jablonski