Interviewer Training Guidelines of
Multinational Survey Programs: A Total
Survey Error Perspective
Daniela Ackermann-Piek
1,
Henning Silber
1,
Jessica
Daikeler
1,
Silke Martin
1& Brad Edwards
21
GESIS – Leibniz Institute for the Social Sciences, Germany
2
Westat, Rockville, Maryland, USA
Abstract
Typically, interviewer training is implemented in order to minimize interviewer effects and ensure that interviewers are well prepared to administer the survey. Leading professional associations in the survey research landscape recommend the standardized implementation of interviewer training. Some large-scale multinational survey programs have produced their own training guidelines to ensure a comparable level of quality in the implementation of training across participating countries. However, the length, content, and methodology of interviewer training guidelines are very heterogeneous. In this paper, we provide a com-parative overview of general and study-specific interviewer training guidelines of three multinational survey programs (ESS, PIAAC, SHARE). Using total survey error (TSE) as a conceptual framework, we map the general and study-specific training guidelines of the three multinational survey programs to components of the TSE to determine how they target the reduction of interviewer effects. Our results reveal that unit nonresponse error is covered by all guidelines; measurement error is covered by most guidelines; and cover-age error, sampling error, and processing error are addressed either not at all or sparsely. We conclude, for example, that these guidelines could be an excellent starting point for new – small as well as large-scale – surveys to design their interviewer training, and that interviewer training guidelines should be made publicly available in order to provide a high level of transparency, thus enabling survey programs to learn from each other.
Keywords: interviewer training guidelines, interviewer effects, multinational survey pro-grams, total survey error
Acknowledgments
The authors thank the participants of the workshop „Dashboard & Field Monitoring“ held in Mannheim, 2018 for their thoughts which encouraged this research idea. We are grateful to Sabine Friedel and Verena Halbherr for helpful details about interviewer training in the ESS and SHARE and detailed cross-checks of the manuscript.
Direct correspondence to
Daniela Ackermann-Piek, GESIS – Leibniz Institute for the Social Sciences, Quadrat B2 1, 68159 Mannheim, Germany
E-mail: [email protected]
Concerns about interviewer effects in interviewer-mediated surveys have accompa-nied generations of survey researchers. Most of the literature on interviewer effects focuses on the description and explanation of these effects after data collection (West & Blom, 2017). However, in order to ensure that interviewer-administered surveys produce high-quality data, it is essential that measures be taken to prevent, or minimize, interviewer effects. One such measure is the implementation of stan-dardized interviewer training. In addition, interviewer training is typically used to ensure that interviewers are well prepared to adequately perform all the tasks they have during the survey implementation.
Interview Survey (NHIS). More specifically, in line with the general guidelines issued by leading professional associations in the area of survey research, the inter-viewer training concepts of national and multinational survey programs include recommendations for study-specific training for inexperienced and experienced interviewers, as well as brief sections on general interviewer training for inexperi-enced interviewers.
However, what is lacking in the literature is a structured comparison of the content of the various interviewer training guidelines of survey programs or train-ing concepts of large surveys – ustrain-ing a theoretical framework. The present article aims to fill this gap by providing a comparative overview of the extent to which the content of training guidelines of the ESS, PIAAC, and SHARE (Börsch-Supan & Jürgens, 2005; ESS, 2016a, 2016b, Malter & Börsch-Supan, 2017; Organziation for Economic Co-operation and Development [OECD], 2011, 2013) are integrated into the conceptual total survey error framework (TSE; Biemer, 2010; Groves et al., 2009; Pennell, Hibben, Lyberg, Mohler, & Worku, 2017; T. W. Smith, 2019). Additionally, we investigate the extent to which the individual components of the TSE are addressed in these guidelines, and we make suggestions for improvements. Specifically, we focus on the topics of interviewer training specified in training guidelines on an international level rather than on interviewer training in specific countries with detailed training content.
The TSE Framework and the Literature on
Interviewer Training
The TSE is a theoretical concept that describes statistical error properties of survey estimates, systematically structured along different error sources (Biemer, 2010; Groves & Lyberg, 2010). Error sources are assigned to each step in the survey life cycle, typically along two dimensions each, either sampling error and nonsampling error (Biemer, 2010) or measurement and representativeness (Groves et al., 2009). As our aim is to compare interviewer training of multinational survey programs along the TSE, we follow the approach of Pennell et al. (2017), who adopted the TSE typology for multinational, multiregional, and multicultural surveys (3MC). Pennell et al. (2017) provide a TSE model which combines the complexity in designing and implementing 3MC surveys with the overall aim to minimize com-parison error (T. W. Smith, 2011). Following the approach of Groves et al. (2009), the authors structure their model along the two dimensions measurement and rep-resentativeness.
Following Groves et al. (2009) and Pennell et al. (2017), the representation dimension of the TSE includes coverage error, sampling error, nonresponse
measurement error, and processing error. Coverage error refers to problems of a not perfectly covered target population in the sampling frame. Sampling error
occurs because only a sample is observed instead of the entire target population. With regard to sampling error, either bias (members of the sampling frame are systematically excluded from selection) or variance (different sets of sample frame elements are selected and each set can have different values in the survey statistic) can occur. Nonresponse error occurs when selected sample members that respond the survey request systematically differ from those who do not respond the survey request. After data collection, post-survey adjustments are typically used to cor-rect for representation errors occurred earlier in the process. However, when post-survey adjustments fail to capture each case of misrepresentation in the sample,
adjustment error occurs. The error components of the measurement dimension
of the TSE are associated with errors in the survey instruments and the question-response process. The first error component of the measurement dimension of the TSE, validity, reflects an error that describes that the theoretical construct is not optimally reflected in the measure. The next error component, measurement error,
occurs when the response given by a respondent differs from the true response. Finally, processing error reflects the incorrect transfer of responses to data stor-age during capturing, coding, or editing of data. All errors described in the TSE may result in biased survey estimates of substantive survey variables. Thus, the aim of survey operations is to minimize the errors under the given time and cost constraints to maximize the survey quality (Schouten, Peytchev, & Wagner, 2017).
The TSE framework is regularly used to describe and structure interviewer errors in interviewer-administered surveys (for an overview, see West & Blom, 2017). As interviewers have many tasks when administering a survey, such as con-tacting sample units, gaining their cooperation, asking questions, and recording answers (e.g., Groves et al., 1992; Loosveldt, 2008; Schaeffer, Dykema, & May-nard, 2010), they can – intentionally or unintentionally – affect a large number of steps in the survey process. In other words, they can be the sources of multiple sur-vey errors. Because interviewer training is organized along interviewer tasks, the TSE framework can be used to structure training content when reviewing general and study-specific interviewer training concepts of multinational survey programs.
on several quality indicators related to measurement error, for example, have iden-tified a positive effect of interviewer training on the application of standardized interviewing techniques (Billiet & Loosveldt, 1988; Dahlhamer, Cynamon, Gentle-man, Piani, & Weiler, 2010; Fowler Jr., 1991), the reduction of item nonresponse (Billiet & Loosveldt, 1988; Daikeler & Bosnjak, forthcoming), and the application of appropriate probing techniques (Billiet & Loosveldt, 1988; Daikeler & Bosn-jak, forthcoming). However, Groves (2005) noted that the literature left open the question whether interviewer training effectively reduces measurement error. And finally, the effects of interviewer training on coverage error, sampling error, and processing error have been addressed only occasionally in the literature (Eckman & Kreuter, 2011; Guest, 1954).
Methodology and Resources
The present study compares interviewer training concepts of three large-scale mul-tinational survey programs in the social sciences, namely, the ESS (ESS, 2016a, 2016b, 2018), PIAAC (OECD, 2011, 2013), and SHARE (Börsch-Supan & Jürges, 2005; Malter & Börsch-Supan, 2017). In most cases, only multinational survey pro-grams have the funds to develop detailed interviewer training guidelines and imple-ment interviewer training accordingly. These programs need predefined detailed specifications for participating countries, because such programs are imperative to ensure a harmonized data collection process across countries, which is a prereq-uisite for obtaining high-quality data (Pennell, Harkness, Levenstein, & Quaglia, 2010; Survey Research Center at the University of Michigan, 2016). Ensuring har-monization across countries also applies to the training of interviewers.
Overall, the ESS, PIAAC and SHARE fulfilled the following selection crite-ria: First, all three are administered by interviewers face-to-face. Second, they have participants from many European countries. Third, in all three cases detailed docu-ments were publicly available that contained information on the survey programs’ interviewer training guidelines. Very often this information is confidential and not accessible.
competencies; a cross-sectional “Survey of Adult Skills” is carried out every 10 years. We used the training material from PIAAC Cycle 1, Round 1, 2011/20121.
Training concepts relate to training guidelines on an international level rather than to interviewer training in specific countries. Comparing the implementation of interviewer training in the various participating countries in detail would be another important research question. Also, we focus our research on general as well as survey specific interviewer training content provided by the three multinational survey programs.
Interviewer Tasks within ESS, PIAAC, and SHARE
When comparing interviewer training guidelines of different survey programs, it is important to take interviewers’ tasks in the surveys and the resulting complexity of their roles into account. First of all, this refers to the target population, as these are the persons with whom interviewers interact. The target population of the ESS and PIAAC is quite similar and refer to the general population aged either 15 years or older (ESS) or between 16 and 65 years (PIAAC). The target population in SHARE also refers to the general population, however, only to persons who are 50 years or older at the time of sampling. In addition, in SHARE, spouses or partners of the sampled person are interviewed as well, if applicable.
Another interviewers’ task for all three survey programs was the adminis-tration of the core questionnaire face-to-face using computer-assisted personal interviewing (CAPI). In addition, PIAAC and SHARE interviewers had to perform additional tasks. In PIAAC, interviewers had to administer a cognitive assessment where respondents worked independently on a number of tasks on the interviewer’s laptop or in a paper booklet under the supervision of the interviewer. For this pur-pose, interviewers switched from their traditional role of asking questions and took on a passive, monitoring role, adapting their behavior accordingly. If the respon-dent opted for the paper-based cognitive assessment, the interviewer additionally had to score some items for routing purposes.2 In SHARE, interviewers adminis-tered a self-completion paper questionnaire to the respondents in some countries. As the target population in SHARE consisted of elderly persons, interviewers had to be able to interact with this special population. A special and new task for some SHARE interviewers was to collect biomarkers from respondents and conduct physical tests (e.g., measuring blood pressure). The average interview duration was
1 For a more detailed overview of the specifications and characteristics of the ESS, PIAAC, and SHARE across all participating countries, see the Appendix.
60 minutes for the ESS (Round 8), whereas SHARE and PIAAC had longer average durations (80 minutes and 90 minutes, respectively).
Interviewer Training Concepts of ESS, PIAAC, and
SHARE
The ESS specifications for the countries distinguish between two types of inter-viewer preparation: training and briefing. ESS interinter-viewers are expected to have previous face-to-face interviewing experience and to be trained in effective door-step interaction, standardized interviewing techniques, and general interviewer behavior before administering the survey instrument. In each round of the ESS, experienced interviewers receive a briefing, whereas inexperienced interviewers should additionally undergo general interviewer training prior to the briefing.
With respect to interviewer training in PIAAC, several features can be highlighted: (1) the extensive interviewer training package (including, e.g., fully scripted training sessions); (2) the train-the-trainer session prior to national inter-viewer training in which the training staff is introduced to the scripts and interview materials; (3) the close monitoring by the international consortium of the imple-mentation of the country-based interviewer training. As PIAAC Round 1, Cycle 1 also included a field trial in which all aspects of the survey – including interviewer-related topics – were tested, the interviewer training sessions for the main study were shortened depending on the performance of the interviewers in the field trial.
For SHARE, the survey programs’ multiplier approach to interviewer training can be highlighted: a centralized train-the-trainer program is conducted to facili-tate decentralized interviewer training in the participating countries. Moreover, all interviewers are expected to have extensive general face-to-face interviewing expe-rience and to have received in-person general interviewer training prior to undergo-ing study-specific trainundergo-ing.
Standards for the Implementation of Interviewer Training
within ESS, PIAAC, and SHARE
The extent to which the implementation of interviewer training is specified differs considerably across the three multinational survey programs (for an overview, see Table 1). As a first impression, when counting the number of pages in the over-all survey specifications which are provided to the participating countries for the respective survey3, it becomes obvious that the specifications for the ESS (65 pages)
Ta bl e 1 ES S, P IA AC , a nd S H A RE S ta nd ard s f or t he I m pl eme nt at io n o f I nt er vi ew er T ra in in g a nd D es ire d I nt er vi ew er Ch ar ac te ris tic s ES S 2 01 6 ( Ro un d 8 ) 1 PI A AC ( Cy cle 1 , R ou nd 1 ) 2 SH A RE (W ave 6 ) 3 G en er al c onc ep t In te rv iew er e xp er ie nc e i m po rta nt : G IT f or i ne xp er ie nc ed i nt er vi ew er s an d r ed uc ed t ra in in g f or e xp er ie nc ed in ter vi ew er s Ex te ns iv e m at er ia l p ro vi de d f or sta nd ar di ze d t ra in in g s es sion s f or a ll in ter vi ew er s Tr ai ni ng f or p re te st a nd m ai n s tud y (th e l at te r c an b e r ed uc ed d ep en di ng on e xp er ie nc e a nd p er fo rm anc e i n pr et es t) M ul tip lie r a pp ro ac h: C en tra liz ed T TT pr og ra m t o f ac ili ta te d ec en tra liz ed na tio na l t ra in in g In te rv iew er s e xp ec te d t o b e ex pe rie nc ed Le ng th o f s ur ve y s pe ci fic a-tion s ( nu m be r o f p ag es ) 65 19 9 542 M at er ia l f or t ra in er s NC m an ua l Se rie s o f p re -s tru ct ur ed s lid es w ith m ov ie c lip s a nd r el at ed m at er ia l (inc l. g ui de lin es f or t ra in in g, s cr ip te d pr ac tic e i nt er vie w) Te ch ni ca l S ta nd ar ds a nd G ui de lin es m an ua l f or N PM t ea m s (p lu s c om pr eh en siv e i nt er vi ew er tra in in g m at er ia
l on p
la nn in g a nd im pl em en ta tion , i nc l. P ow er Po in t slid es , t ra in in g s cr ip ts) In te rv iew er p ro je ct m an ua l Fa ci lit at or g ui de ( inc l. P ow er Po in t slid es , t ra in in g s cr ip ts) CD -ba se d t ra in in
g on g
ES S 2 01 6 ( Ro un d 8 ) 1 PI A AC ( Cy cle 1 , R ou nd 1 ) 2 SH A RE (W ave 6 ) 3 Ma ter ia l f or in ter vi ew er s In ter vi ew er m an ua l H om e-stud y m at er ia l, i nc l. w rit te n ex er ci se s In ter vi ew er m an ua l In te rv iew er m an ua l ( inc l. i ns tru ct ion s fo r b lo od c ol le ct ion w
ith a s
ho rt vi de o) Tr ai ni ng f ac ili tie s a nd eq uip m en t U -s ha pe d c ha ir s et up , t ec hn ic al eq uip m en t A de qu at e s pa ce ( se pa ra te r oo m s, ea ch w
ith a l
ES S 2 01 6 ( Ro un d 8 ) 1 PI A AC ( Cy cle 1 , R ou nd 1 ) 2 SH A RE (W ave 6 ) 3 M od e o f t ra in in g In -p er so n In -p er so n In -p er so n Tr ai ne e g ro up s iz e ---15 –2 0 ---Tr ai ni ng le ng th 6 h rs Pr et es t: 3 6 h rs a ll i nt er vi ew er s M ai n s tud y: 1 5 h rs ( pr et es t in te rv iew er s w ith g oo d r ev iew s); 3 0 hr s ( pr et es t i nt er vi ew er s w ith l es s th an f av or ab le r ev iew s o r i nt er vi ew er s w ith e xp er ie nc
e on o
th er s ur ve ys ), 3 6 hr s i ne xp er ie nc ed i nt er vi ew er s TT T pi lo
t: 2 d
ay s TT T p re te st 1 .5 d ay s TT T m ai
n: 1 d
ay s na tion al t ra in in
g: 2 d
ay s Tr ain in g s ch ed ul in g O ne m on th b ef or e fi eld wo rk s ta rts No e ar lie r t ha n t wo w ee ks b ef or e, bu t p re fe ra bl y t he w ee k i m m ed ia te ly be fo re t he s ch ed ul ed s ta rt o f d at a col le ct io n TT T pi lo t: - --TT T p re te st: - --TTT m ai n: -Na tion al t ra in in g: D ec . 2 01
4 – F
ES S 2 01 6 ( Ro un d 8 ) 1 PI A AC ( Cy cle 1 , R ou nd 1 ) 2 SH A RE (W ave 6 ) 3 Q ua lit y c on tro l N C m us t o ver se e i nt er vi ew er se le ct ion a nd t ra in in g a nd m ak e s ur e th at i n-pe rs on t ra in in g t ak es p la ce D oc um en ta tion o f p la n, s ch ed ul in g, m at er ia l, i m pl em en ta tion , a nd ev al ua tion o f n at ion al t ra in in g Tra in in g r epo rt Re tra in in g a nd a ttr iti on t ra in in g du rin g d at a c ol le ct ion ( if a pp lic ab le) Tr ai ne r c er tifi ca tion In ter vi ew er cer tifi ca tio n No te . S ur ve y p ro gr am s i n a lp ha be tic al o rd er ; i nf or m at ion r et rie ve d f ro m g en er al p ro je ct g ui de lin es o r p ro je ct s pe ci fic at ion s ( se e r ef er enc es b elo w) ; CA I = co m pu te r-a ss ist ed in te rv iew in g; CM S = ca se m an ag em en t s ys te m ; E SS = Eu ro pe an So cia l S ur ve y; F2 F = fa ce -to -fa ce ; G IT = ge ne ra l in te rv iew er t ra in in g; N
C = N
at ion al C oo rd in at or ; N
PM = N
at ion al P ro je ct M an ag em en t; P IA
AC = P
ro gr am m e f or t he I nt er na tion al A ss es sm en t of A du lt C om pe te nc ie s; S H A
RE = S
ur ve y o f H ea lth , A ge in g a nd R et ire m en t i n E ur op e; T
TT = t
ra in -th e-tra in er s s es sion ;
--- = n
o i nf or m at ion inc lud ed i n t he s pe ci fic at ion s.
1 E
SS ( 20 16 b) ; E SS ( 20 16 a); E SS ( 20 18 ).
2 O
EC D ( 20 11 ); O EC D ( 20 13 , C ha pt er 1 0. 4) .
3 B
are not as extensive as those for PIAAC (199 pages) and SHARE (542 pages). This is not surprising when one considers the additional, non-standard, tasks that interviewers in PIAAC and SHARE must perform. In addition, experienced ESS interviewers typically only receive a reduced version of the interviewer training (referred to as “interviewer briefing”).
The interviewer training guidelines of all three survey programs specify that the training should be conducted in-person. All guidelines require measures for controlling the quality of the training (e.g., review of interviewer selection and training report by country). In contrast, other topics are not covered by the sur-vey specifications of all three sursur-vey programs. For example, the ESS and PIAAC specify that training should be scheduled to take place shortly before the start of data collection, whereas this is not addressed in the SHARE survey specifications. Moreover, the ESS and SHARE specifications related to interviewer training do not address training of supervisory staff, training-group size, or the structure of train-ing groups, whereas PIAAC defines these aspects clearly. In addition, the SHARE specifications do not address training facilities and equipment, and the ESS speci-fications do not include information on the evaluation of the interviewer training. Other examples are that in the ESS, for example, the national coordinators, who are responsible for organizing the national interviewer training, are given research-based information on interviewer effects to demonstrate the positive effects of inter-viewer training. The SHARE specifications contain information about the national interviewer training and the train-the-trainer sessions and the PIAAC specifications emphasize the importance of quality controls and provide very detailed guidelines on the implementation of interviewer training.
Mapping of Interviewer Training Content to the
TSE
In this section, we map the specifications for general interviewer training and the program-specific training content of the three multinational survey programs along the TSE framework.
General Interviewer Training for Inexperienced
Interviewers
training for inexperienced interviewers, which go beyond the coverage of nonre-sponse and measurement error. In contrast, the ESS and SHARE interviewer train-ing guidelines include only some examples of topics covertrain-ing interviewer traintrain-ing for inexperienced interviewers.
Table 2 ESS, PIAAC, and SHARE General Interviewer Training Content
from a TSE Perspective for Inexperienced Interviewers
ESS1 PIAAC2 SHARE3
Representation
Coverage error --- ---
---Sampling error --- ---
---Nonresponse
error Doorstep interaction Gaining cooperation (incl. detailed contact and refusal conversion strategies)
Collecting process data information (incl. contact attempt and result of contact attempt)
Measurement
Measurement
error Standardized interviewing (incl. detailed rules)
Asking questions (incl. exercises)
Probing techniques Standardized question-asking Probing
Processing
error --- Recording answers (incl. exercises)
---Content of relevance for multiple TSE components
--- Introduction to survey research (incl. types of survey questions, interviewing terminology) Standards and ethics in survey research (incl. informed consent, data confidentiality, data security, exercises)
Remuneration and administrative aspects
Basics of computer-assisted interviewing (CAI)
---Note. Survey programs in alphabetical order; ESS = European Social Survey; PIAAC = Programme for the International Assessment of Adult Competencies; SHARE = Survey of Health, Ageing and Retirement in Europe; --- = no information included in the guidelines.
1 ESS (2016c); Beullens, Loosveldt, Denies, and Vandenplas (2016). 2 OECD (2013).
The PIAAC specifications for general interviewer training, that affect all compo-nents of the TSE, comprise an introduction, some standards and ethics in survey research, administrative aspects, and instructions on the basics of computer-assisted interviewing (CAI). With regard to nonresponse, these specifications include strate-gies for gaining cooperation, and with regard to measurement error, they relate to question-answering and probing techniques. In addition, processing error is cov-ered by techniques for recording answers. Nevertheless, looking at the proposed length of the training sessions for each topic, it becomes apparent that the focus is clearly on measurement error and on related practice sessions (OECD, 2013, Chap-ter 10.4).
In contrast, the ESS specifications for general interviewer training cover only nonresponse error (doorstep interaction training) and measurement error (train-ing of standardized interview(train-ing). Similarly, the SHARE specifications for gen-eral interviewer training are quite brief and cover only nonresponse error (training in process data collection) and measurement error (standardized interviewing and probing techniques). Neither the ESS nor the SHARE specifications include infor-mation about the length of the general interviewer training.
Program-Specific Interviewer Training for Inexperienced
and Experienced Interviewers
Ta bl e 3 Pr og ra m -S pe ci fic I nt er vi ew er T ra in in g C on te nt o f t he E SS , P IA AC , a nd S H A RE f ro
m a T
SE P er sp ec tiv e f or a ll In ter vi ew er s ESS 1 PI A AC 2 SH A RE 3 Rep re se nt at io n Co ve ra ge er ror Se le ct ing re sp on de nt s Sc re en er a dm in ist ra tion Sa mp lin g e rror ---Non re sp on se er ror Con ta ct s tra te gy ( de ta ile d) G ai ni ng c oo pe ra tion (inc l. r ea lis tic e xa m pl es ) Re fu sa l c on ve rs ion Sc re en er a dm in ist ra tion Lo cat in g s trat eg y Con ta ct s tra te gy G ain in g c oo pe ra tio n Re fu sa l a vo id anc e a nd c on ve rs ion Lo cat in g s trat eg y Con ta ct s tra te gy ( de ta ile d) G ai ni ng c oo pe ra tion ( de ta ile d, f oc us : re pr es en ta tiv en es s) Re fu sa l c on ve rs ion M eas ur em ent M ea su re m en t er ror In str um en t o ve rv iew (inc l. p ra ct ic e, f oc us : s pe ci fic q ue sti on s) St an da rd iz ed int er vi ew ing Sc re en er a dm in ist ra tion In str um en t a dm in ist ra tion (inc l. p ra ct ic e) In str um en t o ve rv iew ( inc l. p ra ct ic e) Pr ob ing Co lle ct ion o
f a d
ESS 1 PI A AC 2 SH A RE 3 C on te nt o f r el ev an ce f or m ul tip le T SE c om po ne nt s In tro du ct ion ( inc l. g oa l, fi nd in gs p re vi ou s ro un ds , d at a q ua lit y i ss ue s, d at a u sa ge ) Lo gi sti cs ( inc l. t ar ge t r es pon se r at e, su pp le m en ta ry ma te ria l , re m une ra tion , fie ldw or k p ro ce du re s, e tc .) A dm in ist ra tiv e t as ks ( con ta ct f or m s) In fo rm at ion f or r es pon de nt s p rio r i nt er vi ew (e .g ., d at a p ro te ct ion ; g en er al i nf or m at ion ES S; d at a c on fid en tia lit y; d at a s to ra ge , e tc .) Ex pe rie nc ed i nt er vi ew er s: c ha ng es s inc e la st r ou nd , c om pa ris on w ith o th er s ur ve ys Q ua lit y c on tro l In tro duct io n Re vi ew o f a dv anc ed m at er ia ls Ca se m an ag em en t s ys te m (inc l. d at a t ra ns m iss ion ) A dm in ist ra tiv e t as ks ( inc l. d isp os iti on co de s, c as e f ol de r, r ec or d o f c on ta ct) De mo a nd p ra ct ic e i nt er vi ew s ( ro le p lay an d/ or l iv e r es pon de nt p ra ct ic e) Q ue sti on -a nd -a ns we r s es sion (a fte r p ra ct ic e) A dd iti on al p ra ct ic e s es sion s Q ua lit y c on tro l a nd m on ito rin g In tro du ct ion ( inc l. g oa l, q ue sti on na ire ov er vi ew) Log ist ic s La pt op o ve rv iew (inc l. i ns ta lla tion c he ck ) Ca se m an ag em en t s ys te m (inc l. p ra ct ic e) Q ue sti on -a nd -a ns we r s es sion N ot e . S ur ve y p ro gr am s i n a lp ha be tic al o rd er ; E
SS = E
ur op ea n S oc ia l S ur ve y; P IA
AC = P
ro gr am m e f or t he I nt er na tion al A ss es sm en t o f A du lt Co m pe te nc ie s; S H A
RE = S
ur ve y o f H ea lth , A ge in g a nd R et ire m en t i n E ur op e;
--- = n
o i nf or m at ion i nc lud ed i n t he s pe ci fic at ion s; o pt ion al t ra in in g con te nt i n i ta lic s.
1 E
SS ( 20 16 c); B eu lle ns e t a l. ( 20 16 ).
2 O
EC
D
(2
01
3).
3 B
A comparison of the training content of the ESS, PIAAC, and SHARE reveals that the survey programs focus on different components of the TSE framework. However, some training content is similar for all three survey programs, for exam-ple, an overall introduction to the survey, which is relevant for multiple compo-nents of the TSE. In addition, all three survey programs offer training content on contacting, gaining cooperation, and refusal avoidance strategies, with the goal of reducing nonresponse error. To address measurement error, the interviewer train-ing specifications of all three survey programs include sessions providtrain-ing an over-view of the survey instruments.
The differences between the training guidelines of the three survey programs reveal that the ESS specifications include very precise information on survey logis-tics (e.g., target response rate, fieldwork procedures), administrative tasks, and information that must be provided to respondents before the interview starts (e.g., data confidentiality, data storage). These topics included in the ESS specifications are relevant for all TSE components as it might affect more than one error source. Similarly, the specifications for interviewer training in PIAAC are quite detailed with respect to administrative tasks, and they additionally include a large section on practical sessions (e.g., question-and-answer sessions, demo interviews), which are also relevant for multiple error sources. In comparison, SHARE does not include detailed specifications for administrative tasks. However, in the training session on mental health, there is a large sub-section on coding conventions. Training in cod-ing conventions aims to reduce processcod-ing error; this is not covered by the specifi-cations of the other two survey programs. Yet, the SHARE specifispecifi-cations for inter-viewer training do not include any information on quality control and monitoring, which is covered by the ESS and PIAAC training content specifications.
Discussion
interviewers vary largely across different survey projects. Thus, the involvement of interviewers related to tasks affecting coverage, sampling, and processing is not part of every survey project, as in the three cases examined here. Second, the measurement and detection of coverage, sampling, and processing errors requires a higher control effort compared to the two other error sources. And third, most sur-veys have experienced a dramatic decrease in response rates in recent years (Beul-lens, Loosveldt, Vandenplas, & Stoop, 2018), which might make the skill of gaining cooperation more salient.
When looking at the interviewer training guidelines of the three survey pro-grams in more detail, we identified several differences between the training content of the three multinational training programs. Training content aimed at reducing nonresponse error was identified in the interviewer training guidelines of all three multinational survey programs. However, training in locating sample units is not mentioned in the ESS guidelines. Training content aimed at reducing measurement error is covered in all three guidelines: the ESS guidelines ar the only guidelines including standardized interviewing, whereas probing techniques and the collec-tion of biomarkers or the administracollec-tion of a cognitive test are included only in the SHARE or PIAAC guidelines. These differences are due, in part, to the scope of the respective studies, which obviously differs across the three surveys we have compared in the present paper. Besides, training content relating to processing error is covered only in the SHARE training guidelines. Moreover, the PIAAC inter-viewer training guidelines include an extensive general interinter-viewer training agenda for inexperienced interviewers, which is only sparsely addressed by the ESS and SHARE. As PIAAC was conducted for the first time in the participating countries and – as general interviewer training forms the basis for additional project-specific training – we suspect that the interviewer training guidelines aimed to ensure that all interviewers working for PIAAC were at a similar level of knowledge. How-ever, all three multinational survey programs require that only interviewers who are trained in general interviewer tasks are employed for the survey. This is in line with Pennell et al. (2017), as interviewers with more interviewing experience are likely to minimize comparison error.
would be a promising avenue for future research. In addition, we use the TSE as theoretical framework for comparing training content of the three multinational survey programs. However, the TSE itself also has limitations. While using the TSE framework does allow us to map the recommended content, it does not allow us to map, for example, the recommended didactic methods.
Moreover, different learning strategies (e.g., class instructions and practical sessions) are implemented mainly in the case of training content related to mea-surement error. Specifically, all three guidelines highlight the importance of prac-tical training sessions. For the development of interviewer training concepts, the application of different learning modes and methods appears to be an important aspect which has only be sparsely taken into account so far (e.g., Daikeler & Bos-njak, forthcoming; Rutgers Online Degrees, n.d.; M. K. Smith, 2002): the field of andragogy addresses this topic and offers principles that are useful to consider when designing interviewer training programs (Tusting & Barton, 2003). For example, following Malcolm Knowles, adults prefer a self-directed approach and learning that is centered around common tasks (Meuler, 2010; Rutgers Online Degrees, n.d.; M. K. Smith, 2002). Moreover, interviewer training materials should take account of the fact that levels of educational attainment and experience vary greatly among adults.
Looking forward, in order to empirically investigate the different effects of general and study-specific interviewer training on the components of the TSE and, thus, on data quality, more experimental studies are needed. These studies should explore the effects of the various interviewer training contents on the differ-ent error sources as well as the interaction of differdiffer-ent error sources. For example, experimental evidence is needed to ensure that the focus on gaining cooperation, which is typical of many interviewer training concepts, contributes to effectively reducing nonresponse. A theoretical foundation could be the organizing model for future research investigating explanations for interviewer effects on multiple error sources, which West and Blom (2017) proposed in their research synthesis on inter-viewer effects. Their proposed model includes interinter-viewer training as a background characteristic that can be modified depending on the sources of interviewer effects identified. Future studies could structure research topics of interviewer training along this model in order to evaluate their impact on the respective survey errors.
Conclusion
involved in these processes, the most important error sources nonresponse and measurement error are broadly covered by all three training guidelines. Altogether, those guidelines could serve, together with the survey guidelines formulated out-side of the context of specific survey programs (e.g., Alcser, et al., 2016; Daikeler, Silber, Bosnjak, Zabal, & Martin, 2017; ISO, 2012; Lessler et al., 2008), as an excel-lent starting point for new – small as well as large-scale – surveys to design their interviewer training. Even the interviewer training of existing multinational survey programs could benefit from learning how other surveys plan and implement inter-viewer training to ensure interinter-viewers are well prepared for all tasks they have to fulfil during the implementation of the respective survey. It might be helpful to define the focus of interviewer training through determining the time devoted to a specific topic dependent on the magnitude of the survey error related to that train-ing topic (West & Blom, 2017). For example, a focus on traintrain-ing in contact and cooperation strategies is undoubtedly a good strategy in times of lower response rates or higher nonresponse bias. However, other components of the TSE should be likewise addressed in the respective interviewer training guidelines.
All this is only possible when interviewer training guidelines and materials are publicly available. Consequently, all survey programs would benefit from a high level of transparency (e.g., published interviewer training material). And since not all survey programs can afford a cost intensive high quality interviewer training, it would be imperative to have a standardized, pre-established training manual from which even smaller surveys can use relevant training modules.
References
American Association for Public Opinion Research (AAPOR, n.d.). best practices for sur-vey research. Retrieved from https://www.aapor.org/Standards-Ethics/Best-Practices. aspx (03/13/2018).
Alcser, K., Clemens, J., Holland, L., Guyer, H., & Hu, M. (2016). Interviewer recruitment, selection, and training. Guidelines for Best Practice in Cross-Cultural Surveys. Survey Research Center, Institute for Social Research, University of Michigan.
Benson, M. S., & Powell, M. B. (2015). Evaluation of a comprehensive interactive training system for investigative interviewers of children. Psychology, Public Policy, and Law, 21(3), 309-322. https://doi.org/10.1037/law0000052
Beullens, K., Loosveldt, G., Denies, K., & Vandenplas, C. (2016). Quality matrix for the Eu-ropean Social Survey, round 7. Retrieved from http://www.euEu-ropeansocialsurvey.org/ docs/round7/methods/ESS7_quality_matrix.pdf (2018/05/02)
Beullens, K., Loosveldt, G., Vandenplas, C., & Stoop, I. (2018). Response rates in the Eu-ropean Social Survey: Increasing, decreasing, or a matter of fieldwork efforts? Survey Methods Insights from the Field. Retrieved from https://surveyinsights.org/?p=9673 Biemer, P. P. (2010). Total survey error: Design, implementation, and evaluation. Public
Billiet, J., & Loosveldt, G. (1988). Improvement of the quality of responses to factual survey questions by interviewer training. Public Opinion Quarterly, 52(2), 190–211. Retrieved from http://www.jstor.org/stable/2749273
Börsch-Supan, A., & Jürges, H. (2005). The Survey of Health, Aging, and Retirement in Europe – Methodology. Mannheim.
Cantor, D., Allen, B., Schneider, S. J., Hagerty-Heller, T., & Yuan, A. (2004). Testing an automated refusal avoidance training methodology. Paper presented at the 60th Annual Conference of the American Association for Public Opinion Research, Phoenix, AZ. Dahlhamer, J. M., Cynamon, M. L., Gentleman, J. F., Piani, A. L., & Weiler, M. J. (2010).
Minimizing survey error through interviewer training: New procedures applied to the National Health Interview Survey (NHIS).
Daikeler, J., & Bosnjak, M. (forthcoming). How to conduct effective interviewer training: A meta-analysis. In K. Olson, J. D. Smyth, J. Dykema, A. Holbrook, F. Kreuter, & B. T. West (Eds.), Interviewer Effects from a Total Survey Error Perspective: CRC Press. Daikeler, J., Silber, H., Bosnjak, M., Zabal, A., & Martin, S. (2017). General interviewer
training curriculum for computer-assisted personal interviews (GIT-CAPI; Version 1, 2017). GESIS Survey Guidelines. Mannheim, Germany: GESIS – Leibniz-Institute for the Social Sciences. https://doi.org/10.15465/gesis-sg_en_022
Durand, C., Gagnon, M.-E., Doucet, C., & Lacourse, E. (2006). An inquiry into the effica-cy of a complementary training session for telephone survey interviewers. Bulletin de Méthodologie Sociologique, 92(1), 5-27. https://doi.org/10.1177/075910630609200103 Eckman, S., & Kreuter, F. (2011). Confirmation bias in housing unit listings. Public Opinion
Quarterly, 75(1), 139-150. https://doi.org/10.1093/poq/nfq066
ESS. (2016a). ESS interviewer briefing: NC manual. London: ESS ERIC headquarters. Ret-rieved from Bergen, European Social Survey Data Archive: http://www.europeansoci-alsurvey.org/methodology/ess_methodology/data_collection.html (2018/04/30) ESS. (2016b). ESS round 8 interviewer briefing: Interviewer manual. London: ESS ERIC
headquarters. Retrieved from Bergen, European Social Survey Data Archive: http:// www.europeansocialsurvey.org/methodology/ess_methodology/data_collection.html (2018/04/30)
ESS. (2016c). Survey specification for ESS ERIC member, observer and guest countries. Retrieved from http://www.europeansocialsurvey.org/methodology/ess_methodology/ survey_specifications.html (04/10/2018)
ESS. (2018). ESS-8 2016 documentation report. Edition 2.1. Retrieved from Bergen, Euro-pean Social Survey Data Archive: https://www.euroEuro-peansocialsurvey.org/docs/round8/ survey/ESS8_data_documentation_report_e02_1.pdf (2018/05/29)
Fowler Jr, F. J., & Mangione, T. W. (1986). Reducing interviewer effects on health survey data (Report No. 141). Retrieved from Rockville, MD: National Center for Health Ser-vices Research and Health Care Technology.
Fowler Jr., F. J. (1991). Reducing interviewer-related error through interviewer training, su-pervision, and other means. In P. P. Biemer, R. M. Groves, L. Lyberg, N. A. Mathio-wetz, & S. Sudman (Eds.), Measurement errors in surveys (pp. 259-278). Hoboken, NJ: John Wiley & Sons.
Groves, R. M. (2005). The interviewer as a source of survey measurement error. In R. M. Groves (Ed.), Survey errors and survey costs (pp. 357-406): John Wiley & Sons. Groves, R. M., Fowler, F. J. J., Couper, M. P., Lepkowski, J. M., Singer, E., & Tourangeau, R.
Lep-kowski, E. Singer, & R. Tourangeau (Eds.), Survey methodology (pp. 269-301). John Wiley & Sons.
Groves, R. M., Fowler, F. J. J., Couper, M. P., Lepkowski, J. M., Singer, E., & Tourangeau, R. (2009). Survey methodology (Vol. 2). Hoboken, NJ: John Wiley & Sons.
Groves, R. M., & Lyberg, L. (2010). Total survey error: Past, present and future. Public Opi-nion Quarterly, 74(5), 849–879. https://doi.org/10.1093/poq/nfq065
Guest, L. (1954). A new training method for opinion interviewers. Public Opinion Quarter-ly, 18(3), 287-299.
Hubal, R. C., & Day, R. S. (2006). Informed consent procedures: An experimental test using a virtual character in a dialog systems training application. Journal of Biomedical In-formatics, 39(5), 532-540.
International Organization for Standardization (ISO). (2012). Market, opinion and social research - Vocabulary and service requirements (ISO Standard No. 20252: 2012 [en]. Geneva, Switzerland: International Organization for Standardization.
Lessler, J. T., Eyerman, J., & Wang, K. (2008). Interviewer training. In E. D. De Leeuw, J. J. Hox, & D. A. Dillman (Eds.), International handbook of survey methodology (pp. 442-478). New York, NY: Taylor & Francis Group.
Loosveldt, G. (2008). Face-to-face interviews. In E. D. De Leeuw, J. J. Hox, & D. A. Dill-man (Eds.), International handbook of survey methodology (pp. 201–220). New York, NY: Taylor & Francis Group.
Malter, F., & Börsch-Supan, A. (2017). SHARE Wave 6: Panel innovations and collecting dried blood spots. Munich: Munich Center for the Economics of Aging (MEA) at the Max Planck Institute for Social Law and Social Policy (MPISOC).
Malter, F., Schuller, K., & Börsch-Supan, A. (2016). SHARE compliance profiles – Wave 6. Munich: MEA, Max Planck Institute for Social Law and Social Policy.
Mayer, T. S., & O’Brien, E. (2001). Interviewer refusal aversion training to increase survey participation. Proceedings of the Annual Meeting of the American Statistical Associ-ation, 2001.
McConaghy, M., & Carey, S. (2005). What happens after ART? Results of two experiments designed to improve response rates with interviewers at the Office of National Statis-tics, UK. Paper presented at the 60th Annual Conference of the American Association for Public Opinion Research, Phoenix, AZ.
Meuler, E. (2010). Didaktik der Erwachsenenbildung - Weiterbildung als offenes Projekt. In R. Tippelt & A. Von Hippel (Eds.), Handbuch Erwachsenenbildung/Weiterbildung (Vol. 4, pp. 973-988). Wiesbaden: VS Verlag für Sozialwissenschaften.
O’Brien, E. M., Mayer, T. S., Groves, R. M., & O’Neill, G. E. (2002). Interviewer training to increase survey participation. Proceedings of the 2002 Joint Statistical Meetings of the American Statistical Association.
OECD. (2011). PIAAC technical standards and guidelines. Retrieved from http://www.oecd. org/skills/piaac/documentation.htm (2014/06/12)
OECD. (2013). The survey of adult skills: Reader’s companion, second edition, OECD skills studies, OECD Publishing, Paris. Retrieved from
https://doi.org/10.1787/9789264258075-en (2016/12/05)
multi-national, multiregional, and multicultural contexts (pp. 269-298). Hoboken, NJ: John Wiley & Sons.
Pennell, B.-E., Hibben, K. C., Lyberg, L. E., Mohler, P. P., & Worku, G. (2017). A total survey error perspective on surveys in multinational, multiregional, and multicultural contexts. In P. P. Biemer, E. D. de Leeuw, S. Eckman, B. Edwards, F. Kreuter, L. E. Lyberg, C. N. Tucker, & B. T. West (Eds.), Total Survey Error in practice (pp. 179-201). Hoboken, NJ: John Wiley & Sons.
Rutgers Online Degrees. (n.d.). The principles of adult learning theory. Retrieved from https://online.rutgers.edu/blog/principles-of-adult-learning-theory/ (2018/05/04) Schaeffer, N. C., Dykema, J., & Maynard, D. W. (2010). Interviewers and interviewing. In P.
V. Marsden & J. D. Wright (Eds.), Handbook of survey research (pp. 437–470). Bingley, UK: Emerald Group Publishing Limited.
Schnell, R., & Trappmann, M. (2006). The effect of the refusal avoidance training experi-ment on final disposition codes in the German ESS-2 (Working Paper 3/2006: Center for Quantitative Methods and Survey Research, University of Konstanz). Retrieved from http://nbn-resolving.de/urn:nbn:de:0168-ssoar-114189
Schouten, B., Peytchev, A., & Wagner, J. (2017). Adaptive survey design. Boca Raton, FL: CRC Press.
Smith, M. K. (2002). Malcolm Knowles, informal adult education, self-direction and and-ragogy. Retrieved from http://infed.org/mobi/malcolm-knowles-informal-adult-educa-tion-self-direction-and-andragogy/ (2018/05/04)
Smith, T. W. (2011). Refining the total survey error perspective. International Journal of Public Opinion Research, 23(4), 464–484. https://doi.org/10.1093/ijpor/edq052 Smith, T. W. (2019). Improving multinational, multiregional and multicultural
comparabili-ty (3MC) using the total survey error (TSE) paradigm In T. P. Johnson, B.-E. Pennell, I. Stoop, & B. Dorer (Eds.), Advances in comparative surveymethods: Multinational, multiregional, and multicultural contexts (3MC) (pp. 1-21). Hoboken, NJ: John Wiley & Sons.
Survey Research Center at the University of Michigan. (2016). Guidelines for best practice in cross-cultural surveys. Survey Research Center, Institute for Social Research, Uni-versity of Michigan.
Tusting, K., & Barton, D. (2003). Models of adult learning: a literature review. Retrieved from London, UK: National Research and Development Centre for Adult Literacy and Numeracy (www.nrdc.org.uk (2018/05/09)).
West, B. T., & Blom, A. G. (2017). Explaining interviewer effects: A research synthesis. Journal of Survey Statistics and Methodology, 5(2), 175–211.
https://doi.org/10.1093/jssam/smw024
A
pp
en
dix
Su rv ey S pe ci fic at io ns a nd C ha ra ct er ist ics o f th e E SS , P IA AC , a nd S H A R E A cr os s a ll P ar tic ip at in g C ou ntr ies ES S 2 01 6 ( Ro un d 8 ) 1 PI A AC ( Cy cle 1 , R ou nd 1 ) 2 SH A RE (W ave 6 ) 3 Sc op e A tti tud es , b el ief s, a nd b eh av io r A du lt co m pe te nci es H ea lth, ag ei ng , a nd re tire m en t Su rve y t yp e Cr os s-se ct ion al ( fir st r ou nd i n 2 00 2, Ro und 8 i
n 2
01
6)
Ev
er
y 2 y
ea rs Cr os s-se ct ion al ( fir st c yc le i n 2 01 1/ 12 ) M ul ti-cy cle ( ev er y 1 0 y ea rs) Lon gi tud in al ( sta rt i n 2 00 4/ 05 , W av e 6 i n 2 01 4/ 15 ) Ev er
y 2 y
ea rs Su rve y m od e F2 F: C A PI F2 F c or e q ue sti on na ire : C A PI Se lf-a dm in ist er ed c og ni tiv e as se ss m ent : C A SI , P A PI F2 F: C A PI Se lf-a dm in ist er ed P A PI f or s en sit iv e qu es tion s i n s om e c ou nt rie s 4 Ph ys ica l me as ur eme nt s Sa m pli ng m eth od Pr oba bi lit y-ba se d Pr oba bi lit y-ba se d Pr oba bi lit y-ba se d # o f p ar tic ip at in g c ou nt rie s 23 24 18 Ta rg et p op ul at ion G en er al , 1 5 y ea rs + G en er al , 1
6 - 6
ES S 2 01 6 ( Ro un d 8 ) 1 PI A AC ( Cy cle 1 , R ou nd 1 ) 2 SH A RE (W ave 6 ) 3 Non -s ta nd ar d i nt er vi ew er ta sk s 7 No Ye s Ye s In te rv iew er r em un er at ion sc he me G en er al a pp ro ac h: f re el anc e c on tra ct , pa id p er c om pl et ed i nt er vi ew In de pe nd en t o f t he n um be r o f co m ple te d i nt er vi ew s ---In te rv iew er w ork loa d M ax . o f 4 8 c as es ( re sp on de nt s a nd no n-r esp on de nt s) M ax . o f 4 0 c om pl et ed a ss es sm en ts p er m ont h O n a ve ra ge 1 5 i nt er vi ew s; m us t n ot ex ceed 5 0 In te rv iew er r ec ru itm en t & hi ring Re sp on sib ili ty o f s ur ve y i ns tit ut e In te rv iew er s w ith g en er al t ra in in g Re sp on sib ili ty o f s ur ve y i ns tit ut e A im i s t o h ire i nt er vi ew er s w ith a t l ea st 2 y ea rs w or ki ng e xp er ie nc e Se le ct ed f ro
m a v
ar ie ty o f j ob -o ffe rin g sou rc es Ap pl ic an ts w ith v ar io us q ua lifi ca tion s Pr oc es s s ho ul d s ta rt a t l ea
st 8 w
ee ks pr io r t o t he s ta rt o f d at a c ol le ct ion Th e a pp ro pr ia te n um be r o f in te rv iew er s h av e t o b e a va ila bl e i n a su ffi ci en t r eg ion al s pr ea d N ot e . S ur ve y p ro gr am s i n a lp ha be tic al o rd er ; C A
PI = c
om pu te r-a ss ist ed p er son al i nt er vi ew in g; C A
SI = c
om pu te r-a ss ist ed s el f-i nt er vi ew in g; E SS = E ur op ea n S oc ia l S ur ve y; F
2F = f
ac e t o f ac e; P A
PI = p
ap er a nd p enc il i nt er vi ew in g; P IA
AC = P
ro gr am m e f or t he I nt er na tion al A ss es sm en t o f A du lt C om pe te nc ie s; S H A
RE = S
ur ve y o f H ea lth , A ge in g a nd R et ire m en t i n E ur op
e;
--- = n o i nf or m at ion a va ila bl e i n t he s ur ve y d oc um en ta tion.
1 E
SS ( 20 16 c); B eu lle ns , L oo sv eldt , D en ie s, a nd V an de np la s ( 20 16 ); E SS ( 20 18 ).
2 O
EC
D
(2
01
3).
3 B
ör sc h-Su pa n a nd J ür ge s ( 20 05 ); M al te r, S ch ul le r, a nd B ör sc h-Su pa n ( 20 16 ); S H A RE ( 20 18 ).
4 A
us tria , C ze ch R ep ub lic , G re ec e, I sr ae l, S lo ve nia , S w itz er la nd .
5 G
iron a a dd ed t o S pa in.
6 44
–1 70 i nt er vi ew er s w er e w or ki ng i n t he fi eld . I nf or m at ion a bo ut t he n um be r o f i nt er vi ew er s t ra in ed w as n ot p ub lis he d.
7 R
