5.6 Ability, Motivation, and Difficulty
5.6.2 Measuring Ability, Motivation, and Difficulty
Having defined ability, motivation, and difficulty, the next task is to determine appro- priate means of measuring these factors. As suggested by Green, Krosnick & Holbrook (2002) and Krosnick (1991), individual traits, such as educational attainment, polit- ical informedness, party membership, and level of party identification, provide useful proxies for respondent ability and motivation. Establishing appropriate measures of difficulty, however, is particularly complicated because difficulty tends to be extremely subjective. The search for appropriate measures of difficulty are further hampered by the fact that most surveys only supplement opinion questions with basic demo- graphic questions. However, using newly collected data from third parties or survey participants, Chapter 7 explores alternative ways of measuring difficulty.
As discussed above, ability includes such things as cognitive sophistication, knowl- edge about political issues, and opinion crystallization. In examining cognitive so- phistication, Krosnick & Alwin (1987) conclude that educational attainment is a good proxy. Since respondents’ ability to answer factual political information questions,
17Achen (1975) proposed that opinion expressions are actually distributions around a central
point—participants may answer the same question differently even if their underlying beliefs do not change because questions are vague.
called political informedness, has been shown to be highly correlated with awareness about political issues (Zaller 1992), measures of political informedness will also be used to capture ability.18
Motivation, on the other hand, will be affected by the interaction of respondent and survey characteristics. For example, political informedness, which captures re- spondent ability, is also an indicator of political involvement and, as such, offers some insight into respondent motivation; topic specific knowledge provides a similar mea- sure of political involvement at the narrower topic specific level. Motivation might also be captured by other measures of a respondent’s political involvement, includ- ing party membership and level of party identification. Politically disenfranchised people, such as racial minorities, women, and low income families, might also have different levels of motivation.19 From the survey design perspective, issues such as
the length of the survey and the relevance of the issues under consideration will affect respondent motivation. By combining these factors, it will be possible to test which characteristics of respondents affect shirking in surveys.
Until recently difficulty has not been conceptualized from an individual perspec- tive. Most of the work on how task difficulty affects respondent behavior has focused on how different survey presentations affect the general behavior of respondents—the analysis of different survey modes (i.e., face-to-face, telephone, and self-completion) is a prominent example of this type of analysis (Green, Krosnick & Holbrook 2002). Unfortunately, question level difficulty measures are scarce and subjective measure of difficulty are even less available. To obtain subjective measures of difficulty, recent surveys have asked respondents to indicate how difficult they found certain ques- tions to answer. This provides a new avenue through which to examine difficulty (Albertson, Brehm & Alvarez Forthcoming, 2004). Since this type of data is not
18Political opinion surveys often include factual knowledge questions designed to establish how
familiar respondents are with politics.
19The direction of this effect is unclear since those who do not have much political influence
through the standard political venues might view surveys as an alternative means of affecting political outcomes or they might simply accept their political marginalization and therefore have very little interest in politics.
available for most surveys, similar difficulty measures may be collected from auxiliary or third-party samples.
Assuming appropriate indicators of shirking can be identified, it is possible to test how ability, motivation, and difficult are related to the probability of shirking. The next two chapters use data collected in the 1996–1997 Indianapolis–St. Louis Election Study (Indianapolis 1996), the 1980 and 1984 National Elections Studies (NES) Pre-Election Surveys, and the 1998–1999 Multi-Investigator Study (MI1998) to explore different means of identifying shirking. Once shirking is identified, factors that influence this behavior can be examined. In Chapter 7, when response times are used to identify shirking, the impact that this type of shirking behavior has on ana- lytic results is explored. In addition, different approaches to accounting for shirking behavior in empirical analysis are considered. Finally, the comparison in Chapter 8, of shirking behavior across survey modes, using data from the Caltech Internet Sur- vey Project, provides another look at how different survey design factors might affect shirking behavior. This work culminates with a discussion the implications that ob- servation of shirking behavior has for the design of surveys and for the analysis of survey data.
Chapter 6
Two Behaviors that Might be
Indicative of Shirking
If you choose not to decide, you still have made a choice (Rush 1990).
As suggested above, two alternative indicators of shirking are explored: respon- dents who provide incorrect answers quickly on factual questions and the proportion of “no opinion” responses. Using these measures, it is feasible to test the effect of respondent characteristics on shirking behavior but measuring difficulty at the survey level is complicated unless the survey was administered using multiple modes.1
Question level difficulty measures are explored as a means of explaining “no opin- ion” responding. Because difficulty measures were not collected in the 1980 and 1984 NES Pre-Election Surveys, auxiliary or third-party difficulty data was collected to supplement the existing data. These difficulty reports are explored below as is the correlation between these difficulty measures and “no opinion” response behav- ior. Unfortunately, these third-party difficulty reports do not appear to be very closely correlated with “no opinion” response rates. Hence, attention is turned to “no opinion” response behavior in the 1998–1999 Multi-Investigator Study (MI1998); the MI1998 collected respondent-reported difficulty measures for six policy questions. Aggregating respondent-reported difficulty across questions provides a measure of subjective survey difficulty that may be implemented with the survey level data from the MI1998.
6.1
The Quick and the Wrong
In many situations it may be impossible to identify shirking at the question level. If there is insufficient data or if it is impossible to determine what question level behavior
1Chapter 8 explores the differences in shirking behavior across telephone and Internet survey
is associated as shirking, it may be easier to examine shirking at the survey level. Hence the first approach to identifying shirking explored here involves examining respondent behavior on factual question—respondents who answer factual questions quickly and provide incorrect responses will be identified with shirking. Assuming that response time is positively correlated with effort, that with enough effort a respondent would be able to provide accurate responses to factual questions, and that behavior on factual questions is indicative of overall survey response behavior, this “quick-wrong” measure provides a simple way to identify shirkers at the survey level. All that is needed is a survey that collected response times for factual questions. Fortunately the 1996–1997 Indianapolis–St. Louis Election Study (Indianapolis 1996) collected extensive response time data.2 In particular, latent timers, measured
in hundredths of seconds, were collected for the three following political knowledge questions: Whose responsibility is it to determine if a law is constitutional or not? What are the first 10 amendments to the constitution called? How much of a majority is required for the U.S. Senate and House to override a presidential veto?3
The raw data indicated latent response times between zero seconds and over three minutes but, as mentioned above, many problems may occur with the coding of re- sponse times. Huckfeldt, Sprague & Levine (2000) recommend excluding from the analysis respondents with response times of less than one second (the minimum time that would be required to read the question) and more than three standard devia- tions above the sample mean.4 Having considered several different transformations of response times, it appears that the distribution of the log of response times most closely resembles a normal distribution.5
2See Mulligan et al. (2003) for a discussion of latent and active response timers.
3See Appendix A.4 for a description of the data used from the Indianapolis–St. Louis Election
Study, 1996–1997.
4This trimming process amounts to excluding 383 respondents, or fifteen percent of the sam-
ple, from the analysis. As long as timer inaccuracies are not correlated with responses, dropping respondents with inaccurate timer data should not affect consistency but may affect efficiency.
5Transformation considered include: the cubic, identify, square, square-root, natural log, inverse,
Figure 6.1 contains distributional graphs for the log of the trimmed response times for three political knowledge questions.6 Even after trimming of outliers, the response
time data appears to be slightly skewed to the right but perhaps this can be explained by examining how response times correspond with accuracy.7
0 .5 1 1.5 Density 4 5 6 7 8 9
Log of Constitution Trimmed Timer
0 .5 1 1.5 Density 4 5 6 7 8 9
Log of Bill of Rights Trimmed Timer
0 .5 1 1.5 Density 4 5 6 7 8 9
Log of Presidential Veto Trimmed Timer
0 .5 1 1.5 Density 4 5 6 7 8 9
Log of the Sum of Trimmed Timers
Figure 6.1: Histograms of Trimmed Latent Response Times for Factual Questions