Making Surveys Work for You: Some Important Design Considerations

(1)

Making Surveys Work

for You: Some Important

Design Considerations

By

Don A. Dillman*

For

Net-Conference on Survey Methods and Measurement

June 15,2011

*Regents Professor in the Department of Sociology and Deputy Director of the Social and Economic Sciences Research Center at Washington State University, Pullman, Washington 99164-4014. [email protected] http://www.sesrc.wsu.edu/dillman

1 c Don A. Dillman June 15, 2011

Welcome…

 …to this web conference on designing effective

surveys.

 I hope that you will find it useful.

 My focus today is on how survey methods are,

changing, and to provide helpful suggestions for navigating through these changes.

 Let me begin with some of the questions you

asked in the material sent to me, and how I’ll go about responding.

c Don A. Dillman June 15, 2011 2

A few of your questions.

 What errors are introduced when modes are combined?  Can landline and cell phones be combined effectively?

 Is it okay to use mail primarily, with telephone follow-up?  Is there bias from internet surveys?y

 How can response rates be improved (Incentives?) and how much is enough?

 What can we do to get youth and minorities to respond?  What is the state of the science on framing opinion questions?  Can you give us practical guidelines for urban vs. rural areas?

Providing answers is a challenge?

 Different answers may fit different situations.

 When I looked at these (and other) questions, I thought I

could be most useful to focus on criteria for decision-making and give examples of what’s possible or practical…

 …and then, responding to questions will allow us to go

deeper into the explanations.

 First, some background.

None of our main data collection

methods are working as well as we

would like when used alone.

 Telephone has ~70% coverage; response rates have

declined steady; now as low as10-15% in national surveys.

 The Internet has “effective” household coverage < 70%;

(access + skill +frequency of use) and response rates are often low.

 In-person methods affordable for only a few surveys, and

coverage is increasing problem.

. c Don A. Dillman June 15, 2011 5

Attempts to circumvent these limitations are prevalent but also have problems

 Cell phones added to RDD samples can improve

coverage of the general public, but are also expensive.

 In addition, cell phone numbers are often carried from one

geographic area to another when people move.

 Effective household Internet access is lower than the

number of connections, because of infrequent use.

 We cannot use email to access general public households.  We have no sampling algorithm equivalent to RDD.

 We can only obtain web response by approaching through postal

mail.

(2)

Mail Survey Methods are

experiencing a revival of interest

 The U.S. Postal Service makes available (through

vendors) all residential addresses (The Delivery Sequence File, or DSF)

 It is now our best household sampling frame and may  It is now our best household sampling frame, and may

be our only source for accessing nearly all (~ 95-97%) households.

 One can request by mail the return of a mail

questionnaire and/or a response via web.

Two design considerations now seem

to dominate survey methodology

 Tailored Design—matching data collection modes to

populations, topics, and survey task. (Your studies intrigue me in this regard.)

 Mixed-mode surveys, using 2y , g nd, 3, rdmode to overcome

problems of single mode, is becoming increasingly important as a way of dealing with the shortcomings of individual modes.

 Thus, we need to figure out where mail, web and

telephone can and cannot help us collect better quality survey data in both situations.

Are Mixed Modes More of a Solution

or More of a Problem?

 Perhaps there is no choice!

 Response rates are declining (telephone)

 Often we cannot reach everyone by single modes.  Sample Frames are less adequate (random digit dialing)p q ( g g)  Cannot contact random samples of internet addresses

 But, are we getting improvement in data quality or perhaps

producing even larger problems by mixing modes?

 Do we get the same measurement across modes?

 Do we reduce non-response error through adding a new mode

or only increasing measurement problems?

To talk about these the need for

tailoring and for mixed-mode designs

I need to provide some context:

 Doing successful sample surveys means that we must

conduct sample surveys in a way that reduces error to the extent practical

the extent practical.

 We also need to recognize that there are multiple

sources of error in sample surveys.

What a Survey Can Do:

 Provide the distribution of a characteristic in a population by surveying only a small portion of the members of that population.

 …which allows us to answer questions like these:  What percent of Minnesota residents now smoke?

 What percent of unemployed adults in Minneapolis use tobacco each day?

 Do students in suburban high schools have higher rates of tobacco use than students in inner city schools?

How do sample surveys allow

us to make precise estimates

for the opinions and

behaviors of millions from a

survey of hundreds or a few

(3)

Coverage Error

Sampling Error

In Order for Surveys to Make Estimates With Known Precision, We Must Limit Four Kinds of Error:

Measurement Error

Non-response Error

Coverage

Error

Occurs because all members of the population are not given an equal or known (non-zero) chance of being selected for the survey.

H it h

How it happens:

Incomplete, poorly constructed and out-of-date lists (e.g. employees, members, and/or customers).

All households do not have telephones.

Many people do not have Internet access or the skills to use it.

For Example

 ~25%+ of households do not have landlines.  Only ~65% of households have internet access from

home (additional 11% from somewhere else).

S i t t ti b d

 Some internet connections are by modem.

 Households without internet access or landline access

differ from others in significant ways.

Occurs because only some members of the population are allowed to participate in the survey while others are systematically excluded.

Sampling

Error

The general (but incomplete) indicator of precision is completed sample size.

Number of completed interviews and precision of estimates at 95% confidence (quadrupling number of completed questionnaires decreases sampling error by one-half).

100 = +/- 10% 1150 = +/- 3% 400 = +/- 5% 2200 = +/- 2%

If 60% of our 400 respondents say they smoke cigarettes each day we can be 95% confident that between 55% and 65% of all people in our population smoke cigarettes each day (ignoring other possible errors).

The Pascal Triangle

and its Application

How is it possible to make estimates with stated confidence (e.g. 95%) that tens of thousands or millions of people have a particular characteristic,

while surveying so few people?

An Inappropriate Perspective

for an Election Survey

 Voters on election day

All Voters are Unique

(4)

A Necessary Perspective in an

Election Survey

Voters on election day

In a two candidate election there are TWO types of voter. Let’s assume 50% are going to vote for each candidate.

Why Does Random Sampling with

Complete Coverage Work?

• It lets probabilities work, as described in a Pascal Triangle.

• The Pascal Triangle shows the probable outcomes of 1The Pascal Triangle shows the probable outcomes of 1 to 10 random draws of respondents from populations in which one-half of respondents have each characteristic.

Chances in Ten Consecutive Draws of Randomly Selecting All Combinations of Red and Blue “Voters”

1 2 3 2 4 8 Number of Draws Sum of Chances 252 3 4 5 6 7 8 9 10 8 16 32 64 128 256 512 1024 21 c Don A. Dillman June 15, 2011

Interpretation of the 10 “Draws”

252

 Chances of 10 Blue - 1/1024

 Chances of 5 red 5 blue - 252/1024

 Chances of 4-6 of each color - 672/1024 22 c Don A. Dillman June 15, 2011

Number of Randomly Needed Depends Mostly on Sample Size and Not on Population Size

Sample Size for +/- 3% Precision 92 341 Population of this size 100 500 341 517 880 985 1,057 1,066 1,068 500 1,000 5,000 10,000 100,000 1,000,000 100,000,000 23 c Don A. Dillman June 15, 2011

An Observation

 Much of the traditional design work in random sample

surveys has been to assure high coverage so that random sampling will be effective, i.e. all people who vote have an equal chance of being selected in the sample

sample.

 But, there are also other sources of error to consider.

(5)

Measurement Error

Occurs because of inadequate question wording, placement in the questionnaire, the interviewer and/or the respondent.

Some known sources of measurement error:

• Some known sources of measurement error: • Mode effects:

Social desirability Primacy/Recency Acquiescence • Question order effects • Different visual layouts

• Measurement differences can be a significant problem when trying to use more than one mode for the same study.

For example, Interviewer Presence

May Have a Normative Influence:

60

How would you rate your health?

50 50 0 10 20 30 40 50

Excellent Good Fair Poor

Per

cent

Mail Telephone Face-to-face

Respondents more likely to give culturally acceptable answer on telephone or in-person. 30 38 43 50 50 42 18 10 10 1 2 2 26 c Don A. Dillman June 15, 2011

Occurs because some people do not respond when requested to answer the survey questions, and they are different in some way from those who do answer.

Non-response Error

F l

For example:

Are older people more likely to respond than younger people?

Are employed people more likely to respond than unemployed people in an employment rate survey?

How $2 Token Cash Incentive in Advance Influenced Response Rates Across Age Groups

60 70 80

1993 Survey of Washington State New Drivers License Holders

Without $2 With $2 0 10 20 30 40 50 <35 36-49 50-60 61+ Age Without $2 28 c Don A. Dillman June 15, 2011

To summarize, it’s important to reduce each kind of error. Maximizing one at expense of others is NOT a good idea.

Coverage

Error Sampling Error

c Don A. Dillman June 15, 2011 29 Measurement

Error

Nonresponse Error

A Case Study on seeking

improvement by mixing modes

 …the Gallup Organization tried to switch from telephone to web for periodic cross-sectional employee studies in 1999, and noticed significant differences in

measurement.

 At the same time, some response rates appeared to be

falling.

 These concerns led to an experiment on the effects of

four survey modes on measurement and switching modes as a means of improving response rates.

(6)

Case Study: an attempt to improve data quality by offering multiple modes

 Four modes were examined:

 Telephone, Interactive Voice Response (IVR), web and mail.  Can response rates be improved by offering two modes

in sequence?q

 What are the measurement consequences?

For details, see: Dillman, Tortora, Phelps, Swift, Kohrell, Berck, and Messer, 2009. “Response rate and measurement differences in mixed-mode surveys using mail, telephone, interactive voice response, and the Internet.” Social Science Research, 38(1), 1-18.

Case Study Comparison:

Telephone, IVR, Mail, and Web

Questions were comparable across modes, using the preferred format for telephone.

Q2. Overall, how satisfied are you with your long distance company?

 1 Not at all satisfied

 2

 3

 4

 5 Extremely satisfied

Q3. Considering what you get and what you pay for it, how would you rate the overall value of your long distance company’s products and services?

 1 Terrible

 2

 3

 4

 5 Outstanding

How the study was implemented

 A sample of individuals that submitted warranty

information on new purchases in order to get access by either mail or telephone.

 A two-step implementation process; offer initial mode

with multiple contacts and after a pause of 2-3 weeks offer the second mode.

 Mail to telephone.  Telephone to mail.  IVR to telephone.  Web to telephone

Response rates improved

significantly after a mode switch was

made

83% 81% 51% _48% 50 60 70 80 90 100

c Don A. Dillman June 15, 2011 34 0 10 20 30 40 50 Additional Response Initial Response

Effects of Initial Modes on choosing

most positive category for five

“long-distance” satisfaction questions

Percent choosing positive labeled end-point

50 43 50 60 ent Telephone IVR Mail Web

c Don A. Dillman June 15, 2011 35 39 26 36 21 39 21 33 22 21 11 29 19 18 9 29 18 16 0 10 20 30 40 Q2 Q3 Q4 Q5 Q6 Perc e Web T I M W T I MW T I M W T I M W T I M W 27

Aural modes produced different

results than visual modes

 Responses to aural modes (telephone, IVR) were

similar.

 Responses to visual modes modes (web, mail) grouped

together.

 Control panels showed that primacy/recency not a cause

of measurement differences;.

 Traditional explanations of differences e.g. social

desirability seemed unlikely.

(7)

Was the higher response rate worth

the cost?

 Probably not, because of the measurement differences

that resulted.

 Telephone obtained more extreme answers regardless of

whether first mode or last mode

 Additional panels showed no primacy/recency effects.  The demographics did not change in a way that

suggested different kinds of respondents answered the second mode.

 Either mail or telephone alone would probably have been

adequate, except for the measurement difference. c Don A. Dillman June 15, 2011 37

Why this example?

 Mixing survey modes is not always a desirable solution.  Doing that requires us to deal with many measurement

issues.

 But using mail-alone may also be undesirable  But, using mail alone may also be undesirable.  To sort out the possibilities and implications, we need to

think about coverage, non-response and measurement.

Two Ways that Switching Modes

May Change Measurement?

1. Mode choice influences question structure and wording.

2. Presence vs. absence of interviewer (e.g.

l h b) i fl d

telephone vs. web) influences respondent answers.

1. How Changes in Wording and

Construction Influence Answers

 Each mode has evolved to favor certain ways of asking

questions to maximize the potential to obtain quality data.

 This tendency comes from the “tradition” of how each

mode was designed to maximize its potential use, regardless of how it might be done in another survey mode.

Two Examples of Question

Construction Differences:

 Not offering the same answer categories in all

modes.

 Asking an open-ended question in one mode and a

l d d d ti i th th d

closed-ended question in the other modes.

Unintentional Construction Effect:

Volunteered Responses on Telephone

To what extent do you favor or oppose expansion of NATO?

Telephone: Mail:

1 Strongly favor 1 Strongly favor

c Don A. Dillman June 15, 2011 42 2 Somewhat favor 2 Somewhat favor

3 Somewhat oppose 3 Somewhat oppose 4 Strongly oppose 4 Strongly oppose 5 NO OPINION

6 DON’T KNOW 7 REFUSE

(8)

Unintentional Construction Difference:

Open vs. Closed Ended

Marital status was asked differently by one

organization on telephone and web (Tortora, 2004).

Telephone: What is your marital status?

Web: Which of the following best describes your marital status? Single

Married Separated Divorced Widowed

Unintentional Construction Difference:

Results from a Telephone and Web

Comparison

What is your marital status? Differences were significant and tracked across monthly data collections.

Telephone (n = 10,380)

Web (n = 9,713)

Difference

Single 26.8 23.5 -3.3 Married 63.1 62.0 -1.1 Separated 7.0 10.5 +3.5 Divorced 1.0 1.5 0.5 Widowed 2.0 2.4 0.4 TOTAL 99.9 99.9

2. How does the presence of interviewer/human interaction influence respondent answers

 A substantial literature shows that mail obtains different answers to many questions than for interview surveys.

 Among the reasons:

 Presence of the interviewer (interaction expectations)

P d h t i ti f i t i d d i t b

 Presumed characteristics of interviewer and desire to be seen in a favorable way.

 Aural and visual information are perceived using different sensory systems and processed using separate neural centers in the brain.

 Aural communication places higher demands on respondent

memory.

Is the presence of an interviewer

helpful or not?

 Interviewers control the delivery of survey information

whereas respondents have control over how they complete the survey in self-administered surveys.

 Interviewers can have positive impacts by clarifying

i f ti d ti ti d t

information and motivating respondents.

 Interviewers can have negative effects because

respondents are more likely to give socially desirable answers and acquiesce (rather than disagree) in the presence of an interviewer.

Interview

er

Presence May Have a

Normative Influence:

Social Desirability

63

60

70 How often do you drive a car after drinking alcoholic beverages? Respondents are more likely to give culturally acceptable answers on the telephone.

1 11 36 0 52 0 2 9 27 0 10 20 30 40 50 Mail Telephone (2= 14.8 p < .001)

Frequently Occasionally Seldom Never Don’t Know

Percent

An Effect of Interviewer Presence:

Acquiescence,

the Tendency to Agree

75 70 70 72 87 87 79 78 81 70 80 90 Mail n=884 Telephone n=400

Percent of respondents who agree with each of 9 opinions on how to encourage seat belt use

c Don A. Dillman June 15, 2011 48 58 39 ₃₆ 27 19 49 48 35 21 0 10 20 30 40 50 60 1 2 3 4 5 6 7 8 9 p Per cen t

(9)

Greater Extremeness on Four-Point

Scales

in General Public Survey

Percent choosing “not a problem” (vs. small, moderate, or serious)

52 49 71 74 60 53 59 60 70 80 T elephone Mail

49 18 32 40 28 10 12 21 41 24 32 34 0 10 20 30 40 50

Mean for 9 Questions: Telephone, 47.1 Mail, 31.9 Difference, 15.2 (Dillman & Mason, 1984)

Will Scalar Questions With Little or No Social Desirability Produce the Same Results Across Modes?

 Telephone respondents tend to give more extreme

positive answers than do web respondents.

 In some cases social desirability, acquiescence or

In some cases social desirability, acquiescence or primacy/recency do not seem to explain why.

Does Presence of an Interviewer Influence Response to Scalar Questions?

 A series of identically worded scalar questions was

tested in a web telephone experiment.

 Please note at this point that we are starting to bridge

between interviewer presence and visual vs. aural communication. These attributes are intertwined.

If we use exactly the same wording for scalar questions in aural vs. visual surveys…

 Will we get the same answers

 A series of evaluations were made for a variety of scalar

questions in random sample student surveys at Washington State University.

Question formats evaluated in

Telephone vs. Web Experiments

 Fully labeled, 1-step.

 5-category scales – 9 comparisons.  7-category scales – 6 comparisons.

Polar point labeled 1 step

c Don A. Dillman June 15, 2011 53  Polar-point labeled, 1-step.

 5-category scales – 13 comparisons.

 Branched 2-step scales.

5-Category Fully Labeled Scales

How satisfied are you with Washington State University as a place to go to school? Would you say you are Very Satisfied, Somewhat Satisfied, Neutral, Somewhat Dissatisfied or Very Dissatisfied? Telephone Survey

(10)

Mean Responses by Mode:

5-Category Fully Labeled Scales

4 5

Telephone Web

*

_*

* *

*

* *

c Don A. Dillman June 15, 2011 55 1 2 3 Q2 Q16 Q24 Q5 Q21 Q25 Q5 Q21 Q25 Satisfied/ Dissatisfied

Agree/Disagree Construct Specific

*p ≤ .05

5-Category Polar-Point Labeled Scales

On a 5-point scale where 5 means very satisfied and 1 very dissatisfied, how do you feel about the quality of INSTRUCTION in the classes you have taken at WSU?

Web Survey Telephone Survey

Web Survey 4 5 Telephone Web

* *

_*

*

**_{* *}**

*

_*

*

Mean Responses by Mode: 5-Category Polar-Point

1 2 3

Q2 Q4 Q6 Q7 Q16 Q24 Q5 Q21 Q25 Q18 Q19 Q9 Q17

Satisfied/Dissatisfied Agree/Disagree Extremely Likely/ Not Likely

Best Possible/ Worst Possible

*p ≤ .05

Summary of Findings on

Mode Effects in Scalar Questions

All six scale formats we tested produced a slightly more positive response on telephone than on web for the vast majority of test items.

• 5 and 7 category fully labeled

c Don A. Dillman June 15, 2011 58 • 5, 7, and 11 category polar point labeled • 7 category 2-step scales

See: Christian, Leah Melani, Don A. Dillman, and Jolene D. Smyth. 2008. “The Effects of Mode and Format On Answers to Scalar Questions in Telephone and Web Surveys.” In Lepkowski, et al. Advances in Telephone Survey Methodology. New York: Wiley-Interscience. pp. 250-275

Conclusions:

Interviewer Effects in

Scalar Questions

 Respondents answer scales more positively and are

more likely to select the extreme positive category on the telephone compared to web, mail, or face-to-face (when a showcard is provided).

 Finding a solution to scalar questions for which social

desirability seems not to apply may be as difficult as when it does apply.

Implications—Presence of Interviewer

 May need to avoid combining telephone and either web

or mail modes, especially for panel surveys, where attempts to measure change from Time 1 to Time 2 are being made.

 Statistical adjustment may be a possibility, but I leave

that to others to figure out.

(11)

However,

 What if we combined mail and web. They are both

visual, and measurement differences should be quite small for most questions.

 Also, if we can use Postal Service addresses without ,

names, maybe we can get people to go to the web.

 Let’s see what might happen if we do that.

2005-2009; Mail-only surveys with incentives added achieve reasonable response rates  Four general public surveys with 12 page questionnaires, $5 token cash

incentive with request, 3-4 follow-ups, respondent-friendly design. (Stern and Dillman, 2006; Smyth et al. 2010; Messer and Dillman, Forthcoming)

 2005 regional Lewiston/Clarkston social

participation survey, Telephone directories 66%  2007 Lewiston/Clarkston Quality of Life Survey

Addressed-based sample (ABS), 71%

 2008 Washington state-wide community satisfaction

Survey, ABS, most recent birthday 55%

 2009 State of Washington Household Economic Survey

Knowledgeable respondent, ABS 68%

Study Procedures

(Smyth, Dillman, Christian and O’Neill, 2010, Applied Behavioral Science)

 12 page questionnaire, 50 items, up to 80 responses (depending

upon branching); a 20-25 minutes survey

 Four contacts.  Pre-notice letter.

 Questionnaire (or web request).

 Thank-you post card.

 Replacement questionnaire (adjusted by treatment).

 $5 token cash incentive included with initial mail questionnaire or web request.

 Data collected November 7, 2007, to January 10, 2008.

Additional background on building mail

implementation systems. (Results of an Experiment, Contacts x Incentive Sent With the Request vs. Post payment).

Incentive

Incentive 11stst_Mailing_Mailing

(%) (%) 22ndnd_Mailing_Mailing (%) (%) 33rdrd_Mailing_Mailing (%) (%) 44thth_Mailing_Mailing (%) (%) No incentive 20.7 36.7 46.7 52.0 $1 C h 40 7 52 0 61 3 64 0 $1 Cash 40.7 52.0 61.3 64.0 $5 Cash 48.7 60.7 66.7 71.3 $5 Check 52.0 62.7 66.7 67.3 $10 Check 44.0 56.7 62.0 66.7 $20 Check 54.0 70.7 75.3 79.3 $40 Check 54.0 63.3 66.0 69.3 Promise of $50 23.3 43.3 53.3 56.7

Note: each treatment group contained 150 subjects (James and Bolstein 1992)

Treatments for Lewston/Clarkston

Study

1. Mail preference: Send mail questionnaire and mention web with initial request.

2. Mail preference: Send mail questionnaire but withhold mention of web for about two weeks. 3. Web preference: Web invitation with no mail

questionnaire, but explain that mail questionnaire will be sent in about two weeks.

4. Equal preference: Encourage response by either mode equally.

Sample Sizes

Treatment n

A. Mail Preference (mention web) 400

B. Mail Preference (no web mention) 400

C. Web Preference (mail in 2 weeks) 600

D. Equal preference (respondent’s choice) 400

TOTAL 1800

(12)

Tailoring to the survey topic and location and creating a unified design between mail and web

In the slides that follow…

 Pictures of location to be surveyed.  Creating common screens for mail and web.  Use of common branding for mail and web.

 Choice of stationary, envelopes and content based upon

rethinking of personalization strategies given that “name” could not be used.

Tailoring/personalizing the survey to

the location and population

 Photos taken of local landmarks, artwork, and

symbols to make survey recognizable and visually attractive.

Mail Questionnaire

 Use of “personalized” images to help respondents feel connected to

survey and to place an emphasis on study area instead of on survey source. Also used similar color and design to connect to web survey.

Unified design between mail (on left) and web (on right)

The opening web page

 Used an entry page similar to front cover of paper survey, still focusing on making the survey recognizable through familiar images.

Design of the web survey—focus on population not sponsor

 Question 2

 Similar design format to paper survey, and use of familiar image in upper left-hand corner of the screen.

(13)

Personalized Correspondence

 All letters used WSU

stationary.

 Photo of questionnaire

cover used to tie different elements together.

Exterior of Envelopes (2nd_{and 4}th_Contacts) Used WSU address labels.

Used a return label showing the photo from survey cover and the survey title to increase familiarity.

Initial vs. Final Response Rates

Treatments Initial Response Rate (%) Final Response Rate (%) Difference Mail preference (mention web early)

54 62 +8

Mail preference

(no web in early contacts)

62 71 +9

Web preference

(no mail in early contacts)

38 55 +17

Equal preference (choice) 52 63 +11 c Don A. Dillman June 15, 2011 75

Final response rate, paper preference

vs. web preference

Treatments Web (%) Paper(%) Total (%)

Mail preference (mention web early)

4 58 62

Mail preference 1 70 71

(no mention web)

Web preference 41 14 55

Equal preference (choice) 13 50 63

Response rate summary

 Postal DSF with addresses only (i.e., no names) can

produce reasonably high response rates—71% mail preference and 55% web preference.

 Using a postal letter to drive people to the web was quite

effective—41% responded by web.

 Adding the mail option after two weeks brought in

considerably more respondents—14%

 Adding a mail option to the web was more effective than

adding a web option to the mail –14% vs. 1%.

Is non-response error a problem?

 It does not help much to improve response rates if our

respondents are different from non-respondents on variables important to the study objectives.

 Thus we need to compare respondent characteristics on  Thus, we need to compare respondent characteristics on

web vs. mail within the different treatment groups.

(14)

Lewiston/Clarkston: Demographics in the web preference treatment by mode of response

Responded by Mail (n = 80) Responded by Web (n = 232) Difference Mean age 61.6 51.4 -10.2 < 50 yrs 26.6 41.7 15.1 > 65 yrs 48.1 17.0 -31.1 % Female 68.4 59.2 -9.2 % Female 68.4 59.2 9.2 % White 93.8 89.2 -4.6

% Employed (full time) 34.7 50.7 16.0

% Retired 52.0 25.6 -26.4

% With 4-year degree 13.9 30.8 16.9

% Married 42.9 74.0 31.1

% Widowed 24.7 2.2 -22.5

c Don A. Dillman June 15, 2011 79 Red indicates significant difference at .05 level.

Web respondents were quite different

from mail on 7 of 24 substantive

attitudes/opinions

Mail Web Diff.

% attached to the area 90.0 80.4 -9.6

% think willingness for community involvement

has increased 47.7 31.7 -16.0

% think fish population increased 18.9 38.0 19.1

% more internet use improves quality of life 43.4 62.1 18.7

% think more cell use improves quality of life 26.9 44.1 17.2

% think enviro. protection is too weak 16.3 30.7 14.4

% gray wolves not threat to domestic animals 2.5 9.9 7.4

But web + mail (all web preference respondents) quite similar to mail preference group.

Mail Preference (n=261) Web Preference (n=312) Difference Mean age 56.2 54.1 -2.1 < 50 yrs 39.4 37.8 -1.6 > 65 yrs 29.1 25.2 -3.9 % Female 52 7 61 6 8 9 % Female 52.7 61.6 8.9 % White 88.9 90.4 1.5

% Employed (full time) 43.2 46.6 3.4

% Retired 36.1 32.2 -3.9

% With 4-year degree 29.6 26.4 -3.2

% Married 56.0 66.0 10.0

% Widowed 11.7 8.0 -2.2

Summary: types of respondents by

mode



Web on its own brings in specific types of

respondents and leaves others out.

 Web and mail used together bring in a wider range of

respondents – comparable to mail used alone.p p

 Our best chance of reducing nonresponse error is to use

both.

What happens when respondents are

given a choice of which mode to use?

Treatments Web (%) Mail (%) Total (%)

Mail preference

(mention web early) 4 58 62

Mail preference

(no mention web) 1 70 71

Web preference 41 14 55

Equal preference (choice) 13 50 63

Response rates for the choice treatments are lower than for the mail preference (no mention) treatment but higher than for the web preference treatment.

Expansion to state-wide data

collection

 A similar model was used, i.e. personalize questionnaire

to the state with pictures.

 We pushed to the web by withholding mail.  Two tests in successive years

 Two tests in successive years.

(15)

Statewide Mail Questionnaire with map and regional pictures

 12” X 8.5” booklet with ‘personalized’ images to help respondents feel connected to survey and to place an emphasis on study area instead of on survey source. Also used similar color and design to connect to web survey. (Messer and Dillman, Forthcoming)

Web Questionnaire

● Designed to emulate the mail

questionnaire and appeal to the local population.

Response Rates From Mail only is higher than web + mail in three studies. But, 2/3of respondents were successfully “pushed” to respond via web.

Mail Only Web+ Mail

Lewiston/Clarkston

Rural Region Survey-2007 71% 41%+ 14% = 55% Washington Community

Survey-2008 57% 31%+ 15% = 46%

Washington Economic

Survey- 2009 _68% _34%_{+ 18% =}_52%

A few conclusions

 We no longer have a good sampling frame for telephone

surveys

 If we mix telephone (and web or mail) we stand a good

chance of obtaining different measurement chance of obtaining different measurement.

 Mail-alone may provide a good option, but

implementation methods affect response.

 Web + mail has lower response than mail-alone, but may

be a good option for future.

Your questions

 You may have questions about what I presented here, or

you may wish to go beyond what I have done here and ask about things not yet discussed.