Looking Beyond Quality Differences: How Do Consumer Buying Patterns Differ by Sample Source?

(1)

Looking Beyond

Quality Differences:

How Do

Consumer Buying Patterns

Differ by Sample Source?

The CASRO Panel Conference February 24th _{& 25}th_{, 2010}

(2)

Background

Since their proliferation began in the 1990s, opt-in, online panels have

been lab- and road-tested to verify their suitability for use in commercial

research.

Differences in results from online versus phone data collection Examples of non-coverage error with online samples

Survey-taking patterns that evidence mindless responding

More recently, debate has turned to the replicability of responses patterns

across opt-in panels.

Gittleman (2009) acknowledged the non-probablistic nature of data of opt-in panels and supported a focus on consistency of results as a quality measure.

Examination of the quality and consistency of data from opt-in panels has

also recently focused on demographic variables and other variables that

have externally verifiable measures.

Yeager, Krosnick, Chang, Javitz, Levendusky, Simpser, and Wang (2009) concluded that non-probability, online sample surveys are always less accurate than probability surveys such as RDD phone surveys.

(3)

Background

Less published work has examined the extent of

consumer-behavior

disparities in data collected using different online sample sources.

Therefore, the current research explored how reported consumer

behavior and related variables differ by online sample source.

(4)

Do different online sample sources

evidence materially different

patterns of consumer behavior?

Key Research Question

(5)

Research Method

This exploratory research examined data from the following samples:

Two opt-in panel samples

Two river samples

Two social networking samples

Analysis will determine whether material differences exist in…

Product category purchasing

Brand awareness Brand usage

Demographic variables

Cheating and related behaviors

(6)

Facebook Ad

(7)

Facebook as a Sample Source

Ads are sold through an auction process.

Facebook offers a recommended bid range, and it changes daily.

Higher bidders enjoy more impressions or exposures.

Facebook reports that on average, .04% to .08% of people exposed to an ad on Facebook click on it.

Facebook users can be targeted by age, gender, geography or birthday. Advertisers can control costs by setting a maximum daily budget. On average, Burke paid $1.21 per click.

Ads must be approved by Facebook, and approval typically takes between six and 24 hours.

(8)

Survey Landing Page

(9)

Click Rates by Age/Gender Groups

Age/Gender Group

Ad Click Rate

65+ females 0.043% 65+ males 0.037% 35-44 females 0.034% 45-64 females 0.033% 45-64 males 0.031% 18-34 males 0.029% 35-44 males 0.028% 18-34 females 0.015% Average 0.031% © 2 0 1 0 B u rk e I n c o rp o ra te d . A ll r ig h ts r e s e rv e d . 9

(10)

Demographic Profiles

and Panelist Behaviors

(11)

Demographics

Panel A Panel B River A River B Social Networking Sample Facebook Female 44% 35% 51% 28% 49% 34% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Panel A Panel B River A River B Social Networking

Sample

Facebook

Employed Full Time

52% 52% 54% 52% 39% 52% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Sample

Facebook

(12)

Demographics

Panel A Panel B River A River B Social Networking Sample Facebook Age $47,630 $49,877 $51,938 _$46,293 $49,738 $39,688 $0 $10,000 $20,000 $30,000 $40,000 $50,000

Panel A Panel B River A River B Social Networking Sample Facebook Household Income 29% _24% _23% 20% 29% 11% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Sample

Facebook

(13)

Panelist Behaviors

Sample

Facebook

Hours Online Weekly

6 8 3 8 4 2 0 1 2 3 4 5 6 7 8 9 10

Sample

Facebook

(14)

Fraudulent Responding

and Mental Cheating

(15)

Incidence of Fraudulents

Panel A Panel B River A River B Social

Networking Sample

(16)

Unusually Fast Survey Completion

Facebook Average Survey Length: 27 minutes

(17)

Incidence of Mental Cheaters

(18)

Measures Related to

Consumer Behavior

(19)

Category Purchasers – Beer

(20)

Purchase Frequency – Beer

Once a week or more often 2-3 times per month Once a month

Every two months Every 3-4 months Less often than every 3-4 months

Volume

(21)

Brand Awareness – Miller Beer

© 2 0 1 0 B u rk e I n c o rp o ra te d . A ll r ig h ts r e s e rv e d . 21 60% 60% 59% 63% 49% 54% 40% _37% _39% 33% 41% _27% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Facebook

(22)

Brand Usage – Miller Beer

(23)

Category Purchasers – Gasoline

(24)

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Facebook Social Networking Sample River B River A Panel B Panel A

More often than twice a week Twice a week Once a week

Every 2-3 weeks Once a month Less often than once a month

Purchase Frequency – Gasoline

(25)

Brand Awareness – BP

© 2 0 1 0 B u rk e I n c o rp o ra te d . A ll r ig h ts r e s e rv e d . 25 40% _36% 44% 46% 33% 25% 45% 47% 37% 38% 37% 57% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Facebook

(26)

Brand Usage – BP

(27)

Category Purchasers – Drug Stores

(28)

Purchase Frequency – Drug Stores

Once a week or more often 2-3 times per month Once a month

(29)

Brand Awareness

(30)

Category Purchasers – Insurance

(31)

Purchase Volume – Insurance

(32)

Brand Awareness

(33)

Brand Usage

(34)

Category Purchasers – Cellular Telephone Service

(35)

Average Monthly Bill – Wireless Telephone Service

(36)

Brand Usage

(37)

Category Purchasers – Credit Cards

(38)

Average Monthly Expenditure – Credit Cards

(39)

Brand Awareness – Visa

© 2 0 1 0 B u rk e I n c o rp o ra te d . A ll r ig h ts r e s e rv e d . 39 83% _80% _80% 86% 74% _69% 17% _19% _20% 14% 23% 16% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Facebook

(40)

Category Purchasers – Notepads

(41)

Brand Usage – Post-It

(42)

Category Purchasers

Panel A Panel B River A River B

Social Networking Sample Facebook Beer Disposable Diapers Cellular Service Facial Tissue Fast Food/Casual Dining Donut or Coffee Shops Batteries Credit Cards Drug Stores Non-401K Investments Insurance Notepads Car Rental Personal Computers Gasoline

(43)

Purchase Frequency/Volume

Social Networking Sample Facebook Beer Disposable Diapers Cellular Service Facial Tissue Fast Food/Casual Dining Donut or Coffee Shops Batteries Credit Cards Drug Stores Non-401K Investments Insurance Notepads Car Rental Personal Computers Gasoline

(44)

Brand Awareness

Social Networking Sample Facebook Miller Beer Energizer VISA CVS Walgreens Fidelity Investments Allstate Geico Post-It Avis Dell Huggies HP Gateway BP Pampers Sprint Verizon Kleenex Puffs Burger King

(45)

Brand Awareness

©201 0 B u rk e I n c o rp o ra te d . A ll r ig h ts r e s e rv e d . 45 88% 84% 84% 84% 85% 84% 80% 85% 85% 82% 82% 97% 93% 93% 95% 97% 96% 93% 98% 98% 96% 97% 0% 20% 40% 60% 80% 100% Dunkin's Donuts HP Geico Energizer VISA Kleenex Post-it Burger King Dell Sprint Verizon 74% 82% 83% 92% 92% 86% 81% 80% 87% 84% 81% 60% 70% 76% 92% 92% 88% 87% 87% 94% 93% 90% 0% 20% 40% 60% 80% 100% Fidelity Investments BP Avis Pampers Huggies CVS Puffs Gateway Walgreens Allstate Miller Beer

(46)

Brand Usage

Social Networking Sample Facebook Miller Beer Energizer VISA CVS Walgreens Fidelity Investments Allstate Geico Post-It Avis Dell Huggies HP Gateway BP Pampers Sprint Verizon Kleenex Puffs Burger King

(47)

Summary

Demographic differences exist based on sample source, with the most

notable differences existing between the social networking samples and the

other sample sources.

Respondents from the Social Networking Sample tended to be younger, they are less likely to be married, and they are more likely to be female.

In contrast, respondents from Facebook are more likely to be male, are less educated, and they have lower household incomes.

Consistent with previous research, rates of fraudulent responding and

mental cheating do differ by panel source.

Of particular note: The Social Networking Sample and the Facebook sample differ materially on both demographic and fraudulent/cheating behavior.

Facebook generally resembles other river samples in that fraudulent/cheating behaviors are minimal.

However, using a Facebook ad as the sole sample source and as executed

for this research is not a tenable approach for most commercial research.

(48)

Production Data from the Facebook Sample

Ad click rate

10,509,614 impressions generated 3,233 clicks to the survey landing page .031% ad click rate

Survey start rate

264 people of the 3,233 people who clicked on the ad answered at least one survey question

8% start rate

Survey completion

70 of the 264 who started the survey finished it 26% completion rate

CPI for completes: $56

CPI for partial interviews (though 15 minutes of survey): $26

Interviewing spanned 20 days, albeit over the end-of-year holidays

(49)

Summary

While some variation does exist in incidence of category purchasing,

aided and unaided awareness, and brand purchase, the differences tend

to be relatively small.

Significant differences on omnibus tests were not found across the vast majority of consumer-behavior/marketing-related variables examined in this research.

However, many of the differences that do exist involve the Social Networking Sample and the Facebook sample.

Sometime these two sample sources differed in the same direction, and sometimes they did not.

(50)

Implications

More research is needed to determine whether these findings are

replicable in lower incidence product categories.

For many product categories, sample quota controls based on

demographic variables must be applied because those variables have

known relationships with attitudes that impact consumer behavior.

Because samples drawn from social networks tend to produce different

results from those obtained from traditional opt-in and river samples,

organizations should do due diligence to determine where their use is

most warranted.

The research industry would be well served to look for ways to introduce

additional scientific rigor into sample blending.

(51)