Research Design. 2014),

Capturing the Moment: A Feasibility Test of Geo-Fencing and

Mobile App Data Collection

Dan Seldin, Gina Pingitore, and Laurie Alexander, J.D. Power;

Chris St. Hilaire, MFour Mobile Research, Inc.

Introduction

There is no doubt that consumer mobile device usage is growing exponentially. In the United States, more than 50% of adults own a smartphone,1 with 65% of all email accessed via mobile devices.2 Checking their wireless phones

approximately 158 times a day, or every 6.5 minutes,3 consumers have integrated mobile devices into their day-to-day activities. During the next few years, a continued migration from desktop computers to mobile phones and tablets is anticipated, with some 2.5 billion people owning a smartphone by 2017.4 According to comScore, 18% of consumers between 18 and 34 years-old use mobile-only platforms, compared with 5% of those who are 35 to 54 years-old, and 3% of those 55 years and older.5 This shift is also impacting the way consumers participate in market research, with 30% of online surveys completed via mobile Web.

The number of apps has also grown exponentially since 2008, when only 800 apps were available in the iTunes App Store. In 2013, there were more than a million available apps. Increased consumer adoption of mobile applications and the increased number of features and functionality available on mobile apps create fresh opportunities to engage consumers in market research. The variety of information that can be captured beyond the standard survey format provides a powerful opportunity to augment typical survey questions. GPS-based location tracking and multimedia capabilities allow consumers to tell us about their experiences “in the moment,” and mobile app capabilities allow us to capture mystery shop-like data on the fly with relatively low overhead vs. traditional mystery shop programs.

However, just because researchers see the tremendous potential in this data collection methodology doesn’t mean consumers will. That is, will consumers really allow an app to access their location information so we can track their whereabouts? Which consumers are most likely to engage in mobile app market research? Do push notifications of survey opportunities increase response rates? Do different question formats cause a bias in the results? Are consumers willing to complete a survey and document their experience via multimedia while they are in the moment?

In prior research, J.D. Power examined the feasibility of collecting customer satisfaction data via mobile-enabled websites and text messaging.6 Both of these methods were shown to be viable data collection methods, although differences in respondent profiles and scoring were observed when compared with traditional online research. Whereas these types of mobile data collection, along with traditional online research, can only be conducted after the experience of interest has occurred, using mobile apps with geo-location technology provides us with the opportunity to catch respondents in the moment and obtain other feedback that might otherwise not be possible to collect, such as actual wait times and photos of “things gone wrong.”

1

Pew Research Center’s Internet & American Life Project, Mobile Technology Fact Sheet, www.pewinternet.org/fact-sheets/mobile-technology-fact-sheet/, Dec. 27, 2013.

2

Movable Ink, “US Consumer Device Preference Report: Q4 2013, The Mobile Takeover.”

3

MFour Mobile Research, “MobiAd,” Monthly Newsletter, June 6, 2013.

4

Emarketer, “Smartphone Users Worldwide Will Total 1.75 Billion in 2014,” www.emarketer.com, Jan. 16. 2014.

5

comScore, “2013 Mobile Future in Focus” (white paper), February 22, 2013.

6

More recently, research conducted by public opinion pollsters Glen Bulger and Trip Mullen7 demonstrated app-based research as a viable and advantageous tool for political pollsters. The study consisted of interviews of 800 U.S. adults over a 2-day period and inquired about a wide range of current political topics. The team found two important advantages of app-based research:

1. The app-based communication medium enabled easier and more cost-effective access to younger voters and minority voters, two groups pollsters have struggled to reach through traditional telephone polling in recent years.

2. The push notification feature allowed researchers to collect hundreds of responses during a span of only a few hours.

While we are able to draw some conclusions from this research, studies conducted on the actual use of mobile apps for market research are missing. Understanding the need to reach consumers where and how they want to be reached, J.D. Power conducted research to explore how consumers respond to surveys deployed via mobile devices while they are in the moment of an experience. The feasibility of using GPS technologies and mobile app surveys to capture consumer feedback was tested across several mobile apps.

Research Design

In January 2013, J.D. Power conducted two studies to test the feasibility of using mobile apps with geo-location to capture consumer feedback for low- and high-incidence events. For the first study, we collaborated with MFour to survey consumers who were shopping for a new vehicle (low incidence). The goal was to assess the response rate funnel and understand where in the process potential respondents were lost. Respondents were first pre-qualified via traditional online surveying as being likely to shop for a new vehicle within the next 2 months (10,033). Among those potential respondents, 9,073 agreed to install the geo-location app and receive push notifications on their mobile device. Respondents then had to be at a dealership for at least 15 minutes in order to receive a survey alert. The questionnaire consisted of 15 items covering the reasons for visiting the specific dealership and what occurred during the visit. As an incentive, respondents received $5 for completing the questionnaire. Respondents validated their location by photographing the dealership.

The goal of the second study also tested the feasibility of capturing consumer feedback via a mobile app, but using a high-incidence event. Moreover, the test aimed to understand the viability of capturing this feedback during the actual experience (mystery shop condition) vs. immediately following the experience (satisfaction survey condition). Additionally, we assessed the impact of using different response options and compared the results with traditional online research. Through our collaboration with uSamp, Research Now, and MFour, three mobile survey apps were used to capture information from consumers who were identified by GPS technologies as being at a grocery store. Respondents who were just beginning to shop were asked to complete a brief mystery shop survey, while those who were already in the process of shopping or who had just completed their shopping were asked to complete a brief satisfaction survey about their grocery shopping experience.

Within the mystery shop condition, respondents were asked to provide feedback that is difficult to capture after the occurrence of the experience of interest, such as finding an item in the store or timing how long it took to checkout. In all, respondents completed nine questions/tasks. While we wanted to capture in the moment feedback, questions about state of mind and emotions were excluded so we could directly compare results with the data captured via the satisfaction survey. In total, 697 mystery shop evaluations were completed.

7

Glen Bolger and Trip Mullen, “There’s an app for that,” Campaign & Elections Magazine (January 7,

Within the satisfaction survey condition, respondents rated various aspects of their experience (16 total

questions/tasks), such as how long it took to checkout, the professionalism of the staff, and the layout and design of the store. There were 1,497 observations in the satisfaction survey condition. In both the mystery shop and

satisfaction survey conditions, respondents were randomly assigned to one of the three response options—textbox, drop-down, or scroll—to view the questions and were asked to validate their location by providing a photograph of either their receipt or filled shopping cart. As in the first study, respondents received $5 for participating.

Sample Sizes

Condition Textbox Drop-Down Textbox Scroll Textbox Drop-Down Bubbles

Mystery Shop 126 139 18 162 115 137 - 697

Satisfaction Survey 230 236 140 627 132 132 705 2146

TOTAL 356 375 158 789 247 269 705 2843

Results from each of the mobile app tests were then compared with the results from a traditional online grocery store satisfaction study. Online panelists from SSI (n = 705) completed a full grocery store experience survey online 1 month after the mobile app surveys were fielded to compare respondent profiles and survey results.

Results: Study One

To assess the feasibility of using a mobile app to survey for a low-incidence study, we first examined the entire process, from survey qualification to completion. Among the 10,033 respondents who were shopping or planning to shop for a new vehicle during the next 2 months, 10% were lost because they did not want to install the MFour mobile app with geo-location services. The majority (53%) of potential respondents were lost because they were not geo-located at a dealership during the 2-month fielding period. Among those who qualified and began the survey, 22% (the second-largest loss of potential sample) were terminated because they were at the dealership for reasons other than shopping for a new vehicle. Another 11% qualified, but did not complete the survey, yielding 2% of the total possible sample completing the survey via the mobile app.

Qualified via Screener

Opted-in for Geo-notification

Visited a Dealership

Started Survey

Qualified as a Shopper

Completed Survey

Results: Study Two

In the second study, response rates were examined for a high-incidence event, grocery shopping. At the time of fielding, uSamp was unable to capture the number of push notifications; thus, completion rates were calculated based only on the Research Now and MFour mobile apps. Among respondents who received a push notification, 24% of Research Now and 23% of MFour respondents completed the survey. Completion rates were 12 percentage points higher when the survey was completed immediately following the shopping experience (satisfaction survey condition) vs. during the shopping event (mystery shop condition).

Additionally, mystery shop surveys took approximately 9 minutes longer to complete than satisfaction surveys, primarily because respondents in the latter survey condition were completing tasks while they shopped. Satisfaction surveys with textboxes took about 1.5 minutes longer to complete than drop-down response options. Moreover, incompletion rates were also 6 percentage points higher for the textbox vs. drop-down/scroll options.

Survey Completion Time (min.) Textbox Drop-Down/Scroll TOTAL

Mystery Shop 17.7 17.7 17.7

Satisfaction Survey 9.6 8.1 8.9

Finally, more respondents said the satisfaction survey was “very easy” to complete than did mystery shop respondents, although no significant differences were found among textbox, scroll, and drop-down response options.

Profile Comparisons

No significant differences were found when comparing the profiles of those who did vs. those who did not start the survey. However, differences were found between the respondent profiles across the vendors included in the study, particularly between Research Now respondents and those from the other two vendors. Research Now responders were older than uSamp and MFour responders (16% between the ages of 45 and 54 years vs. 10%, respectively); had higher levels of education (64% with a 4-year college degree or higher vs. 43%, respectively); and were more often male (52% vs. 35%, respectively). It should be noted, however, that these differences mirror those typically found when panel profiles are compared among traditional online respondents.

Next, we compared mobile app responders with each panel’s overall profile as well as with traditional online grocery store survey responders. While mobile app responders did not differ in gender, education, ethnicity, or rewards program membership from traditional responders, they did differ in age. On average, mobile app responders were 35 years-old, compared with an average age of 40 among traditional online panel responders.

Comparing the Data

While obtaining respondents with generally similar profiles is important, it does not mean the results will be the same between traditional online and mobile app data collection. To make this comparison, we used the Customer Satisfaction Index approach developed by J.D. Power, which takes empirically derived estimates of importance weights and applies them to the corresponding experience ratings to yield a Grocery Store Customer Satisfaction Index that ranges from 100 to 1,000 points. The importance weights derived from the traditional online data were applied to both the traditional online and mobile app data to allow for a direct comparison of index scores. Grocery Store Customer Satisfaction Index scores were found to differ by vendor, with the highest scores among Research Now respondents (783) and the lowest among MFour respondents (722). The combined mobile app scores were approximately 40 points lower than traditional online panel scores. These differences held when respondent profile was controlled.

As part of Study Two, we also assessed the impact various response option types (textbox, scroll, and drop-down) had on scores and found that textbox scores were approximately 10 points higher than drop-down and scroll scores.

In addition to understanding how this methodology impacts satisfaction scores, we also wanted to understand how it impacts the accuracy of self-reported operational metrics, such as time to checkout. We compared the reported

checkout times (minutes) of all three respondent groups: those who took the survey while shopping; those who took the survey directly after shopping; and those who responded online after shopping. Checkout times were found to increase as the time span between the experience and surveying increased. When respondents timed the checkout during the process, it took 5 minutes. Respondents who completed the survey directly after shopping experienced a 10-minute checkout, while it took an average of 12 minutes for traditional online respondents to checkout. When comparing self-reported checkout times across the different response options tested, checkout times tended to be higher and to have more outliers when assessed via a textbox vs. a drop-down or scroll option.

Finally, we wanted to assess the feasibility of capturing additional qualitative data that is not possible to capture via traditional online research. Within traditional online studies, the primary source of qualitative information is verbatims, which are completed by approximately 40% of respondents. Mobile app data collection allows us to collect multimedia qualitative information, such as voice recordings, pictures, and videos of the experience of interest. Mobile app

respondents were more than willing to provide this type of feedback and sent, on average, one optional picture/video of something in the store to highlight their experience.

Conclusions

Overall, our results provide a good first step in understanding how mobile app data collection can be used either as a primary or supplemental form of research. Additionally, we gained insights into the willingness of consumers to engage in this type of research and how these respondents compare with typical online panel respondents. Further, we were able to assess different response options—such as textbox, scroll, and drop-down—and the impact each method had on the resulting data. Based on our findings, mobile app surveys with geo-fencing provide a means for capturing specific in-the-moment feedback, which would otherwise be impossible to obtain following the occurrence of the event of interest. That is, overall completion rates for both low- and high-incidence events were comparable with traditional online completion rates; the mobile surveys were perceived to be easy to complete; such operational metrics as time to checkout were more accurately captured; and respondents were willing to provide qualitative, multimedia feedback. Of course, caution should to be taken when using mobile app data collection with geo-fencing, as we also found some notable differences in comparison with traditional online data collection—specifically, mobile app respondents were younger and overall satisfaction scores were lower. Moreover, our research indicates that survey format impacts not only the data captured, but also the response rates. We found that drop-down/scroll response types were completed more quickly, had fewer dropouts, and yielded better data than textboxes.

Additional research is warranted to replicate our findings with other high and low-incidence events. More testing is also needed to understand how mobile app data collection can be leveraged as an alternative to traditional mystery shops. Future research should assess how many questions respondents are willing to answer and how the data compares with traditional mystery shop evaluations. Further research is also needed to understand how to best optimize the survey experience, both the look and feel and response-option formatting, in order to help increase completion rates and reduce some of the differences with traditional online data collection.