Second method: online panels

At this stage, given the very small and insufficient response rate, an alternative data collection method had to be found, otherwise, the whole study would have to have been abandoned. Considering the existing constraints, including the limited time of a PhD project, very limited budget, high non-response rate of traditional survey methods, it was decided that the use of online panels to supplement the existing responses was the best (and only feasible) form of action. As will be explained below, the use of online panels guarantied the collection of sufficient good quality data in a short amount of time for a small cost.

Online panels are defined by ISO 26362: Access Panels in Market, Opinion, and Social Research as ‘a sample database of potential respondents who declare that they will cooperate with future (online) data collection if selected’ (see in Callegaro and Disogra, 2008, p. 718). These potential respondents register their interests with market research companies and are then contacted on a survey basis. They receive monetary compensation provided they successfully qualify for the individual study and fully complete the questionnaire.

Before detailing the choice of the company and the data collection, some issues need to be clarified. Indeed, notwithstanding a rising use of online panels by academic research (Baker

et al., 2010; Ansolabehere and Schaffner, 2014), the general questions of payment of respondents, representativeness of the sample and quality of the data need to be addressed.

Monetary incentives and their impact:

The use of incentives has received a great deal of attention in the literature, which generally views them positively. The relation between survey participation and incentives is enlightened in the context of economic-exchange theory and social-exchange theory (Dillman, Smyth and Christian, 2009, 2016; Michle et al., 2016). Both theories reach the same conclusion on the positive impact of incentives, however, their justifications are

74

surveys of various monetary incentives, Warriner et al. (1996) demonstrate that only cash incentives have an impact on response rates. Donations to charity and the possibility of winning a lottery prize do not have an effect on the response rates.

Some scholars in Economics and Psychology warn that monetary incentives may have a negative impact on study participation and response rates (Lacetera and Macis, 2010). In particular, Sauermann and Roach (2013, p. 275) explain that ‘contingent pay may undermine actors’ intrinsic or social motivations to engage in a task, potentially resulting in a negative net effect. Such “motivation crowding-out” may occur if individuals feel that incentives are controlling, if pay is interpreted as a sign that the task cannot be “fun”, or if pay leads actors to focus their cost-benefits analysis narrowly on financial aspects’.

Overall the literature views monetary reward as a positive incentive to increase response rates. Online panels are a consequence of these findings in that they try to incentivise people by rewarding them with a small amount of money.

Question of the representativeness of the sample

Traditionally, surveys rely on probability sampling which means that each respondent in the population has the same (known) probability to be included in the sample as any other member of the population (Vehovar, Toepoel and Steinmetz, 2016). Probability sampling guarantees the representativity of the sample and allows generalisation of the results to the population. However, when it is difficult to know about the population, this makes probability sampling impossible, and other sampling technique need to be employed (Uprichard, 2013). Online panels deviate from probability sampling as market research companies use non- probability sampling to build their samples with ‘snowball sampling, banner ads, direct enrolment’ techniques (Callegaro et al., 2014, p. 24). To what extent is this deviation detrimental to the quality of the results?

Before answering that question, Uprichard (2013, p. 7) warns that in any case ‘there needs to be greater acknowledgement that any sample will necessarily fall short of the ideal simply because the praxis of sample design is limited by the possibilities set by the three-pronged configuration of issues, which itself becomes clearer in the context of the research itself’. Some studies have compared results obtained from probabilistic and non-probabilistic samples in order to discover whether the two sampling methods yield similar results. Comparing results from surveys on climate change and the Kyoto Protocol, Berrens et al. (2003) conclude that there is no difference in representativeness and findings between probability and non-probability sampling. A similar conclusion was reached by Sanders et al.

(2007) while comparing survey results from probabilistic and non-probabilistic samples as part of the 2005 British election study.

Quality of the data in the case of online panels:

Following the use of incentives, the question of the quality and reliability of the answers has been raised. Are these respondents only completing surveys to receive money? If so, to what extent are their answers impacted? To what extent does monetary payment attract specific types of respondent leading to the over-representation of specific categories of the population over the rest of the population? Following the rising use in online panels, these concerns have been extended to this mode of data collection.

The main concern relates to what has been referred to as ‘professional respondents’; respondents who answer as many surveys as possible in order to receive monetary compensation (Comley, 2005; Hillygus, Jackson and Young, 2014). It is assumed that professional respondents are only motivated by the incentives, compromising the quality of the data (Matthijsse, De Leeuw and Hox, 2015). However, the literature has offered contradictory findings. It should also be noted that so far the literature has failed to agree on a definition of a professional respondent, more specifically on the threshold transforming a survey taker into a professional respondent (Hillygus, Jackson and Young, 2014). Some argue that professional respondents answer questionnaires more quickly than new respondents, and randomly pick answers (Baker et al., 2010). Garland et al. (2012) claimed that professional survey takers use ‘don’t know’ answers more frequently than non-professionals. It is also argued that professional respondents would be more likely to satisfice. Krosnic (1991), the first one to apply the term ‘satisfice’ to survey respondents’ behaviour, defines it as the various strategies a respondent puts in place to limit the cognitive effort in answering a survey. For instance, a weak satisficing strategy is the selection of the first plausible response among a list of alternatives (Krosnick, 1991). Satisficing would therefore be stronger in professional respondents due to the high number of surveys taken.

76

differences in answers between professional and non-professional survey takers. Finally, Ansolabehere & Schaffner (2014) analysed the results of a similar survey issued at the same time over the Internet (opt-in online panel), by telephone using random digit dialing and by mail (based on a national sample of residential addresses). They found that ‘comparing the findings from the modes using a Total Survey Error approach, (…) a carefully executed opt- in Internet panel produces estimates that are as accurate as a telephone survey and that the two modes differ little in their estimates of other political indicators and their correlates’

(Ansolabehere and Schaffner, 2014, p. 1). In other words, professional respondents do not threaten the quality of the data.

The data obtained from online panels offers a similar quality than traditional survey methods. It can be asked though: who are these participants taking part in online panels? Four categories of online panel respondents have been identified: altruistic nonprofessionals, semi- altruistic, semi-professionals and professionals (Comley, 2005; Matthijsse, De Leeuw and Hox, 2015). Online panels are not only composed of professionals survey takers, altruistic and curious people also join these panels. A study of 90% of the respondents taking part in online panels in the Netherlands reveals that there are no differences in demographic characteristics between professionals and altruistic respondents (Matthijsse, De Leeuw and Hox, 2015). Furthermore, the study showed that monetary reward was not the only incentive for professional survey takers, fun had to be present too (Matthijsse, De Leeuw and Hox, 2015).

In conclusion, , the literature does not provide sufficient evidence to substantiate doubts about data quality obtained from online panels. The presence of professional respondents, for instance, is not a legitimate threat to the quality of the data. Ultimately, the researcher has to exercise a similar level of caution and common sense as with traditional survey methods.

Selection of the market research company: There are several market research companies providing data collection services. I contacted four that are well-known and whose services have been used by academics for their research in social science and behavioural economics. The selection was based on three criteria: a) the market research company needed to have participants in France and the UK; b) the presence of HR managers in their database; c) the total cost needed to be within the allocated budget of £1170.

Administration of the online panels: Contrary to the self-administered questionnaires, payment had to be made to the respondents. With the selected market research company,

participants are rewarded £9 per fully answered questionnaire. The budget of £1170 allowed the collection of a total of 130 questionnaires.

In practice, two specialty panels were used. A specialty panel is defined as a group of people with specific characteristics/profession: one panel was located in France and one located in the UK. Each panel was composed of current HR managers, irrespective of the industry. For each panel, the market research company sent the questionnaires to every French and UK HR manager in their database and kept sending them until the agreed outcome of 130 completed questionnaires was reached.

Administering the questionnaire by a market research company offered several advantages. First, it is a time efficient technique, as it took only 15 days to collect responses. Second, the objective of reaching HR managers is achieved by relying on a specialised panel. Third, by adding an introductory screening question to the questionnaire guaranteed that the potential respondent matched the criteria of being an HR manager.