Phase 2: Measuring control

While qualitative methods such as focus groups are well suited to understanding participants’ attitudes, thoughts and ideas in relation to the subject of research, they are less useful when it comes to investigating how these might be distributed amongst wider research populations. This is mainly due both to the small sample sizes that are usually involved, and the lack of representativeness in the sample of the characteristics of the research population¹². Generalizability of the results can be enhanced by selecting a method which allows for the inclusion of larger and more representative samples. The most common approach employed, and that which is employed here, is the social survey. In a social survey, multiple participants are asked to provide information on the same variables, usually with the aim of finding associations between variables (Vaus, 2013). For example, they may be asked to respond to a questionnaire consisting of specific questions, the answer to which is often constrained in some way (e.g. through selection from certain options, or a certain length of open text response). Because all participants respond to the same questions (or at least a sub-set of them depending on their response to other questions), the results obtained from different participants are directly comparable.

Furthermore, because the results are easily rendered into quantitative data, statistical approaches can be used to explore associations between sets of responses and sets of participants. So long as the sample has been selected in the appropriate way, these can be generalized with a specified degree of confidence to the research population.

The survey approach in itself only refers to the structured method of data collection across participants. What is perhaps more important is the mode of data collection.

One drawback of many questionnaire-based surveys as alluded to above, especially in the context of the acceptance on new technology and services, is that participants are being asked to report on their behaviours and behavioural intentions. There is evidence that such reports are not highly predictive of actual behaviours (Kormos & Gifford, 2014). One potentially preferable alternative would therefore be to systematically observe or otherwise measure actual behaviour. In the case of DSR offerings, for example, this could be achieved by tracking the

12 NB There are circumstances where focus group research can be more generalizable, such as to more specific research populations and where sampling is done in such a way as to approach representativeness.

71

number of people from an appropriately selected sample who sign up to different tariffs. From this it would be possible to tell the relative popularity of the offerings and also (assuming information was also collected about the participants) whether there were any demographic or other differences associated with acceptance.

However, as was made clear in chapter 2, the kinds of DSR offering of interest in this research are not currently available on the market in Great Britain. Offering them for trial purposes in the context of this research would not be possible due to lack of resources. Collaboration in existing trials was a possibility, and this is addressed later in this chapter. Even if it were possible to run a trial of actual product offerings, some form of questionnaire-based survey would still be required to explore perceived control and other factors that may lead to participants deciding whether or not to sign up to any given DSR offering.

Questionnaires may be delivered face to face, on the telephone, in paper format, online, or by some other means. The mode selected for this study was to deliver the survey online. This approach is now commonly employed in social research, due mainly to the ease and relatively low cost with which surveys can be administered to large sample sizes (Tourangeau et al., 2013). There are many previous examples of online surveys being used to investigate acceptance of new energy products, services and scenarios (e.g. Spence et al., 2015; Kranz, 2011; Stragier et al., 2010; Downing & iCaro Consulting, 2009). As well as being quicker and cheaper to deploy, online surveys also yield data which is already in digital form as input directly by the participants themselves, minimizing the probability of coding and other data processing errors. The participation burden is also relatively low – participants can complete the survey when they like (within a specified period), do not have to make appointments and can submit their response with the click of a button. Using online surveys have also been shown to reduce social desirability response bias compared to oral survey methods (Chang & Krosnick, 2010). For many people, the online environment is now more comfortable than, for example, being on the telephone or filling out a form by hand (Tourangeau et al., 2013).

Where criticisms are made of the use of online surveys as compared to other forms of survey, these largely focus on the sampling approaches which accompany them – particular with respect to online panel surveys. A panel consists of individuals who are recruited and retained by research organizations in order to complete surveys.

Firstly, a pre-requisite of being eligible to complete an online survey is that the participant has access to the Internet. This has been a greater problem in the past – it is now estimated that 84% of households in Great Britain have access to the

72

Internet, compared to 57% in 2006 (Office for National Statistics, 2014). However, it should be noted that penetration varies depending on the household make-up. For example, while 96% of households with children have Internet access, just 41% of households with a single adult over the age of 65 do (Office for National Statistics, 2014). While it is possible that people without Internet access at home could still have access elsewhere (such as in the workplace or at a public library), it is likely that Internet access introduces some bias into the sample that is suitable for participation in an online survey.

There are also known to be systematic differences between people who belong to online research panels and the general population. Fulgoni (2014) reports evidence that panel members tend to be heavier Internet users than people who are not survey panel members. This means that the results of such surveys may be more representative of people with relatively high Internet usage compared to the general population. However, these data were collected in 2006, and as the Internet access figures shown above demonstrate, much has changed since then. Response bias may also be an issue, which can occur when the people who respond to a call to participate in a survey differ in important ways from those who do not. However, the polling company YouGov now report higher response rates for online panel surveys than telephone surveys (Eastbury, 2014), suggesting this is now less of a problem for this mode of delivery. Overall, it is considered that online panel surveys can provide a cost effective means of data collection suitable to contribute to answering the research questions posed here. Nevertheless, limitations have been highlighted and the results must be considered in this context.

Having established the online survey as the mode of data collection, it is important to address another aspect of the research questions – that is, the relative control expectations in, and acceptability of, different approaches to DSR. In particular, the interest is in how differences in specific characteristics of DSR offerings (e.g.

whether the response is manual or can be automated) affects acceptability. To explore this, an experimental design was required, involving random allocation of participants to different groups which are exposed to conditions which vary in controlled ways (adapted from Vaus & Vaus (2001: 48)). The use of an experimental approach in survey methodology is known as a survey experiment, or

‘a study that manipulates some feature of a survey protocol’ (Marsden & Wright, 2010: 860). Survey experiments are usually run by assigning participants into groups and asking each group to complete surveys that differ from each other in controlled ways. This often takes the form of variations in information which is

73

provided to participants on which they are later asked to base their responses. For example, Walter (2014) tested the acceptability of wind power projects in Switzerland by describing them in different ways to different experimental groups.

One of the variations was as follows:

Level 1: A citizens’ vote showed that a majority of citizens in your municipality opposes the wind energy project.

Level 2: A citizens’ vote regarding the wind energy project has not been held.

Level 3: A citizens’ vote showed that a majority of citizens in your municipality supports the wind energy project.

By comparing the responses of the different groups, so long as they have been appropriately sampled and all other factors are held constant, it is possible to attribute significant differences in results to the change in information provided between groups. It is this ability to allow statements about causation, rather than simply association, that makes the experimental approach powerful. In the current research different experimental groups were exposed to descriptions of tariffs with characteristics of interest to the study that differed in controlled ways. Specifically this included static and dynamic time of use tariffs, with and without the option of automated response, and direct load control. The precise tariff details and the rationale by which they were arrived at are set out in chapter 5. The Australian study by Stenner et al. (2015) described in chapter 2 applied a very similar approach in the context of DSR tariffs, using a 3x6 experimental design which showed participants one DSR tariff from a choice of six at random, with the additional random inclusion of either a bill protection option of provision of an automated device to response to DSR signals.

While survey experiments are increasingly widely employed by researchers, there are also a number of questions over their use. The main challenge is in assessing their external validity, or the extent to which the results would be relevant in a real-world (rather than survey) setting. Barabas & Jerit (2010) compared the effect of giving different information about government announcements in the context of a survey experiment to the effect of actual announcements, as also measured by surveys. They found that the effects observed in the real-world setting, while not fundamentally different, were smaller than those garnered in the survey experiment.

In this case they attribute the difference to the level of coverage the announcements received in the media, hypothesizing that exposure in the natural experiment was

74

not as large as where people where specifically asked to read the information in the survey experiment. In the context of the current research this is considered to be less of an issue since it is not testing knowledge and how this relates to attitudes, but explicitly sets out to measure acceptability based on presented information about a subject which participants are likely unfamiliar.

Gaines et al. (2007) highlight a number of other critiques of the survey experiment method. Firstly, because survey experiments present information and ask participants to respond in a short period of time, the responses received may be based on little reflection and may not endure for very long. For example, a political advertisement worded in one way rather than another may prompt a survey participant to say they would vote one way rather than another – but would this effect endure long enough to actually affect voting behaviour if they were to see that advertisement in the real world? Both Druckman & Nelson (2003) and Mutz &

Reeves (2005) found in follow-up studies that effects observed in original survey experiments did not last more than a few days to weeks. However, in the case of product or service purchase such as signing up to a DSR tariff, it is not unreasonable to expect that the decision would be made in close temporal proximity to being exposed to information about the tariff (e.g. through reading information on a website description), thereby minimizing this problem.

Another issue with survey experiments highlighted by Gaines et al. (2007) is that they often fail to include a true control group. For example, Kinder & Sanders (1996) studied the effect of how two different framings of affirmative action affected participants’ attitudes towards it. However, they did not include a control group where attitudes were measured without any framing. While significant differences were found between both the framings, the researchers could not say whether the individual framings had any effect simply on existing attitudes (i.e. where there no framing). There is a trade-off to be made here between the extra insight that including a control group may bring, and the cost of either increasing the sample size or reducing the statistical power of the comparison by dividing the sample into smaller groups. The current research did not include a non-DSR (i.e. standard flat rate) electricity tariff as a control. The reason for this is that the technical and economic drivers for moving to wider use of DSR are clear, and policy is expected to make this likely (see chapter 2). The primary interest is therefore between different DSR options, rather than in comparing DSR to the current situation. While including a flat-rate control group would be expected to provide an useful comparison (as indeed it did in the case of Stenner et al. (2015)), the value of this

75

was considered to be outweighed by being able to test five different DSR tariffs as opposed to just four.