Experimental and Questionnaire Design

Experimental design is the third part of designing a choice modelling experiment. The complexity of the experimental design is linked to the numbers of attributes and levels. The greater the number of attributes and levels per attribute, the more complex the design (Bateman et al., 2002). Several methods can be used to statistically account for the different attributes and levels; these are full fractional, fractional factorial, blocking and efficient designs.

Full fractional design involves presenting respondents all possible combinations of attributes in the CM questionnaire. As the numbers of attributes and levels increase so does the size of the full factorial, which results in the total number of choice sets required to be presented to respondents, exceeding the ability of the respondents to cope with the large numbers of choice sets presented to them (Bennett & Adamowicz, 2001; Kragt & Bennett, 2008).

Fractional factorial is one method of addressing this issue, where a subset of all possible combinations is selected. This results in an “orthogonal

44

experimental design that enables attributes to be statistically independent from one another ” (Kragt & Bennett, 2008 p. 15). The alternative to address a large number of choices is to segment the choice tasks into blocks. This will ensure each respondent is only exposed to alternatives of one block of the choice tasks (rather than all the choice tasks in the experiment). This approach, however, requires a large enough sample size to generate the data necessary for this model to work (Bennett & Adamowicz, 2001).

Researchers have more recently suggested, from a statistical perspective, that experimental designs, which form a critical part of stated preference tasks, should convey the maximum amount of information about the parameters of the attributes relevant to a particular choice task. This cannot be guaranteed with an orthogonal fractional factorial design (Hensher, Rose, & Greene, 2005). Orthogonal fractional factorial designs are generated so that statistical independence can be achieved with the attributes, although the statistical efficiency of the design is generally not considered.

Rose, Bliemer, Hensher, & Collins (2008) highlight the more recent shift that has taken place in CE toward the use of efficient (non-orthogonal) designs. Efficient designs are a type of design that attempts to reduce the asymptotic standard errors of parameter estimates (which are the square roots of the diagonal components of the asymptotic variance-covariance AVC matrix). This improves the reliability of the parameter estimates and enables the analyst to reduce the sample size required for obtaining suitable

45

levels of consistency in parameter estimates. This reduction of standard errors for parameter estimates has created a class of experiment designs that are referred to as ‘efficient designs’ (Bliemer, Rose, & Hess, 2008). The efficient component of the design is one that is referred to as producing data to enable the estimation of parameters with the lowest possible standard error (Choice Metrics, 2012). D-Efficient designs are one type of design that is part of this class. The d-error is described by Rose et al. (2008), as a measure of design efficiency to differentiate between designs, where the assumption is made that if the d-error is low, the (co)variances of the parameter estimates are also low. D-efficient designs minimise the elements of the AVC matrix, therefore improving the accuracy of parameter estimates for a design.

A key characteristic of an efficient design is that the analyst requires a prior knowledge of the utility functions in order to generate the components of the AVC matrix. This results in the necessity of undertaking a pilot survey to obtain parameter values which play a critical part in defining the level of efficiency of a design. With the absence of exact parameter values, estimated parameter values are likely to be required during the design phase. Therefore, it is common for the analyst to make certain assumptions as to what values to use, in order to generate the efficient design (Bliemer et al., 2008).

Regarding the questionnaire, CM questions have evolved from conjoint methods and are increasingly being used in environmental economics, to value changes in natural resource quality. Choice methods differ from

46

conjoint methods because individuals are asked to choose from bundles of (environmental) goods, which are described in terms of attributes. When constructing each choice set, Bennett & Blamey (2001) observe that it is important to ensure that each choice task has as much realism associated with it as possible. The question should mimic the decision that one would make when choosing one choice over another. The researcher needs to ensure that the attributes are constructed in a manner that is consistent with actual behaviours associated with the perceived policy problem. A CM survey is structured, and according to Bennett (1999) and Bennett & Adamowicz (2001), is usually presented in the following format:

1. An introduction of the issue under investigation

2. The framing of the environmental good under consideration

3. Statement of the issue

4. Statement of a potential solution

5. Providing the choice of ‘choosing not to choose’

6. Introduction of the choice sets

7. Presentation of the choice sets

8. Follow up questions to explore motivations behind the decisions

9. Socio-economic and attitudinal data collection to assist in verifying

data and checking how well the sample represents the population of interest.

These key steps are required when designing a CM questionnaire, once the attributes and levels have been determined. Each step forms a necessary component in extracting behaviours from the survey group in order to

47

identify the trade-offs that each individual makes between each attribute, to determine the utility they derive from particular attributes (Mogas, Riera, & Bennett, 2005).