Choice experiment design

The elements of a choice experiment are (Louviere, 2001):

1. A choice set, C, containing some number of different alternatives, (a1,…, aj).

3. The levels of attributes, or the different values they could take. These are discrete values and can be either categories, such as ‘present’ or ‘absent’, or different levels taken from a continuous variable, such as several different price levels.

The levels and the attributes are used to define or describe the alternatives in the choice set. For example, given two attributes ‘apple colour’ and ‘price’ and their corresponding levels (red, green) and (low, high), the four possible alternatives in the choice set are shown in Table 4.2.

Table 4.2. Combinations of apple attributes

Price

Colour Low High

Red Red apple, low price Red apple, high price Green Green apple, low price Green apple, high price

The number of alternatives in the choice set is thus a function of the number of attributes and the number of levels. Given K attributes, each with two levels, the number of possible

combinations is 2K ; more generally, the number of combinations is m1 x m2 x …x mK, where

m indicates the number of levels for each k, (1,…,K) (John, 1998). The number of alternatives in the choice set grows rapidly as levels and attributes are added.

The complete factorial includes all the alternatives as described above. However, the number of alternatives in the choice set is usually limited by means of a fractional factorial design (Hahn & Shapiro, 1966; John, 1998; Louviere et al., 2000). Techniques for creating such fractional factorial plans as well as ready-to-use plans are available (Hahn & Shapiro, 1966; John, 1998; Louviere et al., 2000). To determine the main effects of the attributes on

factorial. The main effects are the influences of each attribute in isolation from the other attributes on respondents’ choices. Not included are any two-way, three-way, or more-

complex interactions between or amongst the attributes. It is important to note, however, that by limiting the design to a main effects fractional factorial, some of the higher-order

interactions are confounded with the main effects themselves (John, 1998). Empirical evidence suggests that 70% to 90% of explained variance is a result of main effects; 5% to 15% is a result of two-way interactions; the rest is explained by higher-order interactions (three-way and greater) (Louviere et al., 2000).

There are thus two issues with using fractional factorial designs. The first is the bias introduced into the estimates, because the parameter calculated for the main effect is also capturing any influence from interaction effects. Prior research suggests that this bias is not likely to be very large (Louviere et al., 2000), but the size of the bias is an empirical question. The second issue is that the interactions may in fact be significant and important in the choice process. Using a main effects design corresponds to assuming an additive functional form for utility, in which preferences over choice attributes are separable. However, whether this assumption holds could be tested empirically by designing choice sets that are larger than a main effects design. Two-way interactions between different attribute preferences, for example, could be estimated to determine their size and significance.

Appropriate design of choice sets requires finding an equilibrium amongst the competing demands of realism, orthogonality and balance. Realism is an important consideration in all stated choices research in order to obtain valid statements regarding respondents’ preferences (Bateman et al., 2002); that is, it is important for the validity of the survey results.

Orthogonality in survey design allows researchers to separate the effects of one product attribute from the effects of another, and balance in attribute levels – having all attributes with the same number of levels – is desirable (Louviere et al., 2000). Orthogonality and balance

can be statistically assessed by calculating the D-efficiency of a choice set design (Chrzan & Orme, 2000; Kuhfeld, Tobias, & Garratt, 1994). The statistic is calculated as:

100 * 1 / [ND |(X'X)-1|1/p],

where ND is the number of runs or alternatives, p is the factors in the survey design2, and X is

the ND x p design matrix. Kuhfeld, et al. (1994) caution that D-efficiency is a relative measure

of design efficiency, not an absolute measure. The D-efficiency statistic is thus a way to compare two potential survey designs.

Once the all the alternatives in the full choice set have been constructed, the survey questions can be assembled. For each question, respondents are presented with several different

alternatives (three or four is common) and asked to choose which of them is preferred. Each survey includes several of these choice questions: six to eight questions are recommended (Bennett & Adamowicz, 2001), but researchers may be able to ask up to twenty choice questions without the data declining in quality (R. M. Johnson & Orme, 1996). Several methods for assembling these choice questions from the set of alternatives are available, including random pairing, drawing from statistically similar choice sets, using a ‘mix and match’ approach, and ‘shifting’(Chrzan & Orme, 2000; Louviere et al., 2000). This last has been found to be an efficient design for choice questions (Chrzan & Orme, 2000).

There are a number of issues associated with choice set design (Blamey, Louviere, & Bennett, 2001):

Number of attributes. Larger numbers of attributes lead to more choice alternatives, more- complex choice tasks, but also better descriptions of the alternatives. The task is to balance

simplicity and salience. However, increasing the number of attributes may not affect parameter values, but does affect the model estimation (Louviere, 2001).

Generic versus alternative-specific labels. Whether to provide the different alternatives with meaningful labels (‘government option’ versus ‘private option’, for example) or generic labels (‘option A’, ‘option B’) is important (Bennett & Adamowicz, 2001). Meaningful labels have additional content that must be captured by extra terms in the data analysis.

Opt-out option. Surveys should include some way that respondents can opt out of a choice question (Bennett & Adamowicz, 2001). For some surveys, such as recreation surveys, this might be an option not to participate, while for studies of products this might be the option to stay with the current brand or product (Banzhaf, Johnson, & Mathews, 2001). Including an opt-out option avoids the problem of forcing respondents to make trade-offs, which is inappropriate (Scott, 2002).

Attribute descriptions. In order to assess the impact of a change in an attribute on choice behaviour, the description needs to present that change in a way that is both plausible (for the respondent) and measurable (for the researcher) (Blamey et al., 2001).

Dominated alternatives. In a single choice question, one alternative may be strictly dominated: it may be worse than another alternative for all the attributes. In particular, an alternative from the full choice set may be dominated by the status quo alternative: switching from the current product or situation to the dominated one entails being worse in every dimension. Dominated alternatives are often discarded (e.g., Burton et al., 2001), as it would be irrational for someone to choose an alternative that was worse in every way. However, they have been retained in some research in order to test for rationality (V. Foster & Mourato, 2002).

Plausibility of alternatives. Choice experiments are stated preference methods, so the

possibility of hypothetical bias is ever-present. It is important that the constructed alternatives are plausible and realistic (Blamey et al., 2001), to maintain content validity of the survey and to improve the probability of getting non-hypothetical answers from respondents. In

particular, combinations of attributes that imply large benefits at reduced costs may not be believable (Bateman et al., 2002).