Pilot testing the questionnaire in the member states

Cognitive and pilot tests of the draft questionnaire were undertaken in all three member states (Sweden, Italy and Poland). The findings from these tests are discussed below.

3.10.1 Cognitive testing

Cognitive testing of the questionnaire was undertaken in the week commencing 18 April 2011. In these tests interviewers administered the questionnaire with genuine respondents to:

- test respondents’ comprehension and ability to answer the questions - investigate how respondents interpret the meaning of specific terms - investigate whether there are any missing questions.

Different approaches such as asking the respondent to “think aloud” and describe what they are thinking as they consider and answer the question or more detailed probing of responses given were used to assess respondents’ understanding of the questionnaire.

Cognitive testing also allowed alternative wording or phrasing to be explored. This is especially important when we are working with a questionnaire which has been translated, as is the case in this study.

RAND Europe Implementing the valuation methodology

Accent in three member states

Swiss Economics

The result is to produce a higher quality questionnaire with more reliable data as any ambiguities have been eliminated before the questionnaire is used.

We had aimed to undertake 20 cognitive interviews in each country with the following spread of respondent types:

- five vulnerable consumers - five non-vulnerable consumers - five SMEs

- five large businesses.

Table 3.21 shows the number of cognitive interviews undertaken in each country. From this table it is observed that we met our aims in Sweden, that we obtained nearly enough interviews in Poland (although no interviews were undertaken with large businesses) and that we were short of interviews in Italy.

Table 3.21: Numbers of cognitive interviews by country and type

Sweden Poland Italy Total

Vulnerable 5 3 1 9

Not vulnerable 5 7 5 17

SMEs 5 9 6 20

Large businesses 5 0 2 7

Total 20 19 14 53

The cognitive tests indicated that overall the questionnaire worked well. Specifically, respondents had a good understanding of the questions, although the SP choice experiments were felt to be complex and the overall questionnaire length was felt to be too long. Minor changes to the questionnaire were made as a result of the cognitive interview findings (and additional minor changes were made at the client’s suggestion).

3.10.2 Pilot surveys

Formal pilot surveys were then undertaken with the revised questionnaires. We aimed to undertake 30 pilot interviews in each country, including:

- 15 consumer interviews

- six interviews with vulnerable users - six SMEs and three large businesses.

The pilot surveys exactly replicated the phone–post/e-mail/fax-phone methodology to be used in the main surveys.

The pilot surveys were used to assess:

- the recruitment process - survey response rates

- the clarity and flow of the questionnaire - the appropriateness of the language used

- the accuracy of all routings

- the SP experimental design and understanding of the choice exercises - the interview duration.

The pilot study showed that in general the survey was working as planned. Recruitment rates were reasonable, the survey questions were largely understood and we obtained decent quality data. However, the pilot survey identified a number of issues.

First, the questionnaire was found to be too long. We therefore dropped a number of the longer background questions, including importance questions, ranking questions of importance and satisfaction questions.

We also moved the SP choice exercises so they were undertaken earlier in the questionnaire, after the background questions about use of postal services, so that respondents were less fatigued when they participated in the choice experiments.

We also made the following improvements to the choice exercises:

- The presentation of the SP choice exercises was enhanced, so that when attributes were the same for both alternatives they were shown in a lighter font.

- The Saturday delivery option was dropped from the first and second choice exercises, to simplify the choice tasks.

- The text in the choice exercises was reduced as much as possible. Already the text was quite concise, but reductions in text could be made for the attributes that incorporated percentages. For these, we dropped the percentage descriptions and presented the quantity values only, e.g. “85 out of 100 letters delivered on time”.

- The range of the reliability and loss attributes was extended to make the choices “more different” to respondents (reliability to include 80 out of 100 letters delivered on time, 90 out of 100 letters delivered on time, 95 out of 100 letters delivered on time; 0 out of 100 letters lost, 5 out of 100 letters lost, 10 out 100 letters lost).

- The definition of uniform pricing was clarified. In the pilot survey the uniform pricing level was presented as “Same price to deliver to all locations” for both letters and parcels. There was some question as to whether this included international post, and how this might work, so we changed the text to “Same price to deliver letters/packages to any destination within Poland/Sweden/Italy”. Also for non-uniform pricing options an example was included, e.g. “For example, it might be cheaper for local deliveries and more expensive for more distant destinations”.

- The range of price adjustments was increased to include: −30%, current price, +30%, +50%, +100% and +150%.

- The definition of a secure box was clarified. In the pilot survey the description was

“Delivered to a secure box 100m from your home”. This was extended to say

“Delivered to a secure box 100m from your home where you can collect your letters/parcels at any time”.

- Visual aids were included in Experiments 1 and 2 to emphasise that these are related to letters and parcels, respectively.

- The experimental design was amended by assuming that all attributes are categorical to ensure more choices between intermediate levels and that the design was checked to

RAND Europe Implementing the valuation methodology

Accent in three member states

Swiss Economics

ensure that the number of dominant choices was small (a dominant alternative is an alternative where everything is better (or worse) than the other alternative).