PLEASE SCROLL DOWN FOR ARTICLE

(1)

PLEASE SCROLL DOWN FOR ARTICLE

This article was downloaded by: [Universidad Publica de Navarra]

On: 18 June 2009

Access details: Access Details: [subscription number 773400313] Publisher Routledge

Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK

International Journal of Social Research Methodology Publication details, including instructions for authors and subscription information:

http://www.informaworld.com/smpp/title~content=t713737293

Influence of questionnaire design on response to mail surveys Vidal Díaz de Rada

Online Publication Date: 01 February 2005

To cite this Article de Rada, Vidal Díaz(2005)'Influence of questionnaire design on response to mail surveys',International Journal of Social Research Methodology,8:1,61 — 78

To link to this Article: DOI: 10.1080/1364557021000025991 URL: http://dx.doi.org/10.1080/1364557021000025991

Full terms and conditions of use: http://www.informaworld.com/terms-and-conditions-of-access.pdf This article may be used for research, teaching and private study purposes. Any substantial or systematic reproduction, re-distribution, re-selling, loan or sub-licensing, systematic supply or distribution in any form to anyone is expressly forbidden.

The publisher does not give any warranty express or implied or make any representation that the contents will be complete or accurate or up to date. The accuracy of any instructions, formulae and drug doses should be independently verified with primary sources. The publisher shall not be liable for any loss, actions, claims, proceedings, demand or costs or damages whatsoever or howsoever caused arising directly or indirectly in connection with or arising out of the use of this material.

(2)

Int. J. Social Research Methodology Vol. 8, No. 1, February 2005, pp. 61–78

Influence of Questionnaire Design on

Response to Mail Surveys

Vidal Díaz de Rada

Taylor and Francis Ltd

tsrm100202.sgm

Received 30 September 2000; accepted 11 July 2002

10.1080/1364557021000025991 International Journal of Social Research Methodology 1364-5579 (print) Original Article 2004 Taylor & Francis Ltd 8 0000002004 Vidal Díazde Rada

Department of SociologyPublic University of NavarraPamplonaSpain vidal@unavarra.es.

This paper examines in some detail the influence of the questionnaire design on the ratio and quality of responses to mail surveys. To this end an investigation using Dillman’s total design method for mail surveys was used and carried out in Spain. To minimize the effort required in filling in the questionnaire, Dillman proposes that the questionnaire should be easy to fill in, and stresses the importance of its having an attractive and pleasing off-set. This paper will outline a series of elements to take into consideration on the dimensions and size of the questionnaire, on its cover pages, on the type and colour of paper used, on the ordering of the questions, the envelope and stamps used to send off the questionnaire. This paper lays out the the suggestions made by Dillman as well as those of others who have written on this theme.

Introduction

Most books on research through surveys insist on the importance of the questionnaire design on the ratio and quality of the responses (Czaja & Blair, 1996, pp. 78–85; Manheim & Rich, 1986, p. 157; Newman, 1997, pp. 249–251; Sheatsley, 1983, pp. 223– 225; Sudman & Bradburn, 1987, pp. 229–260). However, a detailed analysis of these and other texts reveal that most of the recommendations refer to the presentation of the questionnaire (Sheatsley, 1983, p. 219), the organization of the questionnaire through topics, the use of ‘change of theme’ questions (Czaja & Blair, 1996, p. 77), instructions for the interviewer (Fowler, 1995, pp. 92–99), the space between the ques-tions (Sheatsley, 1983, p. 224; Sudman & Bradburn, 1987, p. 224), the numeration of the questions, and so on. In the specialized literature available sparse attention has been paid to the dimensions of the questionnaire, the cover pages, or the colour of the paper to be used.

Vidal Díaz de Rada is a Doctor in Sociology and Lecturer in The Department of Sociology in the Public Univer-sity of Navarra, Pamplona, Spain. His areas of specialization are: consumer behaviour, lifestyles, and market research (method of collecting data and non-response). Correspondence to: Vidal Díaz de Rada, Department of Sociology, Public University of Navarra, 31006 Pamplona, Spain. Email: vidal@unavarra.es

(3)

62 V. Díaz de Rada

While some studies have shown up the importance of the graphic design of the ques-tionnaire in face-to-face surveys (Sánchez, 1992, pp. 206–217), most investigations about this point have been done on mail surveys, which is where these points are partic-ularly important. Nevertheless, the number is not very extensive and besides, they reach contradictory conclusions when the different studies are examined in detail (Díaz de Rada, 1999, pp. 238–240).

In this paper we would like to examine in some detail the influence of the question-naire design on the ratio and quality of responses to mail surveys. To this end an inves-tigation using Dillman’s (1978) total design method for mail surveys was used and carried out in Spain. To minimize the effort required in filling in the questionnaire, Dillman (1978, p. 211) proposes that the questionnaire should be easy to fill in, and stresses the importance of its having an attractive and pleasing-to-the-eye off-set. This paper will outline a series of elements to take into consideration on the dimensions and size of the questionnaire, on its cover pages, on the type and colour of paper used, on the ordering of the questions, the envelope and stamps used to send off the question-naire, and so on. In the following section the suggestions made by Dillman are outlined as well as those of others who have written on this theme.

Literature Review Dimensions

After an exhaustive analysis of the research carried out on this theme, an absence of research on analysis of how the dimensions of the questionnaire can influence the ratio and quality of the responses was detected. Dillman (1978), in his total design method, advises designing the questionnaire like a ‘booklet’ with measurements of 15.56 × 20.95 cm, which is roughly equivalent to a 21 × 29.7 cm sheet doubled in half and with a stapled fold (1978, p. 121). Dillman also advises printing the questionnaire on paper that one normally uses and then making a 79% size reduction using a photocopier.

Although Dillman and Frey (1974) and Dillman (1978) point out that this is the best size, in a later study, Dillman (1991, p. 234) considers that this aspect of total design method has been scantily examined and suggests further research on the theme. Erdos and Morgan (1983) suggest a somewhat larger questionnaire (18.5 × 25.5 cm) in their classic study of mail surveys.

Within the sparse research carried out on this subject we highlight the study under-taken in the UK by Jansen (1985), where he compares the different response rates obtained using different sized questionnaires and sent out to the general public. He used the size suggested by Dillman (1978), the size suggested by Erdos and Morgan (1983) and an intermediate size (17.0 × 24.5 cm). The results can be seen in Table 1. An analysis of this table shows that the 18.5 × 25.5 cm size is better than 15 × 21 cm, and the 17 × 24.5 cm size obtains more response than 14.5 × 21 cm. Nevertheless, we cannot achieve an overall conclusion because we cannot compare the four sizes. In Study 1, even a within-study comparison of the effect of format is not possible since type layout and colour are not constant. In Study 2, the repercussions of the number of pages and format are quite large and worthy of comment.

(4)

International Journal of Social Research Methodology 63

Johnson, Parsons, Warnecke and Kaluny (1993) carried out a replica of this study in the USA where they compared the questionnaire of Jansen (14.5 × 21.0 cm) with the measurement 21.6 × 27.9 cm (with standard USA paper) on a sampling about primary care physicians. As can be seen from Table 2, the standard size (21.6 × 27.9 cm) achieved a higher response rate and a lower number of unanswered questions. An anal-ysis of these and other results caused Dillman (1991, p. 237) to suggest that question-naires of reduced dimensions would get a better response from young people while the older population would prefer larger sized measurements.

Cover Page

Another aspect which Dillman raises in his total design method is the question of the design of the cover page of the questionnaire since this is the first thing that the interviewee will see and it may condition his reaction to it (Dillman, 1978; Jenkins and Dillman, 1997). Dillman points out that questions should not be written on the cover page and that it should only hold an interesting title or drawing or some other element

Table 1 Influence of Questionnaire Design on Response Rate

Study 1: Students’ attitudes towards suicide (response rate on day 70)

Type Layout Colour Pages Format (cm) n Response (%) Typewriting white 12 15.0 × 21.0 143 60.1% Designed white 15 15.0 × 21.0 141 51.1% Typewriting yellow 12 18.5 × 25.5 288 66.3% Designed yellow 15 18.5 × 25.5 284 67.6%

Study 2: social and psychological aspects of residential waste separation (response rate on day 40)

Pages Format (cm) n Response (%)

16 17.0 × 24.5 430 81.9%

9 17.0 × 24.5 430 74.8%

16 14.5 × 21.0 430 73.3

9 14.5 × 21.0 430 65.2

Source: Jansen, 1985, pp. 140, 141

Table 2 Dimensions of Questionnaires and Response Quality

Reduced size (14.5 × 21)

Standard size (21.6 × 28)

Total Test statistics

Cooperation rate (%) 58.5 64.2 61.3 X2 = 7.0* Item Non-response (mean no. items) 4.3 2.6 3.4 t = 4.6† Completed questionnaires (%) 50.9 60.8 55.5 x2 = 7.0*

Notes: *p < 0.01 † p < 0.001

Source: Johnson et al., 1993, p. 272

(5)

which may catch the attention of the reader. Below this illustration or whatever, the purpose of the survey could be outlined, a message which stresses the value and impor-tance of the opinions of the interviewee, etc. It would be opportune to put the name and address of the institution below the survey at the bottom of the front page. This page could also be used to hold the instructions on how to go about filling in the ques-tionnaire. (See Appendix 1 for the cover page of the quesques-tionnaire.)

Among the sparse research which has been analysed, the influence of aspects of the work done by Nederhof (1988) should be highlighted. He shows how the drawing or design on the cover page of the questionnaire increases the rate of response, as it ampli-fies the salience attributed to the questionnaire. Nederhof compares the same cover design on a white backdrop (with black print) and a backdrop (with print in white) where the latter gets a response rate of some 15% higher than the former. Although the successive reminders increase the importance given to the questionnaire, the impres-sion produced by the cover page with a black background creates a feeling of heightened respect for the questionnaire in the interviewees (Grembowski, 1985; Nederhof, 1988). Some years later J. J. Dillman and D. A. Dillman (1995) carried out a replica of Nederhof’s research on the general population of Washington, and found that the cover page image did not influence in the response rate. Dillman and Dillman (1995, p. 122) attributed this difference in Nederhof’s results to the fact that he had used a specialized population group within their workplace and thus were possibly more motivated than the general population.

Colour

While Dillman (1978, p. 121) points out that the questionnaire should be printed on white paper so as to avoid any similarity with publicity material, and sent out by first-class mail, other researchers have explored the question of how the colour can have an influence on the questionnaire. Despite the fact that the use of colour paper may make the interviewees associate it with letter-box publicity or low quality, and so be poten-tially counterproductive (Dillman, 1978; Pressley & Tuller, 1977), a great number of researchers indicate that the colour of the questionnaire has a small influence on the response ratio.

Gullahorn and Gullahorn (1963) compared the response rate from a sample of scholarship students with a white paper questionnaire and a green-coloured one. The latter got a response rate of 51% which was some two points more than the white-papered questionnaire—a difference not statistically significant.

An experiment carried out by Matteson (1974) on the members of a professional group got a 24% response with pink-coloured paper, which was four points more than the rate for the white-coloured paper. He concluded that colour did not significantly increase response rate.

Pressley and Tuller (1977) analysed the effectiveness of colour paper from a sample on 280 marketing directors. The highest response came from the white paper (43.7%), although the differences were not significant: yellow paper 38.8%, green paper 37.3% and blue paper 35.8%.

(6)

Jobber and Sanderson (1983) analysed the response rate from a blue-coloured ques-tionnaire on a group of managers from the textile sector. The blue got the highest response rate although the difference was not significant.

W. Crittenden, V. Crittenden and Hawes (1985) compare the effectiveness of yellow-coloured questionnaires on a group of students. The yellow-coloured one got a response rate of 49% compared to 37% from the white-papered one.

Finally, Fox, Crask and Kim (1988) carried out a meta-analysis of the influence of the coloured questionnaires by doing 10 investigations. The green-coloured question-naire got a response rate of some 5% higher than the white-coloured questionquestion-naire.

Various interpretations exist on the influence of colour. Jobber (1986) points out that the use of coloured sheets makes the questionnaire more conspicuous when it is lying on a desk or table which is likely to have a majority of white sheets. Other expla-nations consider the significance of colour in the minds of the interviewees, something which is not wholly relevant to this present study. (The idea of green as an expression of hope, red as an expression of passion, etc.)

Evidence of the Effects of Questionnaire Design Hypothesis

As has been earlier stated, the aim of this study is to show whether the design of the questionnaire affects the quality and rate of response in mail surveys. Within the factors which are included in the design of the questionnaire, we will pay particular attention to the influence of the size, the colour and the presentation of the covers pages of the questionnaire. The considerations outlined above lead us to set down the following hypotheses:

H1: The size of the questionnaire paper has a big influence on the quality and the rate of response in mail surveys:

1. Although a large-sized questionnaire could make the interviewee think that it will take a long time to fill in, nevertheless, the size of the lettering used, the space available to write in, etc. would make it easer to fill in. 2. Moreover, the amplitude of a ‘large-sized’ questionnaire clearly helps to a

great extent to achieve a better quality in the responses, especially from the older age groups (Dillman, 1991, p. 237).

H2: Fox et al. (1988) have shown in their research that a coloured-paper question-naire significantly improves the response rate, fundamentally due to the ‘conspicuousness’ it acquires when it is placed among a pile of normal papers on a desk. Likewise, in this study we shall compare green paper with normal white paper from among the possibilities considered above.

Not only does colour produce an improvement in the response rate, using a post-card in the first follow-up reminder with a different colour to white also produces a higher response rate:

(7)

H3: In relation to the design of cover pages it has been pointed out that these pages should be used to stimulate the interest of the recipient. Our hypothesis is that questionnaires which have some kind of illustration on the cover pages will furnish a higher response rate.

H4: The combination of all the above-mentioned factors (large size, green colour and cover pages) will produce a notable response rate even if this slightly increases the final cost of the survey when it comes to mailing the questionnaires.

Sampling and Implementation Process

With the aim of clarifying some of the questions raised in the previous text, it was decided to make the first ever investigation using the total design method on Spanish society. The sampling size to adequately encompass the region of Navarra was estab-lished at 442 interviews, carrying out 214 in the urban area (Pamplona, 166,279 inhab-itants), 42 interviews in a semi-urban vicinity (Tudela, 26,857) another 42 in a municipality with more than 10,000 inhabitants (Estella, 12,552) and the rest in three municipalities with less than 10,000 inhabitants (Caparroso, 2,331; Corella, 6,329; Los Arcos, 1,346). After taking the initial decision on the sampling proportions, the next step was to choose a document which would have a representative and accurate list of the population under study (frame population).

Some experts have shown that the telephone directory can be used as a frame popu-lation for general-popupopu-lation mail surveys because it is available at little or no cost. In fact, in another paper Díaz de Rada (2001, p. 12) showed the viability of using a two-year-old telephone directory. Although some 12% of the selected sample did not receive the questionnaire, in 7.6% of the cases this was due to its being delivered to an erroneous address. These questionnaires were returned to the investigators. Moreover, a telephone call made some days later revealed that 3.2% of the sampling had not received the questionnaire either. This could have been due to the difficulty in locating the dwelling within a large tower block of flats, where many people are living. The prob-lems with the population frame are considerably reduced in the rural areas.

Dillman’s (1978) total design method (TDM) was followed closely in the design of the questionnaire and the mailing procedures. The most important deviation from the TDM was the fact that a third follow-up letter was not sent out. The difference being that the registered letter was changed for a phone call, as several investigations carried out on specialized collectives have shown that the use of the telephone produces a similar increase in the response rate (Dillman & Moore, 1983; Nederhof, 1988). The completion rate obtained was 60% (Table 3), rounding off at some 65.5% in the urban area and 57.0% in the rural area. A more detailed analysis can be found in Díaz de Rada (2000). As regards the selection of the individual within the home, half the presentation letters indicated that these should be answered by an adult female. In those cases where there might not be any women in the home, the questionnaire should be filled in by an adult male. The other half of the presentation letters should be filled in by the eldest of the daughters. (It must be remembered that the letter is addressed in the name of the person taken from the telephone directory, normally the head of the family.) In those

(8)

cases where there were no daughters it was requested that the questionnaire be filled in by the eldest of the sons, or the person who received the letter. An analysis of the aver-age aver-age and sex of those who received a letter leads us to believe that the interviewees followed these instructions (Appendix 2).

Manipulation

With the aim of testing these hypotheses, eight types of questionnaire were distributed at random to the selected sampling group. Each one of the questionnaires had its own specifications with regard to:

● Measurements: 220 interviewees received a ‘small-size’ questionnaire (14.85 × 21

cm) and 222 interviewees were sent a ‘large-size’ one (21 × 29.7 cm). The question-naires were printed on paper of 21 × 29.7 cm and then reduced in size by using a photocopier, just as Dillman recommends (1978, p. 121).

● Colour of the questionnaire: 220 questionnaires were printed in white and 222 were

printed in green. Of those who got the green paper, the reminder postcard was sent to half of these interviewees in white, while the other half received it with the same green colour. This same strategy was employed with those who got the white paper.

● With regard to the cover page, 222 were designed following the dictums of Dillman

in his TDM, while there were no illustrations made of any kind with the other ques-tionnaire and it simply listed the first questions of the survey.

As regards the illustration on the front, as this survey dealt with consumer habits in the province of Navarra, Spain, it was decided to insert an outline of the contour of the region and municipalities that make up this region (see Appendix 1). Moreover, on each questionnaire the particular municipality dwelling of each interviewee was ‘framed’ with a different-coloured felt pen. These three experimental conditions, with

Table 3 Kinds of Questionnaires Used in the Experiment Questionnaire Sent and Questionnaire Response

Questionnaire Response rate sent response

White:

Small format with cover: 56 46 82.1% Small format without cover: 56 28 50.0% Large format with cover: 52 34 65.4% Large format without cover: 56 40 71.4% Green:

Small format with cover: 54 34 62.9% Small format without cover: 52 30 57.7% Large format with cover: 58 26 44.8% Large format without cover: 56 26 46.4%

Total: 442 264 59.7%

(9)

two possibilities for each one, make up eight types of questionnaire (2 × 2 × 2) whose distribution is shown in Table 3.

Results

Characteristics of the Questionnaire and Response Rates

The importance of these three variables in the response rate reveals that only the colour of the questionnaire has a significant influence on the response, as can be seen in the final part of Table 4. The white questionnaire gets a response rate of 67.3%, some 15% higher than the green questionnaire, in line with the results obtained by Pressley and Tuller (1977). The dimensions of the questionnaire, the existence (or not) of cover pages and the mailing of a reminder postcard of a different colour to white did not affect the response rate in any of these cases.

What exactly is the total influence of these variables? Does some general effect exist in common among them? That is to say, would a particular combination of size, colour

Table 4 Influence of Each Factor on Response Rate

Format Small (14.85 × 21cm) Large (21 × 29.7cm) Response (%) 59.1% 60.4% Non-response 40.9% 39.6% Total: 220 222 Chi-Square (yates) = 0.30066 1 df p = 0.78556 Cover page

Cover page No cover

Response 63.1% 56.4% Non-response 36.9% 43.6% Total: 222 220 Chi-Square (yates) = 1.79270 1 df p = 0.18060 Colour White Green Response 67.3% 52.3% Non-response 32.7% 47.7% Total: 220 222 Chi-Square (yates) = 9.74928 1 df p = 0.00179

Colour of reminder (postcard)

White questionnaire Green questionnaire White postcard Green postcard White postcard Green postcard Response 63.5% 70.7% 47.1% 56.7% Non-response 36.5% 29.3% 52.9% 43.3% Total: 104 116 102 120 Chi-Square (yates) = 0.99368 1 df p = 0.25398 Chi-Square (yates) = 2.03996 1 df p = 0.15321

(10)

and design increase the response rate? Logistic regression would allow us to answer these questions because this method can determine whether, after controlling the effects of format and cover page, the effect of colour is still significant. Of major interest here are the interaction effects since they allow us to assess, for example, whether the effect of colour is the same for each format combinations, or for each cover page combination. The dependent variable is the non-response indicator for an individual: value 1 means response and 0 non-response. As independent variables we shall use the termi-nations outlined in Table 5, along with all its interactions: format by cover page, format by colour, format by colour of reminder, cover page by colour, cover page by colour of reminder, colour by colour of reminder.

An analysis of the 95% confidence interval for the odds ratio reveals the zero influ-ence of each one of the independent terminals in the response rate.1

It may be that there were no differences in the overall response rate, but that there might have been some in the response rate after each mailing, on the premise that certain factors generated a stimulus in the interviewee to make them respond more rapidly. If this were the case it would be interesting to analyse some of the referred aspects in each one of the mailings. An analysis of Table 6 shows that only the cover page shows a distinct response rate after each follow-up. After two reminders the questionnaire with a cover page received 118 responses, while the one without a cover page only reached 92 inter-viewees. This conclusion does not change when the combined influence of all the vari-ables is analysed creating a logistic regression with the terminals used in Table 5.

To finalize this section the socio-demographic traits of those who most responded to the questionnaire are outlined. The great number of independent terminations has meant that we have used a logistic regression stepwise with backward elimination. Table 7 only shows the significant coefficients. A detailed analysis reveals that ‘small-sized’ questionnaires were returned mostly by people under 64 years and by men, while

Table 5 The Influence of All Factors and its Interactions on Response Rate (Logistic Regression)

Variable B odds ratios 95% interval of odds ratios Format 0.1922 1.2119 0.15, 10.05 Cover page −1.3326 0.2638 0.04, 2.06 Colour −1.6629 0.1896 0.02, 1.51 Colour reminder 1.0100 2.7455 0.32, 23.12 Format by cover page 0.2916 1.3385 0.61, 2.94 Format by colour 0.0060 1.0060 0.46, 2.21 Format by colour reminder −0.3677 0.6923 0.31, 1.52 Cover page by colour 0.5493 1.7321 0.79, 3.80 Cover page by colour reminder −0.1659 0.8472 0.39, 1.85 Colour by colour reminder 0.0970 1.1019 0.50, 2.42

Constant 2.1733

Categories of each variable: Format (1 = small size, 0 = large size); Cover page (1 = front cover, 0 = no cover); Colour (1 = white, 0 = green); Colour reminder (1 = white, 0 = green). Reference category is ‘no response’.

(11)

the women and those over 65 responded more to the ‘large-sized’ questionnaires. The men preferred the white-papered questionnaires, while more women responded to the green-coloured ones.

Type of Questionnaire and Quality of the Response

Above it has been shown how the design of the questionnaire has a very slight effect on the response rate. A second aim of this study is to find out how much the design might influence the quality of the response on the premise that a high response rate is not worth much if the data gathered is of a low quality and impedes a proper analysis being made. The quality of the response will be analysed by considering the number

Table 6 Influence of Each Factor on Response Rate After Every Follow-up (Only People Who Have Responded)

Format Small Large (14.85 × 21cm) (21 × 29.7cm) First follow-up 47.0% (62) 42.6% (58) Second follow-up 31.8% (42) 35.3% (48) Third follow-up 21.2% 22.1% Total: 133 137 Chi-Square = .54272 2 df p = 0.7624 Cover page

Cover page No cover First follow-up 43.7% (62) 46.0% (58) Second follow-up 39.4% (56) 27.0% (34) Third follow-up 16.9% 27.0% Total: 143 127 Chi-Square = 6.3025 2 df p = 0.04280 Colour White Green First follow-up 39.7% (58) 50.8% (62) Second follow-up 37.0% (54) 29.5% (36) Third follow-up 23.3% 19.7% Total: 147 123 Chi-Square = 3.3349 2 df p = 0.18872

Table 7 Factors with A Significant Influence on Response Rate (Logistic Regression)

Variable B odds ratios 95% interval of odds ratios Format by age by sex 0.3675 1.4444 1.13, 1.84 Colour by sex 0.8109 2.2498 1.13, 3.85 Constant 0.9780

Categories of each variable: Format (1=small size, 0=large size); cover page (1=front cover, 0=no cover); Colour (1=white, 0=green); colour reminder (1=white, 0=green); Age (1=younger than 65 years, 0=65 and older); Sex (1=man, 0=woman). Reference category is ‘no response’.

(12)

of ‘non-response items’ in the questionnaire along with the presence of ‘double’ answers where only one is required (Deleeuw & Hox, 1988; Faria, Dickinson & Filipic, 1990; Poe, Seeman, McLaughlin, Mehl & Dietz, 1988).

In Table 8 it can be seen that the number of non-response items is less in the ‘large-size questionnaires, without a cover page and with a green colour, although the differ-ences are not significant. In conclusion, it can be said that there is no relation between the type of questionnaire used and the quality of the responses.

Table 8 Kind of Questionnaire and Response Quality

NON-RESPONDENT ITEMS Format Small Large (14.85c × 21c) (21c × 29.7c) None 55.3% 63.4% One 11.8% 4.9%

Two and more 22.9% 31.7%

Total: 170 164

Chi-Square = 5.69011 2 df p = 0.5813

Cover page

Cover page No cover

None 62.5% 55.7%

one 8.0% 8.9%

Total: 176 158 Chi-Square = 1.6272 2 df p = 0.44325 Colour White Green None 58.4% 60.3% One 11.2% 5.1%

Total: 178 156

Chi-Square = 4.2171 2 df p = 0.12141 DOUBLE RESPONSES TO QUESTIONS WHICH REQUIRE JUST ONE SINGLE ANSWER

Format Small Large (14.85c × 21c) (21c × 29.7c) None 92.7% 96.4% Some 7.3% 3.6% Total: 220 222 Chi-Square (yates) = 2.2264 1 df p = 0.13567 Cover page

Cover page No cover

None 93.7% 95.5% Some 6.3% 4.5% Total: 222 220 Chi-Square (yates) = 0.3683 1 df p = 0.54391 Colour White Green None 93.6% 95.5% Some 6.4% 4.5% Total: 220 222 Chi-Square (yates) = 0.74376 1 df p = 0.38846

(13)

Just as we have asked ourselves in the preceding section, we again wonder at this point whether a particular combination of size, colour and design have influenced the quality of the response. In Table 9 the influence of these factors is shown in the number of items non-responded and the double responses to questions which require just one single answer. The interaction between format and cover page increase item non-response, while double responses are affected by format, the interactions format by colour and colour by colour of the reminder.

Conclusion and Discussion

The aim of this study has been to find out what influence the design of the question-naire has on the rate and quality of the response, which is, without a doubt, one of the aspects least researched within social and market research by survey. More precisely, an attempt was made to analyse the influence of the size of the questionnaire paper (14.85 × 21 cm or 21 × 29.7 cm), the colour of the paper (green/white) and the format of the cover page (with cover page/without cover page) in a mail survey carried out in Spain. The results outlined reveal that the colour of the questionnaire is the only element which has an effect on the final response rate, although if we consider the response rate after the first follow-up reminder, the questionnaire with a cover page gets a higher response rate than the one without a cover page. As regards the quality of the responses, none of the aspects considered influences the item non-response or double answers to any significant extent.

Having tested and rejected the first three hypotheses, it now remains to consider to what extent the ‘large-size’ questionnaire (size 21 × 29.7 cm) with a green colour and a cover page produces a response rate higher than the others, as expressed in the fourth hypothesis. An analysis of the first column of Table 10 forces us to reject this hypothe-sis, since this questionnaire got a response rate of 45%—the lowest of any of the ques-tionnaires used. So Table 10 shows the use of a small, white-papered questionnaire with a cover page (response rate 82.1%) or a big, white-papered questionnaire without a

Table 9 Factors with A Significant Influence on Response Quality (Logistic Regression)

Non-respondent items

Variable B odds ratios 95% interval of odds ratios Format by cover page 0.2912 1.3380 1.08, 1.65 Constant 0.2600

Double responses to questions which require just one single answer

Variable B odds ratios 95% interval of odds ratios Format −1.6385 0.1943 0.04, 1.03 Format by colour −1.8409 0.1587 0.04, 0.57 Colour by colour reminder −0.9606 0.3826 0.19, 0.75 Constant 3.8744

Categories of each variable: see Tables 5 and 7

(14)

cover (response rate 71.4%). An analysis of the response rate leaves one in no doubt about this, although it is also necessary to consider some other aspects before opting for one or the other.

Starting with the size of the paper, the choice between the 14.85 × 21 cm size or the 21 × 29.7 cm size will mean doubling the cost of the paper when selecting the latter choice. It must also be taken into account that the format with a ‘cover page’ proposed by Dillman implies adding two pages to the length of the questionnaire, since neither on the first nor the last page should any questions be written. In Table 10 the number of pages of the questionnaire are compared in relation to the quantity of paper used, taking the quantity of sheets with a 21 × 29.7 cm size which are needed for each ques-tionnaire used in this survey. Thus, the ‘small-sized’ quesques-tionnaire without a cover page can be printed on one sheet, while the ‘large-sized’ questionnaire with a cover page requires the use of three sheets.

To the cost of the paper must also be added the weight of the ‘mail-out packet’, the cover letter and the business reply envelope (Dillman, 1983, p. 366). In Spain all letters which weigh less than 20 grams need a stamp of 0.21 Euros, a figure which increases to 0.27 Euros when the letter weighs between 20 and 50 grams (Posts and Telegraphs, 1998, p. 2). In column five of Table 10 it can be seen that there is quite a noticeable difference in the cost of mailing each letter between the large-size design with a cover page and the others.

A joint analysis of column one and column five in Table 11 shows that the large-sized with a cover page questionnaire is the most expensive and that it gives a response rate of between 44.5% and 65.4% depending on the colour of the paper. The most economic questionnaire is the small-sized one without a cover page, with a response rate that ranges between 50% and 57%. The questionnaire that gets the highest response rate is, as has already been noted, the small-sized one with a cover page, which shows an intermediate cost (two sheets).

In conclusion we would advise the use of a questionnaire of 14.85 × 21 cm size, with white paper and with a cover page, which with an ‘intermediate’ price gets a response

Table 10 Paper Used and Cost in Every Questionnaire

Response ratea Number of pages Number of sheetsb Weight (gram) Mail postage (Euro) White:

Small format with cover 82.1% 8 2 −20 0.21 Small format without cover 50.0% 4 1 −20 0.21 Large format with cover 65.4% 6 3 +20 0.27 Large format without cover 71.4% 4 2 −20 0.21 Green:

Small format with cover 62.9% 8 2 −20 0.21 Small format without cover 57.7% 4 1 −20 0.21 Large format with cover 44.8% 6 3 +20 0.27 Large format without cover 46.4% 4 2 −20 0.21

a_{Source: Table 3.}

b_{Standard European paper: 21}_×_29.7cm.

(15)

rate of 82.1%. The reader might react to this advice by recalling the research made by Dillman (1991, p. 237) when he points out that the smaller-sized questionnaires are responded to better by young people, while older people tend to respond better to large-sized ones. It is observed that no relation was found between the size of the ques-tionnaire and the quality of the response rate, although it is true that in this section it is observed that older people tend to respond better to ‘large-sized’ questionnaires. A detailed analysis of the response rate made by the older age group forces us to deny the existence of variations in the quality of the response rate in relation to the size of the questionnaire. However, the meagre sampling done on the older population would counsel us to make this affirmation with prudence.

Acknowledgement

With special thanks to George Buchan.

Note

1.

[1] A confidence interval which does not contain 1 indicates that the parameter is significantly different from zero at the 5% level.

References

Crittenden, W., Crittenden, V., & Hawes, J. (1985). Examining the effects of questionnaire colour and print front on mail survey response rates. Akron Business and Economic Review,16, 51–56. Czaja, R., & Blair, J. (1996). Designing surveys. Thousand Oaks, CA: Pine Forge Press.

Deleeuw, E. D., & Hox, J. J. (1988). The effects of response-stimulating factors on response rates and data quality in mail surveys. Journal of Official Statistics,4, 241–249.

Díaz de Rada, V. (1999). Factores que implementan la eficiencia de las encuestas postales. en Revista Española de Investigaciones Sociológicas,85, 221–251.

Díaz de Rada, V. (2000). Using Dillman’s total design method (TDM) in a South Europe country: Spain. WAPOR Annual Conference, Portland, Oregon, 17 May.

Díaz de Rada, V. (2001). Is it fitting to use the telephone directory as a frame population in surveys? An analysis of the case in Spain. en Bulletin de Methodologie Sociologique,69, 5–17.

Dillman, D. A. (1978). Mail and telephone surveys. New York: Wiley.

Dillman, D. A. (1983). Mail and other self-administered questionnaires. In P. H. Rossi, J. D. Wright, & A. B. Anderson (Eds.), Handbook of survey research. (pp. 359–377). San Diego, CA: Academic Press.

Dillman, D. A. (1991). The design and administration of mail surveys. Annual Review of Sociology, 17, 225–249.

Dillman, J. J., & Dillman, D.A. (1995). The influence of questionnaire cover design on response to

Table 11 Paper Used In Every Questionnaire

Number of pages of the quest. Number of sheets (a) Weight (gram) Postage Small, cover 8 2 –20 35 Small, no cover 4 1 –20 35 Large, cover 6 3 +20 45 Large, no cover 4 2 –20 35

a_{Standard European paper: 21 x 29.7cm.}

(16)

mail surveys. Proceedings of the International Conference on Survey Management and Process Quality, Bristol, pp. 109–114.

Dillman, D. A., & Frey, J. H. (1974). Contribution of personalization to mail questionnaire response as an element of a previously tested method. Journal of Applied Psychology,59, 297–301. Dillman, D., & Moore, D. E. (1983). Improving response to mail surveys: Results from five experiments.

Unpublished manuscript, Washington State University, Pullman, WA. Erdos, P. L. & Morgan, A. J. (1983). Professional Mail Surveys. Malabar, FL: Krieger.

Faria, A. J., Dickinson, J. R., & Filipic, T. V. (1990). The effect of telephone versus letter prenotifica-tion on mail survey response rate. Journal of the Market Research Society,32, 551–568. Fowler, F. J. (1995). Improving survey questions (Applied Social Reseach Methods Series, Vol. 38).

Newbury Park, CA: Sage.

Fox, R. J., Crask, M. R., & Kim, J. (1988). Mail survey response rate (a meta-analysis of selected tech-niques for inducing response). Public Opinion Quarterly,52, 467–491.

Grembowski, G. (1985). Survey questionnaire salience. American Journal of Public Health,75, 1350– 1360.

Gullahorn, J., & Gullahorn, J. (1963). An investigation of the effects of three factors on response to mail questionnaires. Public Opinion Quarterly,27, 276–281.

Jansen, J. H. (1985). Effect of questionnaire layout and size and issue-involvement on response rates in mail surveys. Perceptual and Motor Skills,61, 139–142.

Jenkins, C. R., & Dillman, D. A. (1997). Towards a theory of self-administered questionnaire design. In L. Lyberg, P. Biemer, M. Collins, L. Decker, E. deLeeuw, C. Dippo et al. (Eds.), Survey measurement and process quality. New York: Wiley.

Jobber, D. (1986). Improving response rates in industrial mail surveys. Industrial Marketing Manage-ment,15, 183–195.

Jobber, D., & Sanderson, S. (1983) The effects of a prior letter and coloured questionnaires paper on mail survey response rates. Journal of the Market Research Society,25, 339–349.

Johnson, T. P., Parsons, J. A., Warnecke, R. B., & Kaluny, A.D. (1993). Dimensions of mail question-naires and response quality. Sociological Focus,26, 271–274.

Manheim, J. B., & Rich, R.C. (1986). Empirical political analysis: Research methods in political science. New York: Longman.

Matteson, M. (1974). Type of transmittal letter and questionnaire colour as two variables influenc-ing response rates in a mail survey. Journal of Applied Psychology,59, 535–536.

Nederhof, A. J. (1988). Effects of a final telephone reminder and questionnaire cover design in mail surveys. Social Science Research,17, 353–361.

Newman, L. W. (1997). Social research methods: Qualitative and quantitative approaches. Boston, MA: Allyn and Bacon.

Phippis, P.A., Robertson, K.W., & Keel, K.G. (1991). Does questionnaire colour affect survey response rates? Proceedings of the Section on Survey Research Methods, American Statistical Association, pp. 484–490.

Poe, G.L., Seeman, I., McLaughlin, J., Mehl, E., & Dietz, M. (1988). ‘Don’t know’ boxes in factual questions in a mail questionnaire. Public Opinion Quarterly,52, 212–222.

Posts and Telegraphs. (1998). Tarifas postales y telegráficas año 1998.

Pressley, M. M., & Tuller, W. L. (1977). A factor interactive investigation of mail survey response rates from a commercial population. Journal of Marketing Research,14, 108–111.

Sánchez, M. E. (1992). Effects of questionnaire design on the quality of survey data. Public Opinion Quarterly,56, 206–217.

Sheatsley, P. B. (1983). Questionnaire construction and item writing. In P. H. Rossi, J. D. Wright, & A. B. Anderson (Eds.), Handbook of survey research (pp. 159–230). San Diego, CA: Academic Press.

Sudman, S., & Bradburn, N. M. (1987). Asking questions. San Francisco: Jossey Bass.

Tucker, C., Lepkowski, J. M., Casady, R. J., & Groves, R. M. (1992). Commercial residential tele-phone lists: Their characteristics and uses in survey design. Social Science Computer Review, 10, 158–172.

(17)

Appendix 1 Questionnaire Cover Page

(18)

Is there anything else you would like to tell us about consumer attitudes or any other related topic? If so, please use this space for that purpose.

Also, any comments you wish to make that you think may help us in future efforts to understand the consumer behaviour in Navarra will be appreciated, either here or in a separate letter.

(19)

Appendix 2 Sample by Age and Sex

Man Woman Total

15–24 7 (5.4%) 13 (9.7%) 7.6% 25–34 14 (10.8%) 16 (11.9%) 11.4% 35–44 32 (24.6%) 28 (20.9%) 22.7% 45–54 26 (20.0%) 26 (19.4%) 19.7% 55–64 18 (13.8%) 19 (14.2%) 14.0% 65–76 33 (25.4%) 32 (23.9%) 24.6% Total: 130 134