Putting items into a draft questionnaire

The items have been arranged in six different formats during the course of the research as shown in Table 3.5 below. At each of the first four stages feedback was gathered and changes were made, including removing items, which are detailed in the reports of data gatherings. The different formats were used for data gathering and contributed to the data used for factor analysis of the full data set. Only the 50 items selected for the questionnaire were used in the factor analysis at the end of this research.

Items Delivery Medium

Language Notes Reference See Section Long List (i.e. all 256 items) Paper English/Arabic in parallel

Did not have “don‟t know” response option. 5-point Likert scale

Format A 4.3.1

Internet English 7-point Likert scale Format B 4.3.2 Internet Mandarin Chinese 7-point Likert scale Format C 4.3.3 Emailed Word document. Mandarin Chinese 7-point Likert scale Items grouped into “face valid” sections

Format D 4.3.4

Short List

(i.e. the 50 items selected from the Long List) Internet Mandarin Chinese 7-point Likert scale Format E 5.3 5.6

Internet English 7-point Likert scale

Format F 5.4

5.5

Table 3.5: The formats of the instrument used in the research

3.10.1.1

Questionnaire length

Gillham (2000: 41) notes that it is common for ―the novice researcher […] to try to include […] too many questions‖. Dörnyei (2003: 18) recommends thinking of the

120 slowest reader in a sample as a guide to limiting the length of a questionnaire, and points out that one cannot include everything. Respondent fatigue is another reason to keep questionnaires short and Gillham feels that this is particularly relevant with rating scale questions (which is the type of response format chosen for this

questionnaire). He says ―It is extremely boring to answer a series of scaled-response questions; and people stop thinking about what they are doing‖ (Gillham 2000: 39).

Questionnaire length is usually expressed in terms of how long it takes to complete. The figure of thirty minutes is often mentioned, for example Dörnyei (2003: 18), Aiken (1997: 38), and Fowler (1993: 103). The initial list of items was rather long, but this was necessary in order to deal with all the areas. The latest version of the questionnaire (Formats E and F) has an average completion time of 16 minutes. This is well within the recommended figure.

3.10.1.2

Grouping and sequencing

Sudman & Bradburn (1982: 207) see the order of the questionnaire as requiring the same care as the wording of the items. Cohen et al. (2000: 258) consider that the configuration of the questionnaire indicates ―the overall logic and coherence of the questionnaire to the respondents‖. The order of the questions and the groups into which they are placed contribute to this impression and lead to better data (Sudman & Bradburn 1982: 207).

Aiken (1997: 38) recommends that the items should follow a ―logical conversational sequence‖ and the questionnaire should not ―jump around‖. It is important not to produce a questionnaire that ―seems to meander at random from question to question‖ as ―Respondents who perceive the questionnaire as being done carefully

121 are more likely to be careful in the responses they give‖ (Sudman & Bradburn 1982: 228). Gillham (2000: 25) also recommends that questions should lead logically from one to the next for two reasons: it makes it easier for the respondent to work through, and it prevents ―dotting around‖, i.e. not answering the questions in sequence. This is important because questions are not ―stand alone‖ says Gillham (2000: 25) and if they are not answered in the intended order the contexts of the questions will be different for each respondent. This context effect is defined by Tourangeau, Rips, & Rasinski (2000: 200) as ―the effects of earlier questions on the responses to later ones‖. This arises because questions can stimulate memories and with the memories activated, the responses to following questions can be influenced. The answer the respondent gives will be in the context of those activated memories (Converse & Presser 1986: 38). This is an effect which can make such a great difference that individual respondents can be, in effect ―answering a different questionnaire‖ (Gillham 2000: 12). However, Tourangeau et al. (2000: 216) maintain that context effects are greater when the questions are perceived as being related and ―When questions shift from one topic to the next without warning, respondents are no longer likely to see earlier questions as carrying implications about the meaning of later questions‖. This suggests that ―dotting around‖ may not be so undesirable. There appears to be some disagreement in the literature, about the nature of this effect and I feel that it is so difficult to account for, especially when it is not known whether the effect is present in the questionnaire or not, that it should be looked at only in the event of problems arising.

I put the open ended questions at the end of the questionnaire as they are more difficult to answer, take up more time (Dörnyei 2003: 48, 62), and are perceived as threatening if they occur at the beginning (Sudman & Bradburn 1982: 218, 262).

122 Demographic questions were also placed at the end in line with advice from Sudman & Bradburn (1982: 218) and Dörnyei (2003: 61). However, Converse & Presser (1986: 39) admit that ―It is frequently unclear that one order is better than another‖ as ―each order may reveal a different facet of the issue being studied‖. As with question wording, the order of items in a questionnaire is clearly highly important, but it is frequently a matter of judgement of relative merits in a specific situation.

In the Microsoft Word-based version of the questionnaire (Format D) I grouped the questions by theme as recommended by Aiken (1997: 37), which was designed to ―break up‖ the 256 item questionnaire and make it appear shorter. I sequenced these groupings so as to give a ―sense of structuredness‖ (Dörnyei 2003: 21) and these sections were given headings to orientate the respondents to the focus and help to make them feel more involved (Cohen et al. 2000: 258-259).

3.10.1.3

Questionnaire formatting

―The appearance of the questionnaire is vitally important‖ (Cohen, et al. 2000: 258); the respondents‘ perception of the difficulty of the task can be affected by its

appearance (Sudman & Bradburn 1982: 243). Consequently ―It must look easy, attractive and interesting rather than complicated, unclear, forbidding and boring‖ (Cohen, et al. 2000: 258). This can be helped by aiming for a clean uncluttered look (Gillham 2000: 39; Dörnyei 2003: 19; Cohen et al. 2000: 258). It is a common mistake to attempt to make a questionnaire look shorter by crowding questions together and using a small type face (Sudman & Bradburn 1982: 244). All this advice was kept in mind for all versions of the questionnaire, and following a suggestion by Dörnyei (2003: 21), different colours were used for the different sections in the Microsoft Word-based version and, rather than numbering the questions from 1-256,

123 they were numbered within the sections to help make it appear shorter (Cohen et al. 2000: 258-259). Format F (the latest version) has only 50 items and it was not considered necessary to use formatting to make it appear less forbidding.