Questionnaire Validity

Chapelle and Jamieson (1991) explain validity by dividing it into two types: internal and external. Internal validity refers to the accurate attribution of observed results to the factors that were supposed to be responsible for these results; external validity denotes the applicability of research results to instructional and research contexts other than the one in which the research was carried out. Neuman (2000) adds that both internal and external validity are primarily used in experimental research; the internal validity indicating whether there are possible errors or alternative explanations to account for the results, while the external is to measure the ability to generalise findings from a specific sample to a wider population.

Although it is not possible to have absolute confidence about the measurement of survey validity, some measures are more valid than others. Chapelle and Jamieson (1991) explain that there are many types of validity, such as face, content, criterion, concurrent, predictive, construct, and convergent validity. The easiest and the most common are face and content validity. Face validity is a simple judgement of whether the instrument used seems relevant to the stated objectives, while content validity implies a more systematic matching of items with some defined domain of content (Neuman, 2000). Field experts (e.g. May 1997; Punch 1998; De Vaus 2001) agree that content validity can be tested through the judgement of the scientific community. They

114

indicate that it is becoming common to use a panel of experts to evaluate the instruments used.

To test the clarity of the items used and their relevance to their scales, the face and the content validity of the questionnaire were tested through the following procedures:

1. The questionnaire was translated into Arabic (the local language) and back- translated into English by two language lecturers in the University of Bahrain to assess whether the concepts remained the same. Corrections and changes were made accordingly.

2. The questionnaire was reviewed thoroughly by the research supervisors (Dr. Kevin Orr and Dr. Richard Common) to check the clarity of the questions, their appropriateness for fulfilling the research aims, and to ensure that the meaning reflected the content of the scales.

3. Copies of the questionnaire (in both English and Arabic) were distributed to three specialised assessors in the Bahrain Research and Studies Centre, and three scholars (lecturers, PhD holders) from the University of Bahrain. A letter was addressed to these assessors explaining the aims of the study and asking for their evaluation of validity and whether or not the questions represented the issues under investigation. Copies were later collected and approximately one hour was spent with each assessor, face to face, discussing notes and comments. These mostly concentrated on the need for the sentence structure to be more meaningful, appropriate and clear for participants and for the answer to tally with the choice of Likert scale.

4. A pilot survey was conducted to test the validity of the instruments.

The assessors’ comments resulted in a few changes. Among these, for instance, were the following:

• Questions (e.g. 1, 3, 9, 12 and others) that started with the words “to what extent...?” were changed to a more direct format to avoid misunderstanding.

• Question 10 required revision of the Arabic translation to find clear Arabic phrases that corresponded with management phrases in common use.

115 3.6.5 Piloting

It is essential to pilot survey instruments as fully as possible before the main study takes place. De Vaus (2001) notes that survey instruments must be tested and refined under real-world conditions so that researchers can remove any items that do not yield usable data, add items to fill any data gaps and reword unclear questions in preparation for the main study. Collis and Hussey (2003) add that a good questionnaire should not rely solely on the researcher’s perspective and the design process should be subjected to field testing. They state that, “It is amazing how even a non-specialist friend can spot a glaring error…it may take several drafts, with tests at every stage, before researchers are satisfied that they have got it right” (p. 175). A pilot study, therefore, was conducted in Bahrain in July 2008.

The pilot study of the questionnaire was conducted personally with 10 participants with whom the researcher was able to arrange meetings individually over the four-day period of the event. The questionnaire was discussed question by question and statement by statement with participants to ensure clarity. The pilot study helped to locate repeated, difficult-to-understand or misinterpreted questions, as well as weak questions. Questions that took longer to answer were located in order to consider whether they might need to be reworded or broken down into separate parts. The average time taken to complete the questionnaires was 40-50 minutes; this was later reduced to 30 minutes. Participants of the pilot study provided very useful feedback, as most of them were postgraduate students who had sound knowledge of research methods. They were very cooperative and some expressed their support and appreciation of the topic and considered it beneficial. Two participants suggested that a few items were inapplicable in their situation and that some were ambiguous. Apart from these comments, most participants thought that the questions were, generally, clear and understandable.

All comments were reviewed alongside the feedback from the researcher’s supervisors and changes were made accordingly. The results of the pilot work revealed that the process of data collection should proceed smoothly.