SURVEY INSTRUMENT VALIDATION 110

A pre-test is done to verify content validity. Wynd, et al. (2003) cited Carmines & Zeller (1979) to define content validity as the extent to which the research instrument is able to adequately sample the research domain of interest for the phenomenon being measured. In the case of questionnaire surveys, items of questions should be able to measure the full range of the construct being measured. To achieve this, duplicate questions on similar items are avoided. Thus the full range of variables are covered. Both content validity of each item and of the overall scale are important and are measured (Lynn, 1986)

Although there are arguments about the methods to measure content validity, generally two methods are in use. One is the evaluation approach. In this method, a minimum of three experts (but not more than 10 according to Lynn, 1986) evaluate the questions to ensure that they are able to cover the full range of constructs. They give their ratings which can be subjected to quantitative analysis to test the degree of

agreement between experts. If the degree of agreement is wide, the question is removed. This method is unsuitable if the range of topics covered in the questionnaire is wide. In such cases, many experts from different fields are required. Getting such large panels of experts together will be difficult. The process also will be quite unwieldly and will involve significant amounts of time.

The second method is to conduct a pilot study with a smaller sample size and the opinions of participants regarding the contents, their relevance and practicality are considered for any revision of the questions and format. The pilot study is a smaller version of the full study done for pre-testing of a survey instrument like a questionnaire. It enhances the likelihood of the success of the main study. The pilot study enables identification of problems with items included in the questionnaire, nature, order, framing and time taken to complete the questionnaire. These inputs help to revise the content of the questionnaire (if need be), plan and organise the survey efficiently and effectively. The advantages of a pilot study were highlighted by Oppenheim (1992). The small sample used in a pilot study should be representative and indeed should reflect the larger sample used in the study proper. Otherwise, applicability of the pilot study results to large samples will be doubtful. Tull and Hawkins (1990) have indicated that to focus on a theme by asking questions in a logical manner is the major objective of the sequence survey questionnaire. The survey instrument should be checked carefully to ensure that the various questions are related to the research subject, so that the collected data are useful for addressing the research questions. Saunders et al. (2003) indicated that the survey response rate depends to some extent on the design of the questionnaire. The questionnaire design also affects reliability and validity of the data collected. There is enough evidence to show that response rates, validity and reliability increase when the following points are addressed: careful design of individual questions, clear description of the questionnaire form, clear explanation of the purpose of the questionnaire, pilot testing and a cautiously planned and executed survey administration.

Conducting the pilot study has a significant role in gaining information on attributes or opinions toward a particular theme from a specific population (Sekaran, 2003). These viewpoints have been supported also by Blair and Presser (1992).

Pre-testing the questionnaire is an important step for questionnaire development because it improves the understanding of the research problems (Neuman, 2006; Frazer and Lawley, 2000; Babbie, 1990). According to Neuman (2006), there is a major advantage in conducting a pre-test. The pre-test enhances the understanding and applicability of the survey questionnaire and also helps to know whether the survey questionnaire is adequately developed to measure the key constructs of the study.

The following steps were undertaken for this study. A pre-test was considered necessary to identify potential problems in the survey instrument and to determine both the validity and the reliability of the survey instrument. Having considered the above points, the major advantage of a pilot study is that any problem expected in the development of the questionnaire can be determined directly. Potential biases from respondents when completing questionnaires can be reduced. On the whole, the pilot test was useful to explore whether the questions had been appropriately designed. It also helped to clarify confusing questions by reframing them and confirmed how well the questions were understood. It also helped to ensure the content validity of survey instrument before doing the actual survey

Prior to being piloted, the questionnaire was given to two colleagues and three supervisors in order to check the accuracy and clarity of the phrasing of the questions and the appropriateness of the question types, as well as to give their opinions on the content.

Additionally, a pilot study was undertaken to assess the comprehensibility of the questions more accurately. It was essential to find out whether there is any unclear content or flaws in the design of the questionnaire and framing of questions. There was also a necessity to allow for opportunities for review, and further development of the questions. The pilot study was useful also to assess the time taken to complete the questionnaire by any person. Timing was useful in planning an organising the survey work.

Given the importance of conducting an effective pilot survey, a sample size 20-50 is considered adequate to identify the potential weaknesses associated with the questionnaire (Sheatsley, 1983). Accordingly, a pre-test survey was conducted on MICE travellers using a convenience sampling technique. A convenience sample

was considered appropriate as the sample was drawn from a defined population of business travellers in Riyadh Exhibition Centre and it directly related to the purpose of this research.

The pilot study was carried out on the first day of the Saudi Travel and Tourism Investment Market Forum in Riyadh. The questionnaire was given to 50 delegates and 34 questionnaire completions were achieved (20 Muslim and 14 non-Muslim) and analysis of their responses indicated that there was no difficulty with the questions. Only international participants were approached, as the research was aimed at international delegates. Analysis of the results, as stated above, indicated that there were no problems with answering the questions. The results of the pre- test was expected to result in some modification of questions in the final questionnaire. Since no modification was necessary it was decided the initial changes from the expert opinion was comprehensive. This final questionnaire was used for the survey proper.

Respondents of the pilot study were watched to note how long it had taken them to complete the questionnaire, so that an estimated time for completion could be given to respondents in the final study. The time varied between ten and fifteen minutes for those who were willing to complete the survey. This was a measure of confirmation that the layout and size of the questionnaire would not deter people from answering it.