Development of the Survey Instrument

4.5.1

Design of the Web-based Survey Instrument

The design of the instrument is extremely important in obtaining unbiased answers from respondents. According to Couper, Traugott and Lamias (2001), when the visual design elements complement or support the verbal features of the survey instrument, efficiency and data quality gains may be achieved. Dillman et al. (1999) proposed 10 criterion for respondent-friendly design principles for Web surveys. The main points were to limit advanced features, to include a welcome screen, make the initial question interesting, shorten line length and provide specific instructions for each set of questions. In short, Dillman et al. (1999, p. 14) claimed that ‘a respondent-friendly Web questionnaire is one that interfaces effectively with a wide variety of computers and browsers possessed by respondents’. The authors concluded that the overriding challenge of Web-based survey design was to keep the questionnaires simple. Similarly, Granello and Wheaton (2004) advocated a 12 point procedure that should be followed for the development of Web-based and Email surveys. The 12 steps were used as the basis for conducting the present study, and are listed in Table 4.3, along with what was done in relation to the current research. From the 12 steps, the two most important were the pilot testing and the selection of the panel, which will be considered in more detail in the next section.

Table 4.3 Steps for Conducting Web-based and Email Surveys

12 Steps for conducting surveys Steps taken in current study

Determine the population to be measured Panel of event experts was recruited Determine whether an e-mail or Web-based

survey will be used Web-based survey chosen as most appropriate

Develop the layout of the survey and the type of

format for the questions Preliminary designs were done on Front Page

Write the questions Questions were written

Keep the layout simple, with easy-to-read fonts

and a consistent layout throughout Design and layout were kept simple

Be sure to address informed consent issues, including the name and contact information of the researcher

These issues were addressed in the email that was sent out to the respondents Determine how data will be entered into the

computer Data will be emailed to the researcher from the collecting company

Practice putting in data Pseudo data were entered e.g. all 5’s in the

scale to test response Include “error detection” variables in anonymous

Web-based surveys Not applicable to this research

Pilot the study using a subset of the target

population Instrument was piloted twice using two different design programs for Round One Round Two and Round Three instruments were also tested

Determine the schedule for initial mailing,

including e-mail posting and reminders Time lines set for each round of survey

Download the data frequently Data were downloaded regularly

Source: Adapted from Granello and Wheaton (2004)

4.5.2

Pilot Testing

A number of researchers (See, for example, Wyatt 2000) suggested that piloting the survey using a subset of the population is a vital step in the design process. In particular, the pilot testing should include the following: have participants submit the survey from a number of different computers and Internet connections, using different browsers; employ persons with a variety of levels of technical knowledge; make sure the directions are clear; and encourage users to make mistakes. Finally, it is advisable for the researcher to be present with members of the pilot group to receive feedback firsthand (Granello & Wheaton 2004). After initial development, the survey instrument that was used for the study was pilot tested amongst a convenience sample of event and non-event respondents. Where possible, the researcher was present when the instrument was tested. Based on the feedback obtained from the initial pilot

testing, the Web site and the program being used proved to be inappropriate, mainly due to technical restrictions with the design options. In particular, the program had very limited options for question design. For example, it was not possible to design matrix-style questions that use a Likert-type scale and have a number of different variables.

As a result, a subscription-based survey design program called SurveyMonkey (SurveyMonkey 2005) was used to develop the second version of the survey instrument. This option allowed for increased flexibility in terms of the instrument design (for example, layout of questions and filters) and delivery of results in a usable spreadsheet format. The second version of the survey was pilot tested by the convenience sample used in the first pilot. Feedback was positive, and the program proved to be more reliable and user friendly in comparison to the initial version. Importantly, the new version enabled the technical problems associated with the first version to be overcome. For example, all 20 key impacts could be viewed on the same page, and were also able to be included in a drop-down menu both of which were not available in the earlier version. The instruments developed for Round Two and Round Three of the survey were also pilot tested, as each round was deemed to be important in the context of the Delphi study.