Reliability

Williams & Webb (1994) stated that there was no evidence that the Delphi method was reliable. A different expert panel may produce a different set of results, even if both panels are provided with the same Delphi survey questionnaires under identical conditions (Goodman, 1987). This points to the subjective nature of

115 Delphi data and also raises questions about the generalizability of Delphi’s results (Reid, 1988). The data from Delphi studies are essentially subjective in nature and critics’ concerns over reliability issues in the Delphi methodology are therefore understandable (Sackman, 1975; Rowe, Wright, & Bolger, 1991; Williams & Webb, 1994; Hasson, Keeney, & McKenna, 2000). Reliability or reproducibility could be tested by replication of a Delphi study with a similar group of experts under the same conditions. However, Grisham (2009) felt that even this might not result in the same outcome. One study specifically addressed the reliability issues inherent in Delphi methodology (Duffield, 1993) and was able to demonstrate 93% agreement between two separate Delphi panels, even though they differed slightly in composition and size. Ono & Wedemeyer (1994) looked at the accuracy of predictions made by a Delphi study conducted 16 years earlier and discovered that the earlier study’s findings were accurate in terms of forecasting events and trends in the communications field. Ament (1970) found a similar level of accuracy in their study comparing Delphi forecasting studies in 1964 and 1969. Quintana et al. (2000) looked at results from two parallel Delphi panels and a test-retest performance of one panel; they found a high degree of consistency between both panels’ results, and test-retest consistency in the main panel.

5.3 Summary

While one has to concede that there are a number of unresolved issues with Delphi methodology, its use in the health-care field continues to be widespread. The next chapter describes the three phases of the Delphi survey used in this study. In a typical Delphi survey, the first questionnaire’s results generate the second questionnaire round, the results of which then generate the third questionnaire round and so on, across rounds. Therefore, the next three chapters present the research process as it occurred in practice.

116 Chapter 6

The First Questionnaire Round

6.1 Introduction

The Delphi process used in this study included three rounds and Table 6.1 describes each step of the process. This chapter describes the first Delphi round. Details regarding recruitment of the expert panellists and ethical considerations are presented, and then the design and development of the questionnaire used in the first questionnaire round is discussed with reference to SurveyMonkey. Finally, the methodology used to analyse data and results for round one are presented.

Table 6.1 The Steps of the DelphiProcess

1. Defining the research project. 2. Recruitment of experts.

3. Development of first questionnaire and round one pilot study.

4. Dissemination of first questionnaire and data collection.

5. Analysis of round one data.

6. Development of second questionnaire and round two pilot study.

7. Dissemination of second questionnaire and data collection .

8. Analysis of round two data.

9. Development of third questionnaire and round three pilot study.

10. Dissemination of third questionnaire and data collection.

11. Analysis of round three data. 12. Final report fed back to experts.

117 The National Research Ethics Service advised that the study did not require ethical review by an NHS Research Ethics Committee. The University of Salford ethics committee granted ethical approval for the study and Greater Manchester Primary Care Research Governance Partnership granted research governance approval. It was important to protect the anonymity and confidentiality of the research participants, and this is discussed later in this chapter.

6.1.1 Consent

The information letter soliciting prospective expert participants’ involvement in the study contained information detailing how data would be handled and stored throughout the project. It informed experts that they would be completing the Delphi questionnaires online and that the Web survey host, ‘SurveyMonkey’, would be used to administer the questionnaires (SurveyMonkey, 2011). Experts were informed that they would be able to print the survey, complete it by hand, and then return it by post or facsimile if they so wished. Some studies have suggested that response rates for online surveys are inferior to other methods such as post, telephone, or facsimile (Cook, Heath, & Thompson, 2000; Manfreda et al., 2008; Shih & Fan, 2008). This could be due to poor design on the part of the researcher (Kaplowitz, Hadlock & Levine, 2004) or because recipients’ Internet security identifies the incoming email as ‘spam’ (Sills & Song, 2002). Researchers usually offer a choice of two or more modes of dissemination in an attempt to combat non-response (Dillman, 2007), and a mixed-mode strategy seems to improve overall response rates (Dillman et al., 2009). It can be difficult to ensure informed consent in the absence of a face-to-face meeting (Eynon, Fry, & Schroeder, 2008). In this study, as with other online studies (Chou, Boldy, & Lee, 2002; Alahlafi & Burge, 2005; Farley, 2005; Holmes, 2009; Mostofsky et al., 2010; Webster et al., 2010) it was decided that consent was implied when experts responded by email to inform the PI that they were agreeing to participate in the study. The University of Salford’s ethics committee approved this decision. Potential expert participants were encouraged to contact the PI by email or phone if they had any queries about the research.

118 6.1.2 Confidentiality and Anonymity

Individual experts were directed to the secure ‘SurveyMonkey’ host website through a hyperlink embedded in a personalized email. Using an embedded URL link avoids the risks associated with ‘digital pathogens’, which can occur when downloading infected material from the Web (Duffy, 2002). Experts were contacted via their preferred email addresses, using an NHSmail email account. NHSmail is the secure email and directory service for all NHS staff in England and Scotland; it is the only email system recommended for transferring patient data. It was not considered appropriate to use the PI’s work email address and there were fears that emails sent through a personal yahoo email account might appear unprofessional even if they were not treated as unsolicited mail by spam filters operating at the receiving end. All the survey results were stored on the ‘SurveyMonkey’ website and only the PI could access this data, using a unique username and password. Security was heightened by using a professional subscription to the site. This provided the enhanced security option, SSL encryption, which is the same system used in online banking. It ensured URL link encryption and meant that both the URL link and the survey Web pages were protected throughout all stages of communication and data transmission between the PI and respondents.

Experts’ individual responses were treated as confidential; neither their participation nor the information they provided was revealed to a third party. Experts were guaranteed anonymity during the data collection and report-writing phases, and in any future publication. The identity of the experts was protected by using the blind carbon copy method when group emails were necessary. Using a URL link embedded in a personalized email will return anonymous responses to the ‘SurveyMonkey’ website. Thus, in order to follow up non-responders, the PI had to know the identity of each respondent; experts were informed that they would be assigned a unique identifying code number known only to the PI.

119 6.2 The Expert Panel