How to plan the research design

A Delphi survey takes several weeks, even if the number of participants is small.

It has to be planned in the beginning of the project or, if the necessity to conduct such a study appears late in the course of the project, it is important to realize that the whole process takes several weeks, depending on the number of rounds needed. The next figure illustrates the whole process and the time needed.

KCE Report 187 Qualitative Research Methods 65

Figure 5 – The Delphi process

Adapted from Slocum et al.93

Definition of the problem

Identification and selection of experts

Administration of the first round questionnaire

Analysis of results yes no Report findings Elaboration of 2nd/3rd round questionnaire and administration

Invitation of experts and finalization of the questionnaire (including online set up)

2 w e eks 2 w e eks 2 w e eks 2 w e eks 2 w e eks + 2 weeks

66 Qualitative Research Methods KCE Report 187

2.4.5 Modalities of data collection

Delphi could be administrated ‘paper-and-pencil’ by mail or e-mail.

Online Delphi’s are more and more carried out. Software is available to support the data collection and the analysis (Delphi_Survey_Web (DSW)100, Mesydel©101)

The number of rounds is not necessarily defined a priori (often because of budgetary, time or human resources limitations): data collection must stop when the saturation or the consensus is reached.

2.4.6 Data collection tools

The Delphi method uses iterative (e-)mailed questionnaires in successive rounds. Because there is no interaction between the respondent and the researcher, the formulation of the questions has to be clear, and definitions should be given where necessary.

The questionnaire of the first round encompasses open-ended questions, to identify items to include in the second round.

Next rounds could be exclusively qualitative or composed of closed questions with scales (from totally agree to totally disagree, i.e. from 1 to 9), or combining both qualitative and quantitative questions. They present a synthesis of the results issued from the previous round.

In the case of closed questions, agreement is usually summarized by using the median and consensus assessed by presenting interquartile ranges for continuous numerical scales97. Graphical presentations of the results are welcomed.

In KCE reports the questionnaires used in each round are presented in appendices.

2.4.7 Sampling

Participants have to be carefully chosen because of their expertise, experience or knowledge in the field of the research question. In addition, the variety of positions in the field or opinions regarding the subject, should be covered. In that way, lay people could be added to increase the variety of viewpoints102.

They could be identified through publically available bibliographic information102. Snowballing recruitment could be useful to secure easy agreement to panelist invitation and strengthen panelist retention102.

There is no practical limit to the number of participants in a Delphi survey89.

2.4.8 Human resources necessary

The administrator of the survey develops the questionnaires, identifies, mobilizes and recruits participants, analyses findings and reports them. He/she is responsible for keeping a low attrition rate and insure the coherence between the different steps of the method.

Administrative support could be needed to (e-)mail the questionnaires and manage reminders and answers.

2.4.9 Practical aspects

• It is important to clearly explain the goal of the questionnaire and the way it will be analysed. The redaction of the invitation/introduction letter is thus crucial. “Stressing the practical policy application of the Delphi yield to experts panelists to aid their retention” (Rowe, 2011102, p. 1489).

• The research team should have managers skills to follow up the returned questionnaires and mailing.

• The utilization of online tools could be very useful as well for the research team (rapid results) as for the participants.

• While anonymity in the process of the Delphi is required, “using social rewards for recognition in participation, such as subsequently publishing panel membership listings” (Rowe, 2001102, p. 1489) could improve panelists recruitment and retention.

KCE Report 187 Qualitative Research Methods 67

2.4.10 Analysis

Each step of the Delphi requires a specific analysis.

In a classical Delphi, open-ended questions from round 1 should be content analysed ‘in order to group statements generated by the experts panel into similar areas’95.

Round that uses closed questions should be statistically analysed. Summary statistics are used to decide whether or not consensus is reached. The level of the consensus has to be defined in advance (i.e. 70% of agreement).

There is no agreement on the threshold indicating a consensus, nor how to choose this threshold95. Each researcher has to reflect on it, case by case. The proposals that have reached consensus should be eliminated from the next round.

2.4.11 Reporting of findings

Intermediary results are reported directly in the successive questionnaires. All the consensus and dissensus items are listed and discussed at the end of the process.

2.4.12 Quality criteria

It seems that no consensus exists with regards to the standard of methodological rigor to apply. And that “no definitive evidence exists which

demonstrates the reliability or validity of the technique” (Keeney, 201195, p.

104). This is partly due to the variety of the Delphi surveys and the constant evolutions in this field91.

We have not identified any checklists to assess the quality of a Delphi survey.

However, the following aspects of the survey could be assessed (adapted from Jillson103 and Hasson91):

• Applicability of the method to the specific research problem

• The quality of the composition of the Delphi panel. Participants have to be carefully chosen in function of their expertise and position in the group.

• Design and administration of the questionnaire • Feedback

A Delphi survey should be reviewed in terms of reliability, validity and trustworthiness to judge its worth91.

2.4.13 Examples of KCE reports using the method

• Impact of academic detailing on primary care physicians104

• Burnout among general practitioners: prevention and management72 • Methods for including public preference values in reimbursement

decision making processes for health interventions. Exploration of the feasibility of different models in Belgium (ongoing project, publication foreseen end 2012)