REVIEW ARTICLE
Delphi Technique- A Review
RamyaBalasubramanian
, Deepti AgarwalAbstract
Surveys play a major role in health sciences research. Delphi technique is a kind of detailed survey having diversified uses in dental research. It is a method for the systematic solicitation and collation of judgments on a particular topic through a set of carefully designed sequential questionnaires interspersed with summarized information and feedback of opinions derived from earlier responses. Many important clinical issues do not yield to randomized clinical trials or to stepwise quantitative data analysis. In such cases, professionals use their training and personal anecdotal experience to assist decision making in a variety of practice contexts. In these circumstances, consensus opinion of experts provides a formal structured process for decision making. Consensus procedures also use the resources of all participants, commit them to the project, and enhance the future decision-making ability of the group with regard to resolution of the clinical problem addressed. Delphi process consists of a series of rounds; in each round every participant worked through a questionnaire which has to be returned to the researcher who collected, edited, and returned to every participant, a statement of the position of the whole group and the participant‟s own position about the research issue. Every participant reassesses their initial judgments about the information provided in previous iterations. This article aims at describing the methodology of Delphi technique.
Keywords: Delphi; Survey; Consensus; Expert Opinion.
Introduction
Field of dentistry has seen lot of revolutions and developments. It keeps developing from heaps of research from various parts of the world. Surveys play a major role in health sciences research. Delphi technique is a kind of detailed survey having diversified uses in dental research. For a thousand years of recorded history the Greeks and other peoples, came to a temple in Delphi, a place in Greece to consult prophetess, who was called Pythia. They predicted the future and the divine purpose in a normative way in order to shape coming events to the people. The temple was the locus of knowledge, and the Delphic oracle was probably the largest database of the ancient world.1This gave the world the new technique called Delphi. The Delphi technique was initially developed by the Research and Development Corporation
(RAND), Santa Monica, California, in the early 1960s. The questions of RAND thinkers, at the time, primarily dealt with the military potential of future technology and potential political issues and their resolution. Olaf Helmer, Nicholas Rescher, Norman Dalkey, and others at RAND developed the Delphi method, which was designed to remove conference room impediments to a true expert consensus. 'Delphi' was intentionally coined by Kaplan, an associate professor of philosophy working for the RAND corporation in a research effort directed at improving the use of expert predictions in policy-making(1).
Delphi is a method for the “systematic solicitation and collation of judgments on a particular topic through a set of carefully designed sequential questionnaires interspersed
with summarized information and feedback of opinions derived from earlier responses” (2, 3). The majority of the Delphi efforts during the first decade were for pure forecasting, including both short- and long-range forecasts. Follow-up studies have demonstrated the validity and long-range accuracy of the Delphi technique. While most forecasting studies use Delphi to surface a consensus opinion, others such as the study by Kendall et al. emphasize differences of opinion in order to develop a set of alternative future scenarios. Concept / framework development represents a second type of application of the Delphi method (3).
The present review aims in providing a broader overview of the methodology of Delphi technique.
Consensus methods
Formal consensus methodology theorists argue that opinions of experts generated through structured circumstances can generate a closer approximation of the objective truth than would be achieved through conventional, less formal, and pooling of expert opinion. The methods include the nominal group technique, the Delphi technique, the Glaser „„state-of-the-art technique,‟‟etc.The „„Delphi technique‟‟ and the „„nominal group technique‟‟ are the two methods with the longest history (4).
„Consensus Methods‟ commonly used to synthesise information from conflicting evidence. Consensus methods are primarily concerned with deriving quantitative estimates through qualitative approaches. Delphi Methods concentrate on measuring consensus whereas
the Nominal Group Technique can be used to develop consensus but without the limitations of more informal methods of reaching consensus such as committees, which are prone to domination by powerful individuals and influenced by personalities (5).
Delphi methodology
The Delphi method is based on structural surveys and makes use of the intuitive available information of the participants, who are mainly experts (1).
Many important clinical issues do not yield to randomized clinical trials or to stepwise quantitative data analysis. In such cases, professionals use their training and personal anecdotal experience to assist decision making in a variety of practice contexts. In these circumstances, consensus opinion of experts provides a formal structured process for decision making. Consensus procedures also use the resources of all participants, commit them to the project, and enhance the future decision-making ability of the group with regard to resolution of the clinical problem addressed (4).
The Delphi method is mainly used when long-term issues have to be assessed. It reduces the tacit and complex knowledge to a single statement and makes it possible to judge upon. Content of Delphi studies are always issues about which, unsure and incomplete knowledge exists. Delphi are judgement processes with unsure aspects. For the participation experts are to be involved who on the basis of their knowledge and experience are able to assess in a competent way (1). (Table 1)
Table 1: Comparison between normal surveys and Delphi surveys (3)
Evaluation criteria Traditional survey Delphi study
Representativeness of
sample
Using statistical sampling techniques, the researchers randomly select a sample that is representative of the population of interest.
A Delphi study is a virtual panel of experts gathered to arrive at an answer to a difficult question. Thus, a Delphi study could be considered a type of virtual meeting or as a group decision technique, though it appears to be a complicated survey.
Sample size for statistical
power and significant
findings
Because the goal is to generalize results to a larger population, the researchers need to select a sample size that is large enough to detect statistically significant effects in the population.
The Delphi group size does not depend on statistical power, but rather on group dynamics for arriving at consensus among experts. Thus, the literature recommends 10–18 experts on a Delphi panel.
Anonymity
Respondents are almost always anonymous to each other, and often anonymous to the researcher.
Respondents are always anonymous to each other, but never anonymous to the researcher. This gives the researchers more opportunity to follow up for clarifications and further qualitative data.
Non-response issues
Researchers need to investigate the possibility of non-response bias to ensure that the sample remains representative of the population.
Non-response is typically very low in Delphi surveys, since most researchers have obtained assurance of participation
Attrition effects
For single surveys, attrition (participant drop-out) is a non-issue. For multi-step repeated survey studies, researcher should investigate attrition to assure that it is random and non-systematic.
Similar to non-response, attrition tends to be low in Delphi studies, and the researchers usually can easily ascertain the cause by talking with the dropouts.
Richness of data
The richness of data depends on the form and depth of the questions, and on the possibility of follow-up, such as interviews.
Delphi studies inherently provide richer data because of their multiple iterations and their response revision due to feedback.
Issues for which Delphi is commonly used (6,7)
In the literature, Delphi has been applied in various fields such as program planning, needs assessment, policy determination, and resource utilization. Delbecq, Van de Ven, and Gustafson (1975) specifically indicate that the Delphi technique can be used for achieving the following objectives:
1. To determine or develop a range of possible program alternatives.
2. To explore or expose underlying assumptions or information leading to different judgments.
3. To seek out information, which may generate a consensus.
4. To correlate informed judgments on a topic spanning a wide range of disciplines.
5. To educate the respondent group as to the diverse and interrelated aspects of the topic.
Implementing a Delphi technique
The Delphi technique is well suited as a means and method for consensus-building by using a series of questionnaires to collect data from a panel of selected subjects Delphi, in contrast to other data gathering and analysis techniques, employs multiple iterations. Iterations refer to the
feedback process. The process consists of a series of rounds; in each round every participant worked through a questionnaire which has to be returned to the researcher who collected, edited, and returned to every participant, a statement of the position of the whole group and the participant‟s own position about the research issue. Every participant reassesses their initial judgments about the information provided in previous rounds. More specifically, the feedback process allows the selected Delphi participants to reassess their initial judgments about the information provided in previous iterations. Thus, in a Delphi study, the results of previous iterations regarding specific statements and/or items can change or be modified by individual panel members in later iterations based on their ability to review and assess the comments and feedback provided by the other Delphi panellists (6).
Theoretically, the Delphi process can be continued until consensus is determined to have been achieved. However, Cyphert and Gant (1971), Brooks (1979), Ludwig (1994, 1997), and Custer, Scarcella, and Stewart (1999) point out that three iterations are often sufficient to collect the needed information and to reach a consensus in most cases (6).
Steps in Delphi
Round 1: In the first round, the Delphi process traditionally begins with an open-ended questionnaire (2). The open-ended questionnaire serves as the cornerstone of soliciting specific information about content. The questionnaire could contain ten to fifteen open-ended questions. After receiving subject‟s responses, investigators need to convert the
collected information into a well-structured questionnaire. This questionnaire is used as the survey instrument for the second round of data collection (6). After the first round the data is analyzed and responses to questions could be grouped or categorized by frequency or other criteria (2). The Delphi questionnaires will be administered using e-mail, fax, and the web. The panellists will be free to use whichever of these media was most convenient (3).
Round 2: In the second round, each Delphi participant receives a second questionnaire and is asked to review the items summarized by the investigators based on the information provided in the first round. Accordingly, Delphi panellists may be required to rate or rank-order items to establish preliminary priorities among items. As a result of round two, areas of disagreement and agreement are identified (2, 6).
Round 3: In the third round, each Delphi panellists receives a questionnaire that includes the items and ratings summarized by the investigators in the previous round and are asked to revise his/her judgments or to specify the reasons for remaining outside the consensus. This round gives Delphi panellists an opportunity to make further clarifications of both the information and their judgments of the relative importance of the items. However, compared to the previous round, only a slight increase in the degree of consensus can be expected (2, 6).
Round 4: In the fourth and often final round, the list of remaining items, their ratings, minority opinions, and items achieving consensus are distributed to the panellists. This round provides a final opportunity for participants to revise their
judgments. It should be remembered that the number of Delphi iterations depends largely on the degree of consensus sought by the investigators and can vary from three to five (6).
Subject selection
Regarding the selection of subjects for a Delphi study, choosing the appropriate subjects is the most important step in the entire process because it directly relates to the quality of the results generated. Delphi subjects should be highly trained and competent within the specialized area of knowledge related to the target issue. Investigators need to closely examine and seriously consider the qualifications of Delphi subjects (2, 6).
Linstone, Turoff, and Moore recommend a panel of 15 to 30 participants from the same discipline, or 5 to 10 per category from different professional groupings (8, 9).
Delbecq, Van de Ven, and Gustafson (1975) recommend that researchers should use the minimally sufficient number of subjects and should seek to verify the results through follow-up explorations ten to fifteen subjects could be sufficient if the background of the Delphi subjects is homogeneous. In contrast, if various reference groups are involved in a Delphi study, more subjects are needed. If the sample size of a Delphi study is too small, these subjects may not be considered as a representative pooling of judgments regarding the target issue. If the sample size is too large, the drawbacks inherent within the Delphi technique such as potentially low response rates and the obligation of large blocks of time by the respondents and the researchers can be the result (6).
Procedure for selecting experts (3)
1. Prepare a Knowledge Resource Nomination Worksheet (KRNW): The purpose of the Knowledge Resource Nomination Worksheet is to help categorize the experts before identifying them, in order to prevent overlooking any important class of experts. 2. Populating the KRNW with names: After the
KRNW is completed, the following iterative procedure will be used to populate the categories with actual names of potential experts for the Delphi study. Each heading (disciplines, organizations, and literature) represents a different lens for identifying and considering experts. However, this multiple lens perspective is necessary to identify as many experts as possible.
3. Rank experts: Create sub-lists, one for each discipline. Categorize experts according to appropriate list. Rank experts within each list based on their qualifications.
4. Inviting experts: Invite experts for each panel, with the panels corresponding to each discipline. Invite experts in the order of their ranking within their discipline sublist. Target size is 10-18. Stop soliciting experts when each panel size is reached.
Duration of the study
Delbecq, Van de Ven, and Gustafson (1975), Ulschak (1983), and Ludwig, (1994) recommend that a minimum of 45 days for the administration of a Delphi study is necessary. With regard to the time management between iterations, Delbecq et al. (1975) note that giving two weeks for Delphi subjects to respond to each round is encouraged (6).
Statistical analysis
In the Delphi process, data analysis can involve both qualitative and quantitative data. The major statistics used in Delphi studies are measures of central tendency and level of dispersion (standard deviation and inter-quartile range) in order to present information concerning the collective judgments of respondents (Hasson, Keeney, and McKenna, 2000). Generally, the uses of median and mode are favored. In the literature, the use of median score, based on Likert-type scale, is strongly favored (Hill and Fowles, 1975; Eckman, 1983; Jacobs, 1996) (2,6).
Arriving at conclusion
One criterion recommends that consensus is achieved by having 80 percent of subjects‟ votes falling within two categories on a seven-point scale (Ulschak, 1983). Green (1982) suggests that at least 70 percent of Delphi subjects need to rate three or higher on a four point Likert-type scale (5,6)and the median has to be at 3.25 or higher (6).
Time requirements
Delbecq, Van de Ven, and Gustafson (1975), Ulschak (1983), and Ludwig, (1994) recommend that a minimum of 45 days for the administration of a Delphi study is necessary. With regard to the time management between iterations, Delbecq et al. (1975) note that giving two weeks for Delphi subjects to respond to each round is encouraged (6).
Merits and demerits in technique Merits
1. One of the primary characteristics and advantages of the Delphi process is „subject anonymity‟ which can reduce the effects of
dominant individuals which often is a concern when using group-based processes used to collect and synthesize information (5,6). Additionally, the issue of confidentiality is facilitated by geographic dispersion of the subjects as well as the use of electronic communication such as e-mail to solicit and exchange information (6).
2. Controlled feedback process consists of a well-organized summary of the prior iteration intentionally distributed to the subjects which allows each participant an opportunity to generate additional insights and more thoroughly clarify the information developed by previous iterations (6).
3. The ability to use statistical analysis techniques further reduces the potential of group pressure for conformity (6).
4. The main advantage of the Delphi method is that there is no need for participants (e.g. busy professionals or managers) to meet up and, hence it is a relatively inexpensive method of gaining a large number of responses (5).
5. It also allows the involvement of participants from disparate geographical areas and has been used in international health research (5).
Demerits
1. Potential of Low Response Rates: due to the multiple feedback processes inherent and integral to the concept and use of the Delphi process, potential exists for low response rates and striving to maintain robust feedback can be a challenge (6).
2. Consumption of Large Blocks of Time: the Delphi technique can also be
time-consuming and laborious. Unlike other data collection techniques such as the telephone survey and the face-to-face administration, which can be simultaneously conducted by a group of people and can be completed in a short period of time if the sample size is small, the Delphi technique is iterative and sequential (6,7).
3. Potential of Molding Opinions: the iteration characteristics of the Delphi technique can potentially enable investigators to mold opinions. An assumption concerning Delphi participants is that they are equivalent in knowledge and experience. However, this assumption might not be justified (6).
4. Centralise opinion: the Delphi is a consensus method, it tries to obtain consensus and to „centralise opinion‟ and important minority issues may be missed due to nonconformity of general opinion. Loss of objectivity and researcher bias in analysing findings and generating questions are also possible (5).
Classification of the Delphi technique There are several types of Delphi (8)
The Classic Delphi
The Policy Delphi
The Decision Delphi
The Group Delphi
The Policy, Decision and Group Delphi are variations of the Classic Delphi.
Another author classifies Delphi as conventional, real-time and policy. The conventional Delphi is the classical forum for the prioritization of facts. It consists of a questionnaire sent out to a group
of experts, with a second questionnaire based on the results of the first. Subsequent questionnaires refine and define the facts or proposals, gauging their accuracy or support from the participants. The real-time or modified Delphi is a shorter variant, where the process takes place during the course of a meeting, using mechanisms to summarize responses to the respondents immediately. The policy Delphi is a forum for ideas where the decision maker is interested in having informed group present options and supporting evidence rather than having a group reach a decision (9).
The Delphi technique in Public Health
The Delphi technique is becoming increasingly popular in health and social research and has been used for various purposes in obtaining a wide variety of outcomes.
In the Indian scenario to arrive at the consensus regarding the various health sector reforms a Delphi study was conducted in 2000 and 2001 among experts. The questions were: (i) within the context of health sector reforms, what are the priority issues for India? (ii) is there a consensus in the country on the health sector reforms? If yes, in which areas? (iii) what are the areas where differences exist? And (iv) how should the identified priority issues be addressed? (10). Examples of Delphi in public health include „Policy Delphi‟ applied to achieve national level policy making on child health indicators in Hong Kong (11). In a recent study, a modified Delphi technique was used to develop international policy options for severe acute respiratory syndrome (SARS) and SARS-like emerging infectious diseases and its evaluation reported it to be an effective tool in
certain Public Health debates. This method was also used in evaluation of Public health functions (12-14).A Delphi study was conducted regarding reproductive and maternal health in United Kingdom (5).
Delphi researches in dentistry
The Delphi technique has also been used to achieve consensus of opinion on an array of oral health issues. Delphi studies were used in various branches of Dentistry for policy making, consensus decisions and even in developing dental curriculum (4). Delphi was conducted to find out factors influencing dental decision making (15), to identify core competencies in geriatric dentistry (16). Delphi technique was applied for deriving consensus regarding curriculum for periodontics (17). Modified Delphi survey was used for decision analysis regarding prophylaxis of post-radiation osteonecrosis (18).
A Delphi panel was used to survey criteria for successful periodontal therapy in anterior teeth (19) and also to study the current trends in restorative dentistry in the United Kingdom (20).
In the Indian scenario there are lot of dental public health issues remaining unresolved and the literature review shows very few Delphi studies has been carried out in this context. Consensus in implementing national oral health policy, priorities in oral health delivery and similar issues needed for oral health promotion can be subjected to Delphi research. A model example is the consensus study on the health sector reforms in India (10).
Conclusion
The Delphi method is a versatile research tool that researchers can employ at various points in their research. Use of the Delphi method for forecasting and issue identification/prioritization can be valuable in the early stages, particularly in selecting the topic and defining the research questions (3).
Based on literature review, it appears that Delphi is the most popular consensus method because of the need and value of obtaining consensus opinions and may be applied to evaluate clinical, educational, and policy issues in oral health care (7). It can be considered as a wonderful research tool to derive consensus and solutions in many of the unresolved oral health issues at national and international levels.
Affiliations of the authors:
1. Dr. Ramya Balasubramanian, Senior Lecturer, Department of Public Health Dentistry, R.V.S Dental College and Hospital, Coimbatore, Tamil Nadu, India, 2. Dr. Deepti Agarwal, Senior Lecturer, Department of Public Health Dentistry, Dr.D.Y.Patil Dental College, Pune, Maharashtra, India.Conflict of Interest:
The author(s) declared no conflict of interests.
Source of Funding: Nil.
References
1. Cuhls K. Delphi method. Fraunhofer Institute for Systems and Innovation Research,
Germany. Available at.
http://www.unido.org/fileadmin/import/16959 _DelphiMethod. pdf Accessed on: February 2011.
2. Gould D. Delphi study methodology. Fifth Generation Work - Virtual Organization, Leadership in Virtual Teams, Chapter 3,
76-83. Available at: http://www.seanet.com/
~daveg/Chapter3.pdf. Accessed on:
February 2011.
3. Okoli C, Pawlowski SD. The Delphi method as a research tool: an example, design considerations and applications. Information and Management2004; 42:15–29.
4. Cramer CK, Klasser GD, Epstein JB, Sheps SB. The Delphi Process in Dental Research. J Evid Base Dent Pract 2008;8:211-220. 5. Van E, Pitchforth T, Bishop C, Russell E.
Delphi method and nominal group techniques in family planning and reproductive health research. J Family Planning Reprod Health Care 2006;32(4):249-252.
6. Hsu CC, Sandford BA. The Delphi Technique: Making Sense of consensus. Practical Assessment, Research and Evaluation 2007;12(10):1-8.
7. Gordon TJ. The Delphi method. The Millennium Project Futures Research Methodology. 3:1-29. Available at www.millenium-project.org/millenium/RTD-method.pdf. Accessed on: February 2012. 8. Syed M, Camp R. The POLKA Delphi Study
- Policy Recommendations on Rare Diseases. 1 - 61 Available at:www.eurorids.org/documents/pdf/full_rep ort_delphi Accessed on: February 2012. 9. Marietjier De Villiers, Pierre JT. De Villier L,
Athol P Kent. The Delphi technique in health sciences education research. Med Teach 2005;27(7):639-643.
10. Anand K, Pandav CS, Kapoor SK. Consensus and conflicts in health sector
reforms in India: A Delphi study. Nat Med J India 2002; 15:221-226.
11. A pilot study for public health policy model and development indicator for child health in Hong Kong. Central policy unit Hong Kong special administrative region. The Chinese University of Hong Kong. 2008; 1-6. Available at :www.cpu.gov.hk/English/ documents/newpress.pdf. Accessed on: February 2012.
12. Syed AM, Hjarnoe L, Krumkamp R, Reintjes R, Aro AR. Developing policy options for SARS and SARS-like diseases - a Delphi study. Global Public Health 2010;16:1-13. 13. Syed AM, Hjarnoe L, Aro AR. The Delphi
Technique in developing international health policies: Experience from the SARS Control Project. The Internet Journal of Health 2009;8(2).
14. Chair WA. National Public Health Partnership Group. National Delphi Study on Public Health Functions, Report on Findings, January 2000. National Public Health Partnership online, updated 6 March 2002, Available at: www.nphp.gov.au /publications/phpractice/delphi-body. pdf Accessed on: March 2012.
15. Grembowsky D, Melgrom P, Fisset L. Factors influencing dental decision making. J Publ Health Dent 1988; 48:159-167. 16. Dolan TA, Lauer DS. Delphi study to identify
core competencies in geriatric dentistry. Spec Care Dentist 2001; 21:191-197. 17. Cramer CK, Epstein JB, Sheps SB, Fried H,
Leao AT. Using Delphi technique in a consensual curriculum for periodontics. J Dent Educ 2007; 71(11):1441-1446.
18. Schechter MT, Busser JR. Modified Delphi survey for decision analysis for prophylaxis of post-radiation osteonecrosis. Oral Oncol 2002; 38:574-583.
19. Lightfoot WS, Hefti A, Mariotti A. Using a Delphi panel to survey criteria for successful periodontal therapy in anterior teeth. J Periodontol 2005; 76:1508-1512.
20. Randall RC, Vrijhoef MM, Wilson NH. Current trends in restorative dentistry in the
UK: a Delphi approach. J Dent 2002; 30:177-187.
Corresponding Author
Dr. Ramya Balasubramanian , MDS, Senior lecturer,Department of Public Health Dentistry, R.V.S Dental College and Hospital, Coimbatore, Tamil Nadu, India. email id – [email protected]
