9 Challenge Analysis
A CAUTIONARY NOTE
9.7 DELPHI METHOD
D
elphi is a method for eliciting ideas, judgments, or forecasts from a group of experts who may be geographically dispersed. It is different from a survey in that there are two or more rounds ofquestioning. After the first round of questions, a moderator distributes all the answers and
explanations of the answers to all participants, often anonymously. The expert participants are then given an opportunity to modify or clarify their previous responses, if so desired, on the basis of what they have seen in the responses of the other participants. A second round of questions builds on the results of the first round, drills down into greater detail, or moves to a related topic. There is great flexibility in the nature and number of rounds of questions that might be asked.
When to Use It
The Delphi Method was developed by the RAND Corporation at the beginning of the Cold War in the 1950s to forecast the impact of new technology on warfare. It was also used to assess the probability, intensity, or frequency of future enemy attacks. In the 1960s and 1970s, Delphi became widely known and used as a method for futures research, especially forecasting long-range trends in science and technology. Futures research is similar to intelligence analysis in that the uncertainties and
complexities one must deal with often preclude the use of traditional statistical methods, so explanations and forecasts must be based on the experience and informed judgments of experts.
Over the years, Delphi has been used in a wide variety of ways, and for an equally wide variety of purposes. Although many Delphi projects have focused on developing a consensus of expert
judgment, a variant called Policy Delphi is based on the premise that the decision maker is not interested in having a group make a consensus decision, but rather in having the experts identify alternative policy options and present all the supporting evidence for and against each option. That is the rationale for including Delphi in this chapter on challenge analysis. It can be used to identify divergent opinions that may be worth exploring.
One group of Delphi scholars advises that the Delphi technique “can be used for nearly any
problem involving forecasting, estimation, or decision making”—as long as the problem is not so complex or so new as to preclude the use of expert judgment. These Delphi advocates report using it for diverse purposes that range from “choosing between options for regional development, to
predicting election outcomes, to deciding which applicants should be hired for academic positions, to predicting how many meals to order for a conference luncheon.”18
Value Added
One of Sherman Kent’s “Principles of Intelligence Analysis,” which are taught at the CIA’s Sherman Kent School for Intelligence Analysis, is “Systematic Use of Outside Experts as a Check on In-House Blinders.” Consultation with relevant experts in academia, business, and nongovernmental
organizations is also encouraged by Intelligence Community Directive No. 205, on Analytic Outreach, dated July 2008. As an effective process for eliciting information from outside experts, Delphi has several advantages:
* Outside experts can participate from their home locations, thus reducing the costs in time and travel commonly associated with the use of outside consultants.
* Delphi can provide analytic judgments on any topic for which outside experts are available.
That means it can be used as an independent cross-check of conclusions reached in house. If the same conclusion is reached in two analyses using different analysts and different methods, this is grounds for a significant increase in confidence in that conclusion. If the conclusions disagree, this is also valuable information that may open a new avenue of research.
* Delphi identifies any outliers who hold an unusual position. Recognizing that the majority is not always correct, researchers can then focus on gaining a better understanding of the grounds for any views that diverge significantly from the consensus. In fact, identification of experts who have an alternative perspective and are qualified to defend it might be the objective of a Delphi project.
* The process by which the expert panel members are provided feedback from other experts and are given an opportunity to modify their responses makes it easy for experts to adjust their previous judgments in response to new evidence.
* In many Delphi projects, the experts remain anonymous to other panel members so that no one can use his or her position of authority, reputation, or personality to influence others. Anonymity also facilitates the expression of opinions that go against the conventional wisdom and may not otherwise be expressed.
On the down side, a Delphi project involves administrative work to identify the experts,
communicate with panel members, and collate and tabulate their responses through several rounds of questioning. Several software programs have been developed to handle these tasks, and one of these is hosted for public use (http://armstrong.wharton.upenn.edu/delphi2). The distributed decision support systems now publicly available to support virtual teams include some or all of the functions necessary for Delphi as part of a larger package of analytic tools.
We believe the development of Delphi panels of experts on areas of critical concern should be standard procedure for outreach to experts outside the Intelligence Community.
The Method
In a Delphi project, a moderator (analyst) sends a questionnaire to a panel of experts who may be in different locations. The experts respond to these questions and usually are asked to provide short explanations for their responses. The moderator collates the results from this first questionnaire and sends the collated responses back to all panel members, requesting them to reconsider their responses based on what they see and learn from the other experts’ responses and explanations. Panel members may also be asked to answer another set of questions. This cycle of question, response, and feedback continues through several rounds using the same or a related set of questions. It is often desirable for panel members to remain anonymous so that they are not unduly influenced by the responses of senior members. This method is illustrated in Figure 9.7.
Examples
To show how Delphi can be used for intelligence analysis, we have developed three illustrative applications:
* Evaluation of another country’s policy options: The Delphi project manager or moderator identifies several policy options that a foreign country might choose. The moderator then asks a panel of experts on the country to rate the desirability and feasibility of each option, from the other
country’s point of view, on a five-point scale ranging from “Very Desirable” or “Feasible” to “Very Undesirable” or “Definitely Infeasible.” Panel members are also asked to identify and assess any other policy options that ought to be considered and to identify the top two or three arguments or items of evidence that guided their judgments. A collation of all responses is sent back to the panel with a request for members to do one of the following: reconsider their position in view of others’
responses, provide further explanation of their judgments, or reaffirm their previous response. In a second round of questioning, it may be desirable to list key arguments and items of evidence and ask the panel to rate them on their validity and their importance, again from the other country’s
perspective.
Figure 9.7 The Delphi Technique
* Analysis of Alternative Hypotheses: A panel of outside experts is asked to estimate the
probability of each hypothesis in a set of mutually exclusive hypotheses where the probabilities must add up to 100 percent. This could be done as a stand-alone project or to double-check an already completed Analysis of Competing Hypotheses (ACH) analysis (chapter 7). If two analyses using different analysts and different methods arrive at the same conclusion, this is grounds for a significant increase in confidence in the conclusion. If the analyses disagree, that may also be useful to know as one can then seek to understand the rationale for the different judgments.
* Warning analysis or monitoring a situation over time: An analyst asks a panel of experts to estimate the probability of a future event. This might be either a single event for which the analyst is monitoring early warning indicators or a set of scenarios for which the analyst is monitoring
milestones to determine the direction in which events seem to be moving. There are two ways to manage a Delphi project that monitors change over time. One is to have a new round of questions and responses at specific intervals to assess the extent of any change. The other is what is called either Dynamic Delphi or Real Time Delphi where participants can modify their responses at any time as new events occur or as new information is submitted by one of the participants.19 The probability estimates provided by the Delphi panel can be aggregated to provide a measure of the significance of change over time. They can also be used to identify differences of opinion between the experts that
warrant further examination.
Relationship to Other Techniques
Delphi is easily combined with other techniques, such as Virtual Brainstorming, and techniques for prioritizing, ranking, or scaling lists of information.
Origins of This Technique
The origin of Delphi as an analytic method was described under “When to Use It.” The following references were useful in researching this topic: Murray Turoff and Starr Roxanne Hiltz, “Computer Based Delphi Processes,” 1996, http://web.njit.edu/~turoff/Papers/delphi3.html; and Harold A.
Linstone and Murray Turoff, The Delphi Method: Techniques and Applications (Reading, Mass.:
Addison-Wesley, 1975). A 2002 digital version of Linstone and Turoff’s book is available online at http://is.njit.edu/pubs/delphibook; see in particular the chapter by Turoff on “The Policy Delphi”
(http://is.njit.edu/pubs/delphibook/ch3b1.pdf). For more current information on validity and optimal techniques for implementing a Delphi project, see Gene Rowe and George Wright, “Expert Opinions in Forecasting: The Role of the Delphi Technique,” in Principles of Forecasting, ed. J. Scott
Armstrong (New York: Springer Science+Business Media, 2001).
1. Rob Johnston, Analytic Culture in the U.S. Intelligence Community (Washington, D.C.: CIA Center for the Study of Intelligence, 2005), 64.
2. Paul Bedard, “CIA Chief Claims Progress with Intelligence Reforms,” U.S. News and World Report, May 16, 2008, www.usnews.com/articles/news/2008/05/16/cia-chief-claims-with-intelligence-reforms.html.
3. Donald P. Steury, ed., Sherman Kent and the Board of National Estimates: Collected Essays (Washington, D.C.: CIA Center for the Study of Intelligence, 1994), 133.
4. Reframing is similar to the Problem Restatement technique Morgan Jones described in his book, The Thinker’s Toolkit (New York:
Three Rivers Press, 1995). Jones observed that “the moment we define a problem our thinking about it quickly narrows considerably.”
We create a frame through which we view the problem and which tends to obscure other interpretations of the problem. A group can change that frame of reference, and challenge its own thinking, simply by redefining the problem.
5. Gary Klein, Sources of Power: How People Make Decisions (Cambridge, Mass.: MIT Press, 1998), 71.
6. Gary Klein, Intuition at Work: Why Developing Your Gut Instinct Will Make You Better at What You Do (New York, Doubleday, 2002), 91.
7. Charlan J. Nemeth and Brendan Nemeth-Brown, “Better than Individuals? The Potential Benefits of Dissent and Diversity for Group Creativity,” in Group Creativity, ed. Paul B. Paulus and Bernard A Nijstad (New York: Oxford University Press, 2003), 63.
8. Leon Panetta, “Government: A Plague of Incompetence,” Monterey County Herald, March 11, 2007, F1.
9. Gregory F. Treverton, “Risks and Riddles,” Smithsonian Magazine, June 1, 2007.
10. For information about these three decision-making models, see Graham T. Allison and Philip Zelikov, Essence of Decision, 2nd ed.
(New York: Longman, 1999).
11. For information on fundamental differences in how people think in different cultures, see Richard Nisbett, The Geography of Thought: How Asians and Westerners Think Differently and Why (New York: Free Press, 2003).
12. Richards J. Heuer Jr., Psychology of Intelligence Analysis (Washington, D.C.: CIA Center for the Study of Intelligence, 1999), reprinted by Pherson Associates, LLC, 2007), 155.
13. Charlan J. Nemeth and Brendan Nemeth-Brown. “Better than Individuals? The Potential Benefits of Dissent and Diversity for Group Creativity,” in Group Creativity: Innovation Through Collaboration, ed. Paul B. Paulus and Bernard A. Nijstad (New York: Oxford University Press, 2003), 75–76.
14. Sherman Kent School for Intelligence Analysis training materials.
15. Nemeth and Brown, “Better than Individuals,” 77–78.
16. This definition is from the Red Team Journal, http://redteamjournal.com.
17. Defense Science Board Task Force, The Role and Status of DoD Red Teaming Activities (Washington, D.C.: Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics, September 2003).
18. Kesten C. Green, J. Scott Armstrong, and Andreas Graefe, “Methods to Elicit Forecasts from Groups: Delphi and Prediction Markets Compared,” Foresight: The International Journal of Applied Forecasting (Fall 2007),
www.forecastingprinciples.com/paperpdf/Delphi-WPlatestV.pdf.
19. See Real Time Delphi, www.realtimedelphi.org.