9 Challenge Analysis
9.6 Red Team Analysis 9.7 Delphi Method
CHAPTER
9
Challenge Analysis
C
hallenge analysis encompasses a set of analytic techniques that have also been called contrarian analysis, alternative analysis, competitive analysis, red team analysis, and devil’s advocacy. What all of these have in common is the goal of challenging anestablished mental model or analytic consensus in order to broaden the range of possible explanations or estimates that are seriously considered. The fact that this same activity has been called by so many different names suggests there has been some conceptual diversity about how and why these techniques are being used and what might be accomplished by their use.
There is a broad recognition in the Intelligence Community that failure to question a consensus judgment, or a long-established mental model, has been a consistent feature of most significant intelligence failures. The postmortem analysis of virtually every major U.S. intelligence failure since Pearl Harbor has identified an analytic mental model
(mindset) as a key factor contributing to the failure. The situation changed, but the analyst’s mental model did not keep pace with that change or did not recognize all the ramifications of the change.
This record of analytic failures has generated discussion about the “paradox of
expertise.”1 The experts can be the last to recognize the reality and significance of change.
For example, few experts on the Soviet Union foresaw its collapse, and the experts on Germany were the last to accept that Germany was going to be reunified. Going all the way back to the Korean War, experts on China were saying that China would not enter the war—
until it did.
As we noted in chapter 1, an analyst’s mental model can be regarded as a distillation of everything the analyst knows about how things normally work in a certain country or a specific scientific field. It tells the analyst, sometimes subconsciously, what to look for, what’s important, and how to interpret what he or she sees. A mental model formed through
education and experience serves an essential function; it is what enables the analyst to provide on a daily basis reasonably good intuitive assessments or estimates about what is happening or likely to happen.
The problem is that a mental model that has previously provided accurate assessments and
estimates for many years can be slow to change. New information received incrementally over time is easily assimilated into one’s existing mental model, so the significance of gradual change over time is easily missed. It is human nature to see the future as a continuation of the past. As a general rule, major trends and events evolve slowly, and the future is often foreseeable to skilled intelligence analysts. However, life does not always work this way. The most significant intelligence failures have been failures to foresee historical discontinuities, when history pivots and changes direction.
Such surprising events are not foreseeable unless they are first imagined so that one can start
examining the world from a different perspective. That is what this chapter is about—techniques that enable the analyst, and eventually the intelligence consumer, to evaluate events from a different perspective—in other words, with a different mental model.
“What gets us into trouble is not what we don’t know, it’s what we know for sure that just ain’t so.”
—Mark Twain, American author and humorist
There is also another logical rationale for consistently challenging conventional wisdom.
Former CIA Director Michael Hayden has stated that “our profession deals with subjects that are inherently ambiguous, and often deliberately hidden. Even when we’re at the top of our game, we can offer policymakers insight, we can provide context, and we can give them a clearer picture of the issue at hand, but we cannot claim certainty for our judgments.” The director went on to suggest that getting it right seven times out of ten might be a realistic expectation.2
Director Hayden’s estimate of seven times out of ten is supported by a quick look at verbal expressions of probability used in intelligence reports. “Probable” seems to be the most common verbal expression of the likelihood of an assessment or estimate. Unfortunately, there is no consensus within the Intelligence Community on what “probable” and other verbal expressions of likelihood mean when they are converted to numerical percentages. For discussion here, we accept Sherman Kent’s definition of “probable” as meaning “75% plus or minus 12%.”3 This means that analytic judgments described as “probable” are expected to be correct roughly 75 percent of the time—and, therefore, incorrect or off target about 25 percent of the time.
Logically, therefore, one might expect that one of every four judgments that intelligence analysts describe as “probable” will turn out to be wrong. This perspective broadens the scope of what challenge analysis might accomplish. It should not be limited to questioning the dominant view to be sure it’s right. Even if the challenge analysis confirms the initial probability judgment, it should go further to seek a better understanding of the other 25 percent. In what circumstances might there be a different assessment or outcome, what would that be, what would constitute evidence of events
moving in that alternative direction, how likely is it, and what would be the consequences? As we will discuss in the next chapter, on conflict management, an understanding of these probabilities should reduce the frequency of unproductive conflict between opposing views. Analysts who
recognize a one in four chance of being wrong should at least be open to consideration of alternative assessments or estimates to account for the other 25 percent.
This chapter describes three types of challenge analysis techniques: self-critique, critique of others, and solicitation of critique by others.
* Self-critique: Two techniques that help analysts challenge their own thinking are Premortem Analysis and Structured Self-Critique. These techniques can counteract the pressures for conformity or consensus that often suppress the expression of dissenting opinions in an analytic team or group.
We adapted Premortem Analysis from business and applied it to intelligence analysis.
* Critique of others: Analysts can use What If? Analysis or High Impact/Low Probability Analysis to tactfully question the conventional wisdom by making the best case for an alternative explanation or outcome.
* Critique by others: Several techniques are available for seeking out critique by others.
Devil’s Advocacy is a well-known example of that. The term “Red Team” is used to describe a group that is assigned to take an adversarial perspective. The Delphi Method is a structured process for eliciting opinions from a panel of outside experts.
Reframing Techniques
Three of the techniques in this chapter work by a process called reframing. A frame is any cognitive structure that guides the perception and interpretation of what one sees. A mental model of how things normally work can be thought of as a frame through which an analyst sees and interprets evidence. An individual or a group of people can change their frame of reference, and thus challenge their own thinking about a problem, simply by changing the questions they ask or changing the perspective from which they ask the questions. Analysts can use this reframing technique when they need to generate new ideas, when they want to see old ideas from a new perspective, or any other time when they sense a need for fresh thinking.4
Reframing helps analysts break out of a mental rut by activating a different set of synapses in their brain. To understand the power of reframing and why it works, it is necessary to know a little about how the human brain works. The brain is now believed to have roughly 100 billion neurons, each analogous to a computer chip capable of storing information. Each neuron has octopus-like arms called axons and dendrites. Electrical impulses flow through these arms and are ferried by
neurotransmitting chemicals across the synaptic gap between neurons. Whenever two neurons are activated, the connections, or synapses, between them are strengthened. The more frequently those same neurons are activated, the stronger the path between them.
Once a person has started thinking about a problem one way, the same mental circuits or
pathways are activated and strengthened each time the person thinks about it. The benefit of this is that it facilitates the retrieval of information one wants to remember. The downside is that these pathways become mental ruts that make it difficult to see the information from a different perspective. When an analyst reaches a judgment or decision, this thought process is embedded in the brain. Each time the analyst thinks about it, the same synapses are triggered, and the analyst’s thoughts tend to take the
same well-worn pathway through the brain. Getting the same answer each time one thinks about it builds confidence, and often overconfidence, in that answer.
Fortunately, it is fairly easy to open the mind to think in different ways. The trick is to restate the question, task, or problem from a different perspective that activates a different set of synapses in the brain. Each of the three applications of reframing described in this chapter does this in a different way. Premortem Analysis asks analysts to imagine themselves at some future point in time, after having just learned that a previous analysis turned out to be completely wrong. The task then is to figure out how and why it might have gone wrong. What If? Analysis asks the analyst to imagine that some unlikely event has occurred, and then to explain how it could happen and the implications of the event. Structured Self-Critique asks a team of analysts to reverse its role from advocate to critic in order to explore potential weaknesses in the previous analysis. This change in role can empower analysts to express concerns about the consensus view that might previously have been suppressed.
These techniques are generally more effective in a small group than with a single analyst. Their effectiveness depends in large measure on how fully and enthusiastically participants in the group embrace the imaginative or alternative role they are playing. Just going through the motions is of limited value.
Overview of Techniques
Premortem Analysis reduces the risk of analytic failure by identifying and analyzing a potential failure before it occurs. Imagine yourself several years in the future. You suddenly learn from an unimpeachable source that your estimate was wrong. Then imagine what could have happened to cause the estimate to be wrong. Looking back from the future to explain something that has happened is much easier than looking into the future to forecast what will happen, and this exercise helps identify problems one has not foreseen.
Structured Self-Critique is a procedure that a small team or group uses to identify weaknesses in its own analysis. All team or group members don a hypothetical black hat and become critics rather than supporters of their own analysis. From this opposite perspective, they respond to a list of questions about sources of uncertainty, the analytic processes that were used, critical assumptions, diagnosticity of evidence, anomalous evidence, information gaps, changes in the broad environment in which
events are happening, alternative decision models, availability of cultural expertise, and indicators of possible deception. Looking at the responses to these questions, the team reassesses its overall
confidence in its own judgment.
What If? Analysis is an important technique for alerting decision makers to an event that could happen, or is already happening, even if it may seem unlikely at the time. It is a tactful way of suggesting to decision makers the possibility that they may be wrong. What If? Analysis serves a function similar to that of Scenario Analysis—it creates an awareness that prepares the mind to recognize early signs of a significant change, and it may enable a decision maker to plan ahead for that contingency. The analyst imagines that an event has occurred and then considers how the event could have unfolded.
High Impact/Low Probability Analysis is used to sensitize analysts and decision makers to the possibility that a low-probability event might actually happen and stimulate them to think about
measures that could be taken to deal with the danger or to exploit the opportunity if it does occur. The
analyst assumes the event has occurred, and then figures out how it could have happened and what the consequences might be.
Devil’s Advocacy is a technique in which a person who has been designated the Devil’s Advocate, usually by a responsible authority, makes the best possible case against a proposed analytic judgment, plan, or decision.
Red Team Analysis as described here is any project initiated by management to marshal the
specialized substantive, cultural, or analytic skills required to challenge conventional wisdom about how an adversary or competitor thinks about an issue. See also Red Hat Analysis in chapter 8.
Delphi Method is a procedure for obtaining ideas, judgments, or forecasts electronically from a
geographically dispersed panel of experts. It is a time-tested, extremely flexible procedure that can be used on any topic or issue for which expert judgment can contribute. It is included in this chapter because it can be used to identify divergent opinions that challenge conventional wisdom. It can also be used as a double check on any research finding. If two analyses from different analysts who are using different techniques arrive at the same conclusion, this is grounds for a significant increase in confidence in that conclusion. If the two conclusions disagree, this is also valuable information that may open a new avenue of research.