9 Challenge Analysis
9.2 STRUCTURED SELF-CRITIQUE
S
tructured Self-Critique is a systematic procedure that a small team or group can use to identify weaknesses in its own analysis. All team or group members don a hypothetical black hat and become critics rather than supporters of their own analysis. From this opposite perspective, they respond to a list of questions about sources of uncertainty, the analytic processes that were used, criticalassumptions, diagnosticity of evidence, anomalous evidence, information gaps, changes in the broad environment in which events are happening, alternative decision models, availability of cultural expertise, and indicators of possible deception. As it reviews responses to these questions, the team reassesses its overall confidence in its own judgment.
“Begin challenging your own assumptions. Your assumptions are your windows on the world. Scrub them off every once in a while, or the light won’t come in.”
—Alan Alda, American actor
When to Use It
You can use Structured Self-Critique productively to look for weaknesses in any analytic explanation of events or estimate of the future. It is specifically recommended for use in the following ways:
As the next step after a Premortem Analysis raises unresolved questions about any estimated future outcome or event.
As a double check prior to the publication of any major product such as a National Intelligence Estimate.
As one approach to resolving conflicting opinions (as discussed in chapter 10 under Adversarial Collaboration).
The amount of time required to work through the Structured Self-Critique will vary greatly depending upon how carefully the previous analysis was done. The questions listed in the method as described later in this section are actually just a prescription for very careful analysis. To the extent that these same questions have been explored during the initial analysis, the time required for the Structured Self-Critique is reduced. If these questions are being asked for the first time, the process will take longer. As analysts gain experience with the self-critique, they may have less need for certain parts of it, as those parts will have been internalized and will have been done during the initial analysis (as they should have been).
Value Added
When people are asked questions about the same topic but from a different perspective, they often give different answers than the ones they gave before. For example, if a team member is asked if he or she supports the team’s conclusions, the answer will usually be “yes.” However, if all team members are asked to look for weaknesses in the team’s argument, that member may give a quite different response.
This change in the frame of reference is intended to change the group dynamics. The critical perspective should always generate more critical ideas. Team members who previously may have suppressed questions or doubts because they lacked confidence or wanted to be good team players are now empowered to express those divergent thoughts. If the change in perspective is handled well, all team members will know that they win points with their colleagues for being critical of the
previous judgment, not for supporting it.
Potential Pitfalls
The success of this technique depends in large measure on the team members’ willingness and ability to make the transition from supporters to critics of their own ideas. Some individuals lack the
intellectual flexibility to do this well. It must be very clear to all members that they are no longer performing the same function as before. Their new task is to critique an analytic position taken by some other group (actually themselves but with a different hat on).
To emphasize the different role analysts are playing, Structured Self-Critique meetings should be scheduled exclusively for this purpose. The meetings should be led by a different person from the usual leader, and, preferably, held at a different location. It will be helpful if an experienced facilitator is available to lead the meeting(s). This formal reframing of the analysts’ role from advocate to critic is an important part of helping analysts see an issue from a different perspective.
The Method
Start by re-emphasizing that all analysts in the group are now wearing a black hat. They are now critics, not advocates, and they will now be judged by their ability to find weaknesses in the previous analysis, not on the basis of their support for the previous analysis. Then work through the following topics or questions:
Sources of uncertainty: Identify the sources and types of uncertainty in order to set reasonable expectations for what the team might expect to achieve. Should one expect to find: (a) a single correct or most likely answer, (b) a most likely answer together with one or more alternatives that must also be considered, or (c) a number of possible explanations or scenarios for future development? To judge the uncertainty, answer these questions:
Is the question being analyzed a puzzle or a mystery? Puzzles have answers, and correct answers can be identified if enough pieces of the puzzle are found. A mystery has no single definitive answer; it depends upon the future interaction of many factors, some known and others unknown. Analysts can frame the boundaries of a mystery only “by identifying the critical factors and making an intuitive judgment about how they have interacted in the past and might interact in the future.”9
How does the team rate the quality and timeliness of its evidence?
Are there a greater than usual number of assumptions because of insufficient evidence or the complexity of the situation?
Is the team dealing with a relatively stable situation or with a situation that is undergoing, or potentially about to undergo, significant change?
Analytic process: In the initial analysis, did the team do the following. Did it identify alternative hypotheses and seek out information on these hypotheses? Did it identify key assumptions? Did it
seek a broad range of diverse opinions by including analysts from other offices, agencies, academia, or the private sector in the deliberations? If these steps were not taken, the odds of the team having a faulty or incomplete analysis are increased. Either consider doing some of these things now or lower the team’s level of confidence in its judgment.
Critical assumptions: Assuming that the team has already identified key assumptions, the next step is to identify the one or two assumptions that would have the greatest impact on the analytic judgment if they turned out to be wrong. In other words, if the assumption is wrong, the judgment will be wrong. How recent and well-documented is the evidence that supports each such assumption? Brainstorm circumstances that could cause each of these assumptions to be wrong, and assess the impact on the team’s analytic judgment if the assumption is wrong. Would the reversal of any of these assumptions support any alternative hypothesis? If the team has not previously identified key assumptions, it should do a Key Assumptions Check now.
Diagnostic evidence: Identify alternative hypotheses and the several most diagnostic items of evidence that enable the team to reject alternative hypotheses. For each item, brainstorm for any reasonable alternative interpretation of this evidence that could make it consistent with an alternative hypothesis. See Diagnostic Reasoning in chapter 7.
Information gaps: Are there gaps in the available information, or is some of the information so dated that it may no longer be valid? Is the absence of information readily explainable? How should absence of information affect the team’s confidence in its conclusions?
Missing evidence: Is there any evidence that one would expect to see in the regular flow of intelligence or open source reporting if the analytic judgment is correct, but that turns out not to be there?
Anomalous evidence: Is there any anomalous item of evidence that would have been important if it had been believed or if it could have been related to the issue of concern, but that was rejected as unimportant because it was not believed or its significance was not known? If so, try to imagine how this item might be a key clue to an emerging alternative hypothesis.
Changes in the broad environment: Driven by technology and globalization, the world as a whole seems to be experiencing social, technical, economic, environmental, and political changes at a faster rate than ever before in history. Might any of these changes play a role in what is happening or will happen? More broadly, what key forces, factors, or events could occur independently of the issue that is the subject of analysis that could have a significant impact on whether the analysis proves to be right or wrong?
Alternative decision models: If the analysis deals with decision making by a foreign government or nongovernmental organization (NGO), was the group’s judgment about foreign behavior based on a rational actor assumption? If so, consider the potential applicability of other decision models, specifically that the action was or will be the result of bargaining between political or bureaucratic forces, the result of standard organizational processes, or the whim of an authoritarian leader.10 If information for a more thorough analysis is lacking, consider the implications of that for confidence in the team’s judgment.
Cultural expertise: If the topic being analyzed involves a foreign or otherwise unfamiliar culture or subculture, does the team have or has it obtained cultural expertise on thought processes in that culture?11
Deception: Does another country, NGO, or commercial competitor about which the team is making judgments have a motive, opportunity, or means for engaging in deception to influence U.S. policy or to change your behavior? Does this country, NGO, or competitor have a past
history of engaging in denial, deception, or influence operations?
After responding to these questions, the analysts take off the black hats and reconsider the appropriate level of confidence in the team’s previous judgment. Should the initial judgment be reaffirmed or modified?
Relationship to Other Techniques
One version of what has been called Devil’s Advocacy is similar to Structured Self-Critique in that one member of the team is designated to play the role of Devil’s Advocate. That member takes one of the team’s critical assumptions, reverses it, and then argues from that perspective against the team’s conclusions. We believe it is more effective for the entire team to don the hypothetical black hat and play the role of critic. It should not be one team member trying to persuade the rest of the team that they are wrong while acting out a role that he or she may actually disagree with. See information on Devil’s Advocacy later in this chapter.
Origins of This Technique
Richards Heuer and Randy Pherson developed Structured Self-Critique. A simpler version of this technique appears in Randy Pherson, “The Pre-Mortem Assessment,” in Handbook of Analytic Tools and Techniques (Reston, Va.: Pherson Associates, LLC, 2008).