Assessment of Cause and Effect
8.1 KEY ASSUMPTIONS CHECK
A
nalytic judgment is always based on a combination of evidence and assumptions, orpreconceptions, which influence how the evidence is interpreted.2 The Key Assumptions Check is a systematic effort to make explicit and question the assumptions (the mental model) that guide an
analyst’s interpretation of evidence and reasoning about any particular problem. Such assumptions are usually necessary and unavoidable as a means of filling gaps in the incomplete, ambiguous, and
sometimes deceptive information with which the analyst must work. They are driven by the analyst’s education, training, and experience, plus the organizational context in which the analyst works.
“An organization really begins to learn when its most cherished assumptions are challenged by counterassumptions.
Assumptions underpinning existing policies and procedures should therefore be unearthed, and alternative policies and procedures put forward based upon counterassumptions.”
—Ian I. Mitroff and Richard O. Mason, Creating a Dialectical Social Science:
Concepts, Methods, and Models (1981).
Key Assumptions Check is one of the most commonly used techniques, because intelligence analysts typically need to make assumptions to fill gaps. These are often assumptions about another country’s intentions or capabilities, the way governmental processes usually work in that country, the relative strength of political forces, the trustworthiness or accuracy of key sources, the validity of previous analyses on the same subject, or the presence or absence of relevant changes in the context in which the activity is occurring. It can be difficult to identify assumptions, because many are socio-cultural beliefs that are held unconsciously or so firmly that they are assumed to be truth and not subject to challenge.
When to Use It
Any explanation of current events or estimate of future developments requires the interpretation of evidence. If the available evidence is incomplete or ambiguous, this interpretation is influenced by assumptions about how things normally work in the country of interest. These assumptions should be made explicit early in the analytic process.
If a Key Assumptions Check is not done at the outset of a project, it can still prove extremely valuable if done during the coordination process or before conclusions are presented or delivered. If the Key Assumptions Check was done early in the process, it is often desirable to review the
assumptions again later in the process, for example just before or just after drafting a report.
Determine whether the assumptions still hold up or should be modified.
Value Added
Preparing a written list of one’s working assumptions at the beginning of any project helps the analyst:
Identify the specific assumptions that underpin the basic analytic line.
Achieve a better understanding of the fundamental dynamics at play.
Gain a better perspective and stimulate new thinking about the issue.
Discover hidden relationships and links between key factors.
Identify any developments that would cause an assumption to be abandoned.
Avoid surprise should new information render old assumptions invalid.
A sound understanding of the assumptions underlying an analytic judgment sets the limits for the confidence the analyst ought to have in making a judgment.
The Method
The process of conducting a Key Assumptions Check is relatively straightforward in concept but often challenging to put into practice. One challenge is that participating analysts must be open to the possibility that they could be wrong. It helps to involve in this process several well-regarded analysts who are generally familiar with the topic but have no prior commitment to any set of assumptions about the issue at hand. Keep in mind that many “key assumptions” turn out to be “key uncertainties.”
Randy Pherson’s extensive experience as a facilitator of analytic projects indicates that approximately one in every four key assumptions collapses on careful examination.
Here are the steps in conducting a Key Assumptions Check:
* Gather a small group of individuals who are working the issue along with a few “outsiders.”
The primary analytic unit already is working from an established mental model, so the “outsiders” are needed to bring other perspectives.
* Ideally, participants should be asked to bring their list of assumptions when they come to the meeting. If this was not done, start the meeting with a silent brainstorming session. Ask each
participant to write down several assumptions on 3 × 5 cards.
* Collect the cards and list the assumptions on a whiteboard for all to see.
* Elicit additional assumptions. Work from the prevailing analytic line back to the key arguments that support it. Use various devices to help prod participants’ thinking:
Ask the standard journalist questions. Who: Are we assuming that we know who all the key players are? What: Are we assuming that we know the goals of the key players? When: Are we assuming that conditions have not changed since our last report or that they will not change in the foreseeable future? Where: Are we assuming that we know where the real action is going to be?
Why: Are we assuming that we understand the motives of the key players? How: Are we assuming that we know how they are going to do it?
Use of phrases such as “will always,” “will never,” or “would have to be” suggests that an idea is not being challenged. Perhaps it should be.
Use of phrases such as “based on” or “generally the case” suggests that a challengeable assumption is being made.
When the flow of assumptions starts to slow down, ask, “What else seems so obvious that one would not normally think about challenging it? If no one can identify more assumptions, then there is an assumption that they do not exist, which itself is an assumption subject to challenge.
* After identifying a full set of assumptions, go back and critically examine each assumption.
Ask:
Why am I confident that this assumption is correct?
In what circumstances might this assumption be untrue?
Could it have been true in the past but no longer be true today?
How much confidence do I have that this assumption is valid?
If it turns out to be invalid, how much impact would this have on the analysis?
* Place each assumption in one of three categories:
Basically solid.
Correct with some caveats.
Unsupported or questionable—the “key uncertainties.”
* Refine the list, deleting those that do not hold up to scrutiny and adding new assumptions that emerge from the discussion. Above all, emphasize those assumptions that would, if wrong, lead to changing the analytic conclusions.
* Consider whether key uncertainties should be converted into intelligence collection requirements or research topics.
When concluding the analysis, remember that the probability of your analytic conclusion being accurate cannot be greater than the weakest link in your chain of reasoning. Review your assumptions, review the quality of evidence and reliability of sources, and consider the overall difficulty and
complexity of the issue. Then make a rough estimate of the probability that your analytic conclusion will turn out to be wrong. Use this number to calculate the rough probability of your conclusion turning out to be accurate. For example, a three in four chance (75 percent) of being right equates to one in four chance (25 percent) of being wrong. This focus on how and why we might be wrong is needed to offset the natural human tendency toward reluctance to admit we might be wrong.
Figure 8.1 on the next page shows apparently flawed assumptions made in the Wen Ho Lee espionage case during the 1990s and what further investigation showed about these assumptions. A Key Assumptions Check could have identified weaknesses in the case against Lee much earlier.
Relationship to Other Techniques
The Key Assumptions Check is frequently paired with other techniques because assumptions play an important role in all structured analytic efforts, and it is important to get them right. For example, when an assumption is critical to an analysis, and questions remain about the validity of that assumption, it may be desirable to follow the Key Assumptions Check with a What If? Analysis.
Imagine a future (or a present), in which the assumption is wrong. What could have happened to make it wrong, how could that have happened, and what are the consequences?
There is a particularly noteworthy interaction between Key Assumptions Check and Analysis of Competing Hypotheses (ACH). Key assumptions need to be included as “evidence” in an ACH matrix to ensure that the matrix is an accurate reflection of the analyst’s thinking. And analysts frequently identify assumptions during the course of filling out an ACH matrix. This happens when an analyst assesses the consistency or inconsistency of an item of evidence with a hypothesis and concludes that this judgment is dependent upon something else—usually an assumption. Users of ACH should write down and keep track of the assumptions they make when evaluating evidence against the hypotheses.
Figure 8.1 Key Assumptions Check: The Case of Wen Ho Lee
Source: 2009 Pherson Associates, LLC.
Quadrant Crunching (chapter 5) and Simple Scenarios (chapter 6) both use assumptions and their opposites to generate multiple explanations or outcomes.
Origins of This Technique
Although assumptions have been a topic of analytic concern for a long time, the idea of a specific analytic technique to focus on assumptions originated in the CIA’s Sherman Kent School for
Intelligence Analysis. The discussion of Key Assumptions Check in this book is from Randy Pherson, Handbook of Analytic Tools and Techniques (Reston, Va.: Pherson Associates, LLC, 2008), and training materials from the Sherman Kent School.