2.1 Four Categories of Analytic Methods
2.2 Taxonomy of Structured Analytic Techniques
CHAPTER
2
Building a Taxonomy
A
taxonomy is a classification of all elements of the domain of information or knowledge.It defines a domain by identifying, naming, and categorizing all the various objects in this space. The objects are organized into related groups based on some factor common to each object in the group. This book presents a taxonomy that defines the domain of structured analysis, which is but one part of the larger domain of intelligence analysis methods in general.
Figure 2.0 shows four broad categories of analytic methods. The focus of this book, the structured analysis category, is broken down further into eight categories. This chapter describes the rationale for these four broad categories and identifies the eight categories of structured analysis.
The word taxonomy comes from the Greek taxis meaning arrangement, division, or order and nomos meaning law. Classic examples of a taxonomy are Carolus Linnaeus’s hierarchical classification of all living organisms by kingdom, phylum, class, order, family, genus, and species that is widely used in the biological sciences, and the periodic table of elements used by chemists. A library catalogue is also considered a taxonomy, as it starts with a list of related categories that are then progressively broken down into finer
categories.
Development of a taxonomy is an important step in organizing knowledge and furthering the development of any particular discipline. Rob Johnston developed a
taxonomy of variables that influence intelligence analysis but did not go into any depth on analytic techniques or methods. He noted that “a taxonomy differentiates domains by
specifying the scope of inquiry, codifying naming conventions, identifying areas of interest, helping to set research priorities, and often leading to new theories. Taxonomies are
signposts, indicating what is known and what has yet to be discovered.”1
Figure 2.0 Taxonomy Outline
Source: 2009 Pherson Associates, LLC.
Robert Clark has described a taxonomy of intelligence sources.2 He also categorizes some analytic techniques commonly used in intelligence analysis, but not to the extent of creating a
taxonomy. To the best of our knowledge, a taxonomy of analytic methods for intelligence analysis has not previously been developed, although taxonomies have been developed to classify research
methods used in forecasting,3 operations research,4 information systems,5 visualization tools,6 electronic commerce,7 knowledge elicitation,8 and cognitive task analysis.9
After examining taxonomies of methods used in other fields, we found that there is no single right way to organize a taxonomy—only different ways that are more or less useful in achieving a specified goal. In this case, our goal is to gain a better understanding of the domain of structured analytic
techniques, investigate how these techniques contribute to providing a better analytic product, and consider how they relate to the needs of analysts. The objective has been to identify various
techniques that are currently available, identify or develop additional potentially useful techniques, and help analysts compare and select the best technique for solving any specific analytic problem.
Standardization of terminology for structured analytic techniques will facilitate collaboration across agency boundaries during the use of these techniques.
2.1 FOUR CATEGORIES OF ANALYTIC METHODS
Intelligence analysts employ a wide range of methods to deal with an even wider range of subjects.
Although this book focuses on the developing field of structured analysis, it is necessary to identify some initial categorization of all the methods in order to see where structured analysis fits in. Many researchers write of only two general approaches to analysis: qualitative vs. quantitative, intuitive vs.
empirical, or intuitive vs. scientific. Others might grant that there are three—intuitive, structured, and scientific.
Whether intelligence analysis is, or should be, an art or science is one of the long-standing
debates in the literature on intelligence analysis. As we see it, intelligence analysis has aspects of both spheres. The range of activities that fall under the rubric of intelligence analysis spans the entire range of human cognitive abilities, and it is not possible to divide it into just two categories, art and science, or to say that it is only one or the other. The extent to which any part of intelligence analysis is either art or science is entirely dependent upon how one defines “art” and “science.”
“The first step of science is to know one thing from another. This knowledge consists in their specific distinctions;
but in order that it may be fixed and permanent, distinct names must be given to different things, and those names must be recorded and remembered.”
—Carolus Linnaeus, Systema Naturae (1738)
The taxonomy described here posits four functionally distinct methodological approaches to intelligence analysis. These approaches are distinguished by the nature of the analytic methods used, the type of quantification if any, the type of data that are available, and the type of training that is expected or required. Although each method is distinct, the borders between them can be blurry.
* Expert judgment: This is the traditional way most intelligence analysis has been done. When done well, expert judgment combines subject matter expertise with critical thinking. Evidentiary reasoning, historical method, case study method, and reasoning by analogy are included in the expert judgment category.10 The key characteristic that distinguishes expert judgment from structured analysis is that it is usually an individual effort in which the reasoning remains largely in the mind of the individual analyst until it is written down in a draft report. Training in this type of analysis is generally provided through postgraduate education, especially in the social sciences and liberal arts, and often along with some country or language expertise.
* Structured analysis: Each structured analytic technique involves a step-by-step process that externalizes the analyst’s thinking in a manner that makes it readily apparent to others, thereby
enabling it to be reviewed, discussed, and critiqued piece by piece, or step by step. For this reason, structured analysis often becomes a collaborative effort in which the transparency of the analytic process exposes participating analysts to divergent or conflicting perspectives. This type of analysis is believed to mitigate the adverse impact on analysis of known cognitive limitations and pitfalls.
Frequently used techniques include Structured Brainstorming, Scenarios, Indicators, Analysis of Competing Hypotheses, and Key Assumptions Check. Structured techniques can be used by analysts who have not been trained in statistics, advanced mathematics, or the hard sciences. For most
analysts, training in structured analytic techniques is obtained only within the Intelligence Community.
This situation is now changing, however; quite a few colleges and universities have initiated programs in intelligence or homeland security analysis, and some of the course curricula include structured techniques, such as Analysis of Competing Hypotheses.
* Quantitative methods using expert-generated data: Analysts often lack the empirical data needed to analyze an intelligence problem. In the absence of empirical data, many methods are designed to use quantitative data generated by expert opinion, especially subjective probability judgments. Special procedures are used to elicit these judgments. This category includes methods
such as Bayesian inference, dynamic modeling, and simulation. Training in the use of these methods is provided through graduate education in fields such as mathematics, information science, operations research, business, or the sciences.
* Quantitative methods using empirical data: Quantifiable empirical data are so different from expert-generated data that the methods and types of problems the data are used to analyze are also quite different. Econometric modeling is one common example of this method. Empirical data are collected by various types of sensors and are used, for example, in analysis of weapons systems.
Training is generally obtained through graduate education in statistics, economics, or the hard sciences.
No one of these methods is better or more effective than another. All are needed in various
circumstances to optimize the odds of finding the right answer. The use of multiple methods during the course of a single analytic project should be the norm, not the exception. For example, even a highly quantitative technical analysis may entail assumptions about motivation, intent, or capability that are best handled with expert judgment and/or structured analysis. One of the structured techniques for idea generation might be used to identify the variables to be included in a dynamic model that uses expert-generated data to quantify these variables.
Of these four methods, structured analysis is the new kid on the block, so to speak; so it is useful to consider how it relates to the other methods, especially how it relates to expert judgment. Expert judgment combines subject-matter expertise and critical thinking in an activity that takes place largely in an analyst’s head. Although the analyst may gain input from others, the analytic product is
frequently perceived as the product of a single analyst, and the analyst tends to feel “ownership” of his or her analytic product. The work of a single analyst is particularly susceptible to the wide range of cognitive pitfalls described in Psychology of Intelligence Analysis.11
Structured analysis follows a step-by-step process that can be used by an individual analyst, but it is done more commonly as a group process, as that is how the principal benefits are gained. As we discussed in the previous chapter, structured techniques guide the dialogue between analysts with common interests as they work step-by-step through an analytic problem. The critical point is that this approach exposes participants with various types and levels of expertise to alternative ideas,
evidence, or mental models early in the analytic process. It can help the experts avoid some of the common cognitive pitfalls. The structured group process that identifies and assesses alternative perspectives can also help to avoid “groupthink,” the most common problem of small-group processes.
When used by a group or a team, structured techniques can become a mechanism for information sharing and group learning that helps to compensate for gaps or weaknesses in subject-matter
expertise. This is especially useful for complex projects that require a synthesis of multiple types of expertise.
2.2 TAXONOMY OF STRUCTURED ANALYTIC TECHNIQUES
Structured techniques have been used by Intelligence Community methodology specialists and some analysts in selected specialties for many years, but the broad and general use of these techniques by the average analyst is a relatively new approach to intelligence analysis. The driving forces behind the development and use of these techniques are (1) an increased appreciation of cognitive limitations and pitfalls that make intelligence analysis so difficult, (2) prominent intelligence failures that have
prompted reexamination of how intelligence analysis is generated, (3) policy support and technical support for interagency collaboration from the Office of the Director of National Intelligence, and (4) a desire by policymakers who receive analysis that it be more transparent as to how the conclusions were reached.
Considering that the Intelligence Community started focusing on structured techniques in order to improve analysis, it is fitting to categorize these techniques by the various ways they can help achieve this goal. Structured analytic techniques can mitigate some of the human cognitive limitations, side-step some of the well-known analytic pitfalls, and explicitly confront the problems associated with unquestioned assumptions and mental models. They can ensure that assumptions, preconceptions, and mental models are not taken for granted but are explicitly examined and tested. They can support the decision-making process, and the use and documentation of these techniques can facilitate information sharing and collaboration.
A secondary goal when categorizing the structured techniques was to correlate categories with different types of common analytic tasks. This makes it possible to match specific techniques to individual analyst’s needs, as will be discussed in chapter 3. There are, however, some techniques that fit comfortably in several categories because they serve multiple analytic functions.
The eight categories of structured analytic techniques, which are listed below, are described in detail in chapters 4–11. The introduction to each chapter describes how that specific category of techniques helps to improve analysis.
Decomposition and Visualization (chapter 4) Assessment of Cause and Effect (chapter 8) Idea Generation (chapter 5) Challenge Analysis (chapter 9)
Scenarios and Indicators (chapter 6) Conflict Management (chapter 10) Hypothesis Generation and Testing (chapter 7) Decision Support (chapter 11)
1. Rob Johnston, Analytic Culture in the U.S. Intelligence Community (Washington, D.C.: CIA Center for the Study of Intelligence, 2005), 34.
2. Robert M. Clark, Intelligence Analysis: A Target-Centric Approach, 2nd ed. (Washington, D.C.: CQ Press, 2007), 84.
3. Forecasting Principles Web site, www.forecastingprinciples.com/files/pdf/methodsselectionchart.pdf.
4. Russell W. Frenske, “A Taxonomy for Operations Research,” Operations Research 19, no. 1 (January–February 1971).
5. Kai R. T. Larson, “A Taxonomy of Antecedents of Information Systems Success: Variable Analysis Studies,” Journal of Management Information Systems 20, no. 2 (Fall 2003).
6. Ralph Lengler and Martin J. Epler, “A Periodic Table of Visualization Methods,” undated, www.visual-literacy.org/periodic_table/periodic_table.html.
7. Roger Clarke, Appropriate Research Methods for Electronic Commerce (Canberra, Australia: Xanax Consultancy Pty Ltd., 2000), http://anu.edu.au/people/Roger.Clarke/EC/ResMeth.html.
8. Robert R. Hoffman, Nigel R. Shadbolt, A. Mike Burton, and Gary Klein, “Eliciting Knowledge from Experts,” Organizational Behavior and Human Decision Processes 62 (May 1995): 129–158.
9. Robert R. Hoffman and Laura G. Militello, Perspectives on Cognitive Task Analysis: Historical Origins and Modern Communities of Practice (Boca Raton, Fla.: CRC Press/Taylor and Francis, 2008); and Beth Crandall, Gary Klein, and Robert R.
Hoffman, Working Minds: A Practitioner’s Guide to Cognitive Task Analysis (Cambridge, Mass.: MIT Press, 2006).
10. Reasoning by analogy can also be a structured technique called Structured Analogies, as described in chapter 8.
11. Richards J. Heuer Jr., Psychology of Intelligence Analysis (Washington, D.C.: CIA Center for the Study of Intelligence, 1999), reprinted by Pherson Associates, LLC, 2007.