Abstract1
Intelligence analysis requires detecting and ex-ploiting patterns hidden in complex data. When the critical aspects of a data set can be effec-tively visually presented, displays become pow-erful tools by harnessing the pattern-recognition capabilities of human vision. To this end, shape, color, and interactive techniques are widely util-ized in intelligence displays. Unfortunately, the volume and complexity of intelligence data has outstripped our ability to visualize that data. Un-der the ARDA GI2Vis program we have over-come this limit with a broad new class of visu-alization techniques based on the innate human ability to perceive motion. These “kinetic” visu-alizations encode attributes of data using motion, and use motion to query for and highlight pat-terns in intelligence data. By conveying complex data directly to the pre-attentive perceptual sys-tem, these displays require less time and cogni-tive effort to detect patterns than purely static displays. In this work we describe kinetic dis-plays for network analysis, detection of patterns and correlations across multiple displays of geo-spatially registered event data, and visualization of structures in high-dimensional data sets such as imagery-derived intelligence. We also present the theoretical basis for this work, and briefly describe the results of published experiments demonstrating the utility of kinetic displays.
1. Introduction
Visualization techniques attempt to reduce the cognitive effort required to understand and extract patterns from intelligence data by utilizing the strengths of the human
1
This study was supported and monitored by the Advanced Research and Development Activity (ARDA) and the Na-tional Geospatial-Intelligence Agency (NGA) under Contract Number NMA401-02-C-0019.
perceptual system. Human vision can extract meaning (structure, trends, outliers) from massive volumes of data, and this works well up to the point where data com-plexity outstrips our abilities to present data in a usable fashion to the perceptual system. Current techniques for visualizing information from massive, complex (high-dimensional, linked) data are hampered by visual com-plexity and fragmentation.
As part of the ARDA Geospatial Intelligence Informa-tion VisualizaInforma-tion (GI2Vis) Program (Phase II) we have developed a new class of visualizations that enable ana-lysts to readily perceive visual patterns in complex and high-dimensional data sets that pose serious challenges to other visualization techniques. These new visualizations leverage a human perceptual capability that has not been exploited in previous visualization methodologies. The human visual system automatically attributes object-ness to elements that share a common motion, even if they are dissimilar, widely separated in the field of view, or sur-rounded by clutter or camouflage. We call this phenome-non motion-induced perceptual grouping.
We have developed a family of Kinetic Visualizations
based on various ways of employing motion to stimulate perceptual grouping. Unlike animations, which produce motion as a side effect of displaying sequential instanta-neous snapshots of time-varying data, kinetic visualiza-tions use motion to encode and present complex and high dimensional time-invariant data. These allow analysts to achieve greater insight into the data being presented. Ki-netic visualization techniques can also be effectively used to visualize time-varying data.
Kinetic Visualizations: A New Class of Tools for Intelligence Analysis
Robert J. Bobrow, Aaron Helsinger
BBN Technologies
10 Moulton Street
Cambridge, MA 02138
{rusty, ahelsing} @bbn.com
Keywords: Visualization, Fusion
Multivariate data representations are often visually impenetrable because they are so complex, and be-cause the various graphical codes may perceptually interfere with each other (Healey 1999). Multiple views, on the other hand, allow the user to extract in-teresting aspects of the data for inspection, but suffer greatly from visual fragmentation. (Bartram and Ware 2002)
One of the key advantages of kinetic visualizations is that, by making strong use of human pre-attentive visual processing, they speed up the detection of relevant struc-tures in the visualization, and make it possible for the analyst to see patterns immediately that might otherwise be imperceptible. The goal of our research is to develop kinetic visualization techniques that will enable analysts rapidly to perceive and understand patterns embedded in massive amounts of data, and to convey their findings effectively to policy makers. During Phase II of the GI2Vis program, we have explored the use of kinetic visualizations for intelligence-analysis displays, and found them useful in many areas. We present a brief de-scription of several of these applications here. It is im-portant to note, however, that a full appreciation of these visualizations requires seeing live demonstrations – it is impossible to capture the essence of kinetic visualiza-tions (motion) in a static image on paper.
Our research has been focused on three particularly promising types of kinetic visualization techniques: di-rect periodic motion to highlight related display ele-ments; motion brushing to select data elements in one view and highlight them in multiple other views; and
moxel (motion element) encoding to represent high di-mensional data. The advantages of these kinetic-visualization techniques include perceptual grouping: the ability to reveal global patterns of relationships among data elements that are otherwise obscured; high dimensional structure visibility: the ability to reveal structure in high dimensional data sets; and contextual viewing: the ability to display diverse data sets in context and in relation to each other.
Our work builds on an increasing focus over the last twenty years in new ways of enhancing the utility of plays. Researchers have explored the use of stereo dis-plays, blinking or animated components, novel iconogra-phy, and other techniques. Our work leverages this previ-ous applied research, but also builds on basic research in human vision, particularly motion perception. Our kinetic visualizations rely on the advances in computer graphics driven by the gaming industry, and show how motion can address some of the challenges of data analysis.
In the remainder of this paper, we highlight three par-ticularly powerful kinetic visualizations, and relate our work to future needs. In our first example, kinetic visu-alizations of networks use direct periodic motion group-ing to facilitate network analysis. We then use motion brushing to show multiple views of a single data set, us-ing motion to correlate the displays. Finally, we encode multiple attributes of data in the ways individual display elements move, allowing rich displays of imagery-based intelligence data.
2. Displays for Network Analysis
Many problem domains can be mapped into a graph and visually displayed in the form of a node-link diagram. Typically, nodes represent entities while the links repre-sent relationships between the nodes. Many variants on
node-link diagrams allow for various types of nodes to be represented by shapes or colors, and various relationships to be represented by varied graphical styles for the links. Such diagrams are used for computer and communica-tions network analysis and management, social network analysis (including terrorist networks) and many other applications.
Node-link diagrams allow for the rapid comprehension of the inter-relationships between entities, and for certain patterns of links to be immediately evident. Unfortu-nately, a major drawback is that they do not scale well to larger applications. A graph with more than twenty or thirty nodes and at least twice as many links between them becomes visually incomprehensible, although many applications require the viewing of much larger graphs.
Various attempts have been made to increase the size of the graph that can be readily comprehended. One pro-posed solution has been to represent the information structure in 3D (Fairchild et al. 1988; Parker et al. 1998; Chen et al. 2001). Unfortunately, research suggests that merely showing a static-perspective picture of a graph laid out in 3D is no more comprehensible than a 2D lay-out (Ware and Franck 1996). It is important to include non-pictorial depth cues to actually perceive more infor-mation. Accordingly, our research to date has focused on adding motion to 2D network displays, and our future work will attempt to combine this with stereo depth. When both stereo depth and kinetic depth cues are pro-vided it may be possible to see a structure approximately three times as large (Ware and Franck 1996). Adding other kinetic cues makes it possible to answer questions about networks with thousands of nodes (Ware and Bo-brow 2005).
Our approach to network visualization is to build a sys-tem in which (one or more) subsets of a large network diagram are highlighted by being set into oscillatory mo-tion. The visual effect of this motion is quite striking – user-selected subsets stand out clearly from the back-ground of the entire graph, and it becomes almost effort-less to visually determine properties of these subsets, despite the potential clutter of hundreds or thousands of other non-moving “background” nodes and links. At the same time, both the structure of the background and the relation of the background to the highlighted sets are clearly visible. In the example displays, the set of high-lighted nodes and links is chosen by topological close-ness (link distance) to a point selected with the mouse (e.g., the direct and indirect contacts of an individual) – but in general the highlighted nodes and links can be chosen in response to a question (e.g. the individuals known to have attended a meeting in June 2001, and all individuals they are known to have contacted in the two months thereafter). This allows, for example, an analyst to readily see the associates of an individual of interest, or deduce the key players in an inter-related network of individuals.
We have performed experimental analysis of the benefit of motion in these displays. In one set of published
ex-periments we demonstrated a noticeable benefit (quicker response and reduced error rate) when using motion to highlight a node and its links. Later experiments have shown that motion can be combined effectively with other highlighting techniques to permit rapid and accu-rate answers to questions involving relations among nodes and links in graphs with up to 3200 nodes (on a high resolution display). (Ware and Bobrow 2004, 2005)
We have demonstrated these results in randomly gener-ated graphs of several thousand nodes, as well as in natu-rally occurring graphs such as email traffic within a mid-sized organization.
In a particularly enlightening display (Figure 1), we analyzed email traffic at BBN Technologies. We col-lected data about email traffic among employees of BBN for a particular date range. In this dataset, nodes repre-sent (anonymous) people, and links reprerepre-sent email rela-tionships. We color-coded people by department within BBN, and displayed links between nodes that exchanged more than a (user settable) number of messages.
With this dataset, the questions are of the type, “Who talks to whom?” (Who are the big email-ers? Do people talk mostly within their own department or across de-partments? Do departments tend to have a few points of contact with the rest of the company?)
BBN has over 600 employees. Needless to say, that translates into a lot of email, and many email relation-ships. Trying to reasonably represent a 600-node network diagram, and make sense of it, is nearly impossible. In particular, this dataset is highly inter-related; standard network displays, for example, will not capture the data. In this context, we introduced motion.
The primary use for motion in this kind of display is to show what has been selected. In our implementation, the user selects a node by clicking on it. As a result that node, and the links from that node, oscillate in a small circle (keeping the links intact). This allows the user to immediately see the connections to a given node. In our
example, that means that one can see at a glance what departments this person is in email contact with, the por-tion of email traffic due to this person, etc.
Our experience shows that using motion to highlight se-lected nodes and links in a network display allows ana-lysts to examine much larger networks and more quickly see, with a low error rate, patterns in the data. This al-lows quicker analysis and deeper analysis, and alal-lows analysts to make more effective use of their skills.
3. Cross-Display Correlation
Multiple displays are often used to examine various as-pects of complex objects. For example, consider a data set of events. Each data point (object) has many attrib-utes, including location and timing, participants (with multiple properties), outcomes, etc. In order to make it easier to see each of the various attributes of an event, one might show each event in multiple displays, such as a map, a temporal histogram and several scatter plots. For a typical example, see our display analyzing security events in Afghanistan (Figure 2). While this is a powerful technique, it creates a fragmentation problem, because information about the same event is scattered across mul-tiple visualizations, and it can take considerable cogni-tive effort for the analyst to keep track of multiple repre-sentations of the same data object, much less to keep track of inter-related data across multiple views.
The problem with multiple displays involves how to minimize the cognitive effort needed to maintain a
corre-Multiple views, on the other hand, allow the user to ex-tract interesting aspects of the data for inspection, but suffer greatly from visual fragmentation. Extra visual elements must be introduced into the display to link the spatially disparate points to avoid a cognitively effort-ful visual search. These in turn introduce more visual complexity. (Bartram and Ware 2002)
Figure 1. Kinematic Network Display
spondence between items in different visualizations. This is commonly addressed with the use of coordinated brushing. Brushing is a mechanism found in many data visualization systems for interactively selecting subsets of objects in a visualization so that they can be high-lighted, deleted, or masked as a group. In coordinated brushing, objects in one display (window) are brushed and highlighted, and the same data objects are similarly highlighted in the other visualizations in which they ap-pear.
Some displays use color or thicker borders for marking a brushed selection in multiple displays. But if the dis-plays use color to encode attributes of the data objects, then color highlighting hides information about those objects. We have found that coordinated motion triggered by user action – that is, motion brushing – can be used to link data distributed across multiple displays, allowing the user to drill down from one display to another associ-ated display while maintaining context.
We use motion to allow analysts to discover patterns in the data that are only visible by looking at multiple dis-plays simultaneously. By using motion to group data points in one display and highlight those points in one or more other displays, the analyst can effectively process a much larger quantity of data while preserving essential context. This allows analysts to discover patterns in the data that are visible only when looking through multiple dimensions of the data or multiple views of the same data. This has been noticeably effective in our multiple-window security event analysis display (Figure 2). Using motion to group items across displays also allows the user to switch to a new display while preserving the mo-tion highlight of a selected dataset, without losing con-text or focus.
Another obvious use for motion-based brushing is to coordinate multiple displays when viewing a complex multi-dimensional dataset from multiple perspectives.
This includes hyper-spectral data from aircraft and satel-lites, as in the data from NASA’s Aviris imager. To dem-onstrate the power of motion for cross-display correlation in hyper-spectral data, we visualize the Moffett Field image
(http://aviris.jpl.nasa.gov/html/aviris.freedata.html). In our display (Figure 3) we show four perspectives on this dataset. One display shows an overhead image in the vis-ual spectrum, while others show scatter and histogram plots based on infrared reflectivity. In our live demon-stration we show how to use motion brushing to make a selection in one display and see the same data highlighted (in context) in each of the other displays. This allows the user to correlate what they are seeing in each display without significant cognitive effort. In particular, in some displays the selected data may not form a contiguous re-gion; and yet the motion of the data points provides per-ceptual grouping, as effective as if they were enclosed in a separate region. The moving icons are immediately recognizable as those selected or seen in a previous dis-play, allowing the analyst to focus instead on the new context for the data points, or new shape of the region in this view.
Kinetic displays that use motion to represent a selection across multiple displays can readily support multiple se-lections, each of which might oscillate in a distinctive direction or with a distinctive frequency, across all of the displays. In this case the user sees both the relation of multiple selection sets, and the overall context.
Our work has shown that motion provides an effective means for correlating selections and views across multi-ple displays with low cognitive effort. This allows ana-lysts to work with more complex datasets, including data that require multiple perspectives.
4. Direct Display of High-Dimensional
Data
So far we have discussed the use of kinetic visualization techniques to make it easy to see and understand (already detected) structures in complex contexts. While these results are striking and useful, they only touch the sur-face of what can be done with motion. In particular, ki-netic visualizations can be used to allow analysts to find new patterns in high dimensional data sets. Adding mo-tion to existing technologies for displaying high-dimensional data is comparable to going from gray-scale images to color images – but the shift from gray-scale to color merely involves going from 1-dimensional data values to 3-dimensional values, while adding kinetic cod-ing allows us to go from 3-dimensional values to 10 or more dimensions in a single image.
Datasets with more than three data dimensions (representable by color) are difficult to visualize all at once. For example, within the imagery domain one can think of polarimetric and thermal images and combinations of Synthetic Aperture Radar (SAR) and optical imagery. It is also possible to display
high-IR ScatterPlot Bands 100,150 IR Point Stacked Histogram False color image IR ScatterPlot Bands 50, 80
dimensional non-image data such as the properties of message traffic between computers under different conditions such as normal load, port scanning, and attack. It is possible to automate the search for known classes of patterns in such high-dimensional data, but no automated technique is the equal of a trained analyst in discovering unexpected patterns. When looking at standard images, analysts are able to pick out patterns and anomalies without having to know exactly what it is that they are looking for – trained human visual perception lets them “know it when they see it.” To exploit human visual pat-tern recognition with high-dimensional data (for exam-ple, hyper-spectral data), the problem is to be able to present images of all the data to the analyst directly.
The key to solving this problem is to use patterns of motion to encode multiple data dimensions, just as we use red, green and blue (or hue, saturation and bright-ness) to encode three data dimensions as color. The sim-plest and most controllable way of generating motion to represent some of the attributes of n-dimensional data items is to encode data attributes as motions of icons or glyphs whose color, shape, or other visual properties rep-resent the other displayed dimensions. The motions can be thought of as two-dimensional motions in display space for simple planar icons, or three-dimensional (3D) motions (if we represent non-motion encoded dimensions by visual/physical properties of 3D objects).
In our displays we represent each data point in the im-age as a small 3D element for which dimensions of the data determine shape, color, and location, but which also
moves in ways that are data driven. Such motion elements (or moxels) can move in three dimensions (vertically, forward/back and left/right) as well as rotationally (pitch, roll and yaw). Even if we limit ourselves to simple sinu-soidal motion in each of these six dimensions, the palette of possible motions that can be readily computed and displayed is large, and the key issue is to choose motions that provide strong grouping cues. Distinct data dimen-sions control various aspects of these motions, including frequency, amplitude, and phase. The result is that we have eighteen parameters that can be controlled by data dimensions.
Choosing the most visually effective mapping from data dimensions to visual parameters is not yet a science. Our starting point for exploring this is the research by Bartram and Ware (2002), and Bartram, Ware, and Cal-vert (2001). We have demonstrated that depending on the distribution of data-dimension values, we can effectively encode between seven and ten dimensions as motion. When added to the three dimensions provided by color, we have shown that we can provide visual renditions of 10-dimensional data in which shapes and gradients de-fined by any one (or several) of the dimensions can be readily seen.
The resulting displays (see Figure 4) look something like a field of tall grass blowing in the wind; each blade of grass is recognizable in itself, but is also moving with the wind. You can see the larger trends due to the wind,
but if a cat is moving through the field, you can at the same time infer its location due to the separate motion of the grass as it passes. It is exactly this richness, the
mul-tiple levels of information that are instantly available to the user of these displays, that we are seeking to lever-age. In the example display shown here, we encode mul-tiple segments of spectral data in a single image. As a result, features in the landscape may become visible (or more distinct) than in a standard display or multiple dis-plays. For example, there are apparent paths through the green sections that become more apparent under motion. A trained analyst may be able to discern shapes and pat-terns not visible in any single display with a single glance. This compares to the more tedious approach of looking at each of multiple views independently, and relying on the analysts’ ability to mentally correlate these large datasets.
By using motion to encode dimensions of the data, we provide for a much richer display that makes optimal use of the human pre-attentive visual system, freeing the con-scious mind to perform more complex cognitive analyses.
5. Future Work and Conclusions
During the past phases of the ARDA GI2Vis program we developed Kinetic Visualizations. This new class of visualization techniques facilitates analysis of much more complex and high-dimensional data. We have also pro-duced and published experimental scientific evidence of the advantages of certain types of kinetic visualizations (particularly network visualizations).
We are in the early stages of developing the science and technology of kinetic visualization. In the current phase of the GI2Vis program we expect to do more basic re-search on how most effectively to use motion to encode not only data dimensions, but also meta-data (e.g., provenance and reliability estimates), and to do so in a way that makes the data most useful in the context of the
broader analytic process. Our plan covers both basic re-search on perceptual aspects of motion perception, and applied research on how best to incorporate kinetic visu-alizations in larger analytic tool suites.
We expect to prototype a number of new kinetic encod-ings, perform initial evaluations of their utility in helping analysts detect patterns, and then execute more thorough experiments on the most promising encodings. We will integrate the kinetic visualization components into a visualization toolkit. We will use this toolkit to explore the most effective ways of interacting with kinetically encoded data. We also plan to prototype a large network-exploration facility on very-high-resolution displays, ex-plore techniques for interacting with such visualizations, and carry out experiments to study the visual queries per-formed by users in the process of exploring patterns in large networks.
There is considerable basic and applied research needed to realize fully the potential of this new visualization paradigm. We have investigated only a small number of the kinetic-visualization mechanisms and techniques that we have identified. We have explored only a small por-tion of the very large space of promising kinetic visuali-zation mechanisms and techniques.
We strongly believe, however, that even basic research is carried out best in the context of real analytic prob-lems. To this end, we have already begun to apply our techniques to a small number of problems in the intelli-gence community. We are very much interested in learn-ing about new classes of analytic problems, and it is our hope that members of the intelligence community will be inspired by this work to suggest possible applications that we have not yet considered, leading to potential collabo-ration with groups looking for improved visualization techniques. Because the effective application of kinetic visualization depends strongly on the work environment and goals; good examples of real world, tough analysis problems are essential.
As we have noted, intelligence analysis requires the ex-ploration, analysis, and presentation of complex and of-ten high-dimensional data. While intelligent software can pre-filter some datasets, nothing replaces the value of a well-trained analyst studying the data. We have built a new class of visualizations using motion that allows an analyst to understand much more data, in context, with much less cognitive effort. By leveraging the human abil-ity to immediately perceive and understand patterns of motion, we have demonstrated several new kinetic visu-alizations. Motion effectively highlights points of interest in context, as in our network displays. Motion can also be used to correlate and group data in context across
multi-ple displays, as in our use of motion brushing. Addition-ally, moving icons, or moxels, allow us to encode many more dimensions of data in a single image. Our early demonstrations of these techniques hint at the promise that kinetic visualizations hold for multiple intelligence disciplines.
Acknowledgments
This work represents the efforts of many colleagues at BBN Technologies, U-Mass Lowell and UNH. This study was supported and monitored by the Advanced Research and Development Activity (ARDA) and the National Geospatial-Intelligence Agency (NGA) under Contract Number NMA401-02-C-0019. The views, opinions, and findings contained in this report are those of the author(s) and should not be construed as an official Department of Defense position, policy, or decision, unless so desig-nated by other official documentation.
References
Bartram, L. and Ware, C. 2002. Filtering and brushing with motion. Information Visualization. 1(1): 66-79. Bartram, L.; Ware, C. and Calvert, T. 2001. Moving icons: Detection and distraction. Proceedings of INTERACT 2001.
Chen, C. and Paul, R.J. 2001. Visualization of a knowl-edge domain’s intellectual structure. IEEE Computer. 34: 65-71.
Fairchild, K.M.; Poltrock, S.E. and Furnas, G.W. 1988. SemNet: three-dimensional graphical representations of large knowledge bases. In Cognitive Science and Its ap-plications for Human-Computer Interaction. (R. Guin-don. Ed) Lawrence Erlbaum, London, pp. 201-233. Parker, G.; Franck, G. and Ware, C. 1998. Visualization of large nested graphs in 3D. Journal of Visual Lan-guages and Computing. 9: 299-317.
Ware, C. and Bobrow, R. 2004. Motion to support rapid interactive queries on node-link diagrams. ACM Trans. Appl. Percep. 1: 1-15.
Ware, C. and Bobrow, R. 2005. Supporting Visual Que-ries on Medium Sized Node-Link Diagrams. Information Visualization. 4(3). (In press).
Ware, C. and Franck, G. 1996 Evaluating Stereo and Mo-tion Cues for Visualizing InformaMo-tion Nets in Three Di-mensions. ACM Transactions on Graphics. 15(2): 121-139.