How To Visualize And Analyze Sonar Data

4D environmental data visualization and analysis. What we have learned from the unique challenges in working with complex temporal and spatial (4-dimensional) data derived from modern fisheries sonar systems.

Tim Pauly, Greg Lee, John Corbett and Matthew Wilson SonarData Pty Ltd, 110 Murray Street, Hobart, Tasmania 7001

Abstract

Modern fisheries sonar systems present unique challenges in data complexity and data quantity. Visualization is essential to all stages of working with fisheries sonar data:

quality assurance; filtering; analysis and hypothesis forming; and presentation of results.

SonarData’s software products meet these challenges through a commitment to

continuous development supporting the requirements of leading researchers around the world.

SonarData’s Echoview software has been extended to provide a unique 4-dimensional visualization and analysis environment to meet the demands of new-generation fisheries multibeam and scanning sonar systems. This has guided developers and users in further understanding the requirements for visualization and analysis of fisheries acoustics data and their fusion with other environmental data. Through data fusion and the advancing capability of computing technologies, we present these requirements with a view to achieving outcomes for better understanding environmental dynamics and change detection, and ecological systems.

The ability to fuse data from multiple sources in a versatile 4-dimensional visualization environment provides new opportunities for observation and interaction, and insights into marine ecosystems. It will ultimately yield the analysis methodologies of the future.

Keywords: fusion, visualization, fisheries, acoustic, software, scanning sonar, multibeam sonar

Contact Author: Tim Pauly

Phone +61 362315588 fax +61 362341822 Email: [email protected]

Part 1: Background

The nature of sonar data

Active fisheries sonar provides backscatter data which is a convolution of pulses of sound through space and time. The convolution typically includes transmission properties (e.g. beam pattern through the water column and vessel motion) and the in- situ scattering properties of particles and objects in the water column, along with the structure and composition of the seabed. The echotrace or echogram is a data visualization of this backscatter for narrow beam echo sounders. It presents a representation of the world beneath a vessel traversing the ocean surface. The echogram is so intuitive a representation that it can obscure complexities and

assumptions which might not be understood or checked. So the echogram provides us

with a duality. First; a powerful and intuitive data representation for a complex set of

processes, and second; processes if scrutinized – that can be highly challenging to deconvolve.

The sustained use of the echogram as a data representation is testament to the fact that it provides high bandwidth contextual information. The user gets an immediate picture of the water column, its contents and the seabed. It is often only when we are faced with artifacts within an echogram that are difficult to explain, that we are challenged to interpret the complex processes represented within the convolution.

Visualize the current state of the data

Foote et al. 1991 described a system combining echogram visualization, data scrutiny and quantitative echo integration analysis. Echoview’s development has followed a design objective to visualize the current state of the data within each step of an analysis.

This necessitates the visualization of all variables of interest, that is; channels, features and attributes. For example, Figures 1 and 2 shows the novel visualization of attributes such as phase and angle data, along with logical data types for filtering or partioning regions and individual samples within echograms. Integrated as a single application this provides a functionally rich palette for users.

High level support for visualization throughout a process provides feedback

opportunities, not only for users but also developers. By representing a process as a

a) GPS track plot;

b) raw angle data;

c) Sv raw pings;

d) TS raw pings

a

b

c

d

data flow diagram, the results of each step can be quickly visualized and checked. This contributes to efficient and robust working methods for users and code developers. The outcome is processed echogram data providing, for example, tracked targets that can be visualized (Figure 2). The next logical step is to place results back into a representation of space and time (a 4D scene) providing the user with a wider environmental context to their data (Figure 2 and 3).

Figure 2:

Visualization for each step of an analysis – primary to transformed data:

a) primary TS raw pings;

b) logical bit mask, grey = true;

c) bit mask applied to TS;

d) single target detection applied to masked angle and TS data;

e) 4D scene with echogram curtain and 3D tracked target from single target detections.

a

b

c

d

e

Support for complex instruments

4D scenes enable representation of results of echogram analysis in context with other environmental data, but they also provide a powerful intuitive representation for the complex data from new scanning sonar and multibeam systems (Figure 3).

Summary: Significant outcomes of this journey

Intuitive representations of complex data through powerful visualization enhances information bandwidth to the brain. If coupled with a supporting environment of tools for investigating subtle and multi-faceted aspects of the data significant value is added.

Visualization of intermediate results at each step in an analysis provides efficient feedback when using and developing processes. The requirement for some generality to achieve this also provides greater opportunity for deeper insight and development of creative methods.

Developing immersive “real world” visualizations provides enhanced opportunities for insight into data. Visualizations that can provide rapid contextual information adding value to fused data.

Future developments need to provide a generalized, responsive and immersive visualization and analysis environment, which addresses the following: efficient and elegant data access; visualization tools that support large complex data sets;

represent real world phenomenon as objects; who’s attributes can be queried and relationships explored.

Part 2: The Future

This section presents selected aspects of a data fusion project underway at SonarData.

Our aim is to provide a suite of innovative tools which significantly enhance the ease with which scientists manipulate complex marine spatio-temporal environmental data. In particular, we seek to facilitate the fusion of different data types, so that the analyst can

a

b

Figure 3: 4D scenes - a) scanning sonar data with horizontal and vertical modes, and 3D school detection;

b) multibeam data with sector plot curtain, soundings classification { school, noise & seabed} indicated by color, and combined with 3D schools detection.

pose novel questions regarding feature relationships in an integrated context. For example, such tools might target: dynamics - the study of change over short time

intervals; change detection - the study of change over long time intervals; the bridging of specialist domains such as physical oceanography and fisheries research by enabling coincident visualization and analysis of trans-disciplinary results; fusion of coastal habitat data with a variety of fisheries data. Contributing to future work on these types of

problems requires assembling a wide variety of knowledge and skills. Here we will briefly discuss how the use of data flows, visualization, and the use of topology might contribute to such work.

Data flows

Data flow diagrams provide a pivotal tool for developing, recording, visualizing and communicating the elements of analysis methods. Enhanced as an object model - maintaining relationships between objects - data flows can provide an efficient and powerful research tool. Nested data flows provide a hierarchical assembly enabling consideration of a process at differing scales, both horizontally (traversing a process) and vertically (examining the detail of a nested method). A data flow also provides metadata for an analysis process – it is a visual summary of the components that comprise a method.

At the most abstract level, data flow objects are categorized as data sources, operators or visualizers. A data source provides a logical handle to render a heterogeneous data storage mechanism transparent to the user, whether these be local files, distributed databases or web services. Operators provide a user-extensible set of functions that transform the data passing through them. These may range from simple logical and arithmetic functions through to sophisticated filters and complex transformations.

Visualizers provide the mechanism by which the end product of a data flow is observed, from simple graphs through to complex 4D objects embedded in a data scene.

Visualization

Human beings have highly evolved visual systems. Vision and cognition are entwined.

These are the reasons that a picture is worth a thousand words. For humans, to ‘see’ is to understand.

Visualization is critical to data analysis. Usually, visualization provides the first contact with data, revealing structure that cannot otherwise be readily absorbed. Since John Tukey’s seminal 1977 work “Exploratory Data Analysis”, visualization has also been an integral part of how we interrogate data. After Tukey, changes in computer systems modified visualization methods, but not the goals: through visualization we familiarize ourselves with data, form new hypotheses and explode established ones.

There are two components to visualizing data: graphical display and model fitting. These are often applied alternately in an interactive fashion. We need graphics, of course, because visualization implies a process in which information appears in a visual display.

Fitting models to data is equally important. Fitting a model and visualizing deviations between the model and the data can reveal important structural features that would otherwise remain obscure. Consider plotting simple graphs of raw data: model fitting is applied by imagining what the data would look like under certain assumptions, and checking that the data display is conformant. More sophisticated models require additional methods, but the process remains the same.

Tools matter. To visualize data effectively we must choose the right equipment. It is easy to be dazzled by a visually appealing data display. We have a tendency to be misled into thinking we are absorbing relevant information when we see a lot of detail. But the success of a visualization tool should be based on the extent to which it enhances understanding. When data are prepared for visualization, the data are encoded in a display method. When viewed by the analyst, information is visually decoded from the display. Human visual perception is a vital link in the process. No matter how clever the choice of data and no matter how technologically impressive the encoding, visualization tools fail if the decoding fails. The art is in achieving an optimal match between the data, the nature of the enquiry, and visualization method.

Understanding visual perception allows informed choices between tools. Some display methods lead to efficient and accurate decoding, others don’t. Bill Cleveland’s Trellis Graphics system provides a notable example of perception-informed methods. His book

“Visualizing Data” is organized around applications of the visualization tools that became Trellis Graphics, showing the role of each tool in “Exploratory Data Analysis” and the class of problems that it helps to solve. It demonstrates the power of having the correct tool for the job: several visualizations reveal obvious effects that were missed in the original analyses.

Topology and data fusion

SonarData’s data fusion project is working on the utilization of topology to provide advanced functionality in true data fusion problem domains.

Spatial data can generally be described by a combination of geometric and topological properties. The geometric description includes the set of spatial locations of objects within the data set. For example, the set of spatial coordinates that are used to

approximate the position of a continuous line through space would be considered part of the geometrical description of a data set. On the other hand, the topological description is concerned with the spatial relationships between objects in a data set. Such a description may include assertions such as “point A lies within polygon B”, or “line C forms part of the boundary of polygon D”, or “polygon E partially overlaps polygon F”.

In general, topological properties can be inferred from the geometric description of the data if the geometric description is complete. In other words, we can discover the set of topological relationships between objects in a data set by analyzing the spatial locations of all objects. For this reason, storage of topological information has typically been viewed as unnecessary, particularly in GIS applications where data storage space has been a premium consideration. Instead, topological relationships have been inferred dynamically as required, and discarded.

In recent years there has been a shift in the GIS field towards utilizing topological

relationships for more advanced spatial data analysis and the persistence of topological

information. Applications of this approach include the enforcement of quality control

restrictions across multiple data sets. For example, topological rulesets describing the

allowed relationships between multiple spatial data themes are commonly used to

maintain data quality in critical spatial repositories. These topological rules act as filters

on modifications to the data, allowing modifications to be made only if all the topological

constraints are met. For example, consider an aquaculture license management

scheme in which licenses are granted to allow farming within specific regions.

Topological constraints such as “no fish farm license polygon shall overlap with a marine reserve polygon” ensure the consistency of the license allocations.

In the realm of data fusion, topology offers a powerful mechanism for discovering and analyzing spatial relationships between disparate data sources. With the advent of the data fusion project, we are implementing a highly generalized way of storing both geometric and topological data, along with a set of generalized analysis operations to take advantage of the fusion of varying data types.

In particular, we envisage supporting the fusion of multiple data sets while maintaining topological relationships between all original spatial objects and therefore available to subsequent analysis operations..

Our definition of data structures for storing fused data are general enough to handle objects of arbitrary dimensionality, whether these be regular (gridded) or irregular

(vector) data types. So, for example, a set of individual fish tracks could be fused over a bathymetric surface, overlayed with a gridded sea surface temperature map, and data from a 3-dimensional current model. Attributes from each of these objects, and the spatial relationships between them, would all be available for analysis. We believe that this structure offers great potential for enabling novel hypothesis formation and testing.

A simple example of such an analysis may be the rapid identification of all time intervals in which a target fish species was found within a particular water mass. Such a question may require the integration of fishery trawl data with ocean model and observation data, in order to verify, classify and track a water mass through time.

The generalization of data structures beyond the traditional 2 and 3-dimensional structures most often used in the GIS domain will allow for more powerful analysis in high dimensional feature spaces. Strong support for the 4

(temporal) dimension will provide a sound basis for work in change detection and dynamics.

It will also be possible to utilize higher dimensional data structures and operations to allow analysis in rich feature spaces. For example, we may investigate relationships between species by examining the structure of clusters of several species observations.

The structure of these clusters could be examined in the abstract space defined by water temperature, salinity and depth, season, and productivity. By creating structures in this high dimensional space and utilizing generalized analysis functions, queries become very powerful yet remain straightforward.

A strong emphasis on the topological description of spatial data also opens up the possibility of dealing with poorly specified geometric information (or observational uncertainty) using constructs such as fuzzy boundaries. By allowing specification of spatial locations with confidence measures constrained by a well defined topology, we can retain analytical power and help to quantify spatial error. Classification work may greatly benefit from underlying support for such fuzzy data representations.

Summary

SonarData’s data fusion project aims to deliver a powerful combination of data access, fusion, analysis and visualization. It targets diverse, feature-rich spatio-temporal

environmental data sets and seeks to provide new opportunities for analysis by offering

innovative tools in a novel environment. These aims will be achieved by core

technologies that provide highly responsive and immersive visualization, and highly generalized object and topology models.

We aim to enhance the excitement of discovery that visualization brings to data analysis, to provide a suite of tools that makes the interesting absolutely fascinating and the mundane easy, and ultimately to offer ‘inter-ocular traumatic impact’ - understanding that hits you between the eyes.

References

K. G. Foote, H. P. Knudsen, R. J. Korneliussen, P. E. Nordbo, and K Roang, Postprocessing system for echo sounder data, J. Aoust. Soc. 90 (1), 37-47, 1991.

W. S. Cleveland, Visualizing Data

Hobart Press, 1982.

J. Tukey, Exploratory Data Analysis, Addison-Wesley, 1977.