Virtual Environments in Scientific Visualization STEVE BRYSON
Computer Sciences Corporation
Numerical Aerodynamic Simulation Systems Division NASA Ames Research Center
Moffett Field, Ca.
ABSTRACT
The use of virtual environment (virtual reality) techniques for scientific visualization are surveyed. Les-sons learned from various existing applications are stated, with regard to implementation areas such as com-putation, graphics, data management, and user interface. The fruitful interplay of virtual reality and scientific visualization is stressed.
1. Why Scientific Visualization is a Good Application of Virtual Reality
Scientific visualization [McCormick et. al. 1987] is the use of computer graphics to create visual images which aid in the understanding of complex, often massive numerical representations of scientific concepts or results. Such numerical representations, or data sets, may be the output of numerical simulations as in computational fluid dynamics (CFD) or molecular modelling, recorded data as in geological or astronomical applications, or constructed shapes as in using computers to visualize topological arguments. These simulations are often contain high-dimensional data in a three-dimensional volume. The display of phenomena associated with this data may involve complex three-dimensional structures. Virtual reality displays aid in the unambiguous display of these structures by providing a rich set of spatial and depth cues. Virtual reality interface concepts allow the rapid and intuitive exploration of the volume containing the data, enabling the various phenomena at various places in that volume to be explored, as well as providing simple control of the visualization environment through interfaces integrated into the environment.
Evidence for the naturalness of the application of virtual reality to scientific visualization is the relatively rich set of applications which have appeared in spite of the immature state of virtual reality technology. These include the Virtual Windtunnel (see figure 1) [Bryson and Levit 1991][Bryson and Gerald-Yamasaki 1992] and Virtual Spacetime [Bryson 1992b]. Molecular modelling [Brooks et. al. 1990], scanning tunnelling microscope display and control [Taylor et. al. 1993], and medical visualization systems [i.e. Bajura et. al. 1992] have been developed at the University of North Carolina at Chapel Hill. A system to investigate cosmic structure formation has been implemented at the National Center for Supercomputing Applications [Song and Norman 1993]. The Electronic Visualization Laboratory at the University of Illinois Chicago has implemented several scientific visualization applications [Cruz-Neira et. al. 1993a][Cruz-Neira et. al. 1993b]. In addition, there is the large number of proposals for the VROOM event, a showcase of applications of virtual reality to scientific visualization to be held at SIGGRAPH 94.
There are several reasons why virtual reality and scientific visualization are well matched in addition to the inherently three-dimensional display and control described above. Scientific visualization is oriented towards the informative display of abstract quantities and concepts, as opposed to attempting to realistically represent objects in the
real world. Thus the graphics demands of scientific visualization can be oriented towards accurate, as opposed to realistic, representations. Graphical representations can be chosen which are feasible with current technology. Further, as the phenomena being represented are abstract, a researcher can perform investigations in virtual reality which are impossible or meaningless in the real world. In this way the best aspects of computer simulation and intuitive real-world interaction are combined. Finally, scientific visualization encompasses a wide set of problems, some of which are feasible with current technology, some of which are not. Problems can be identified which can immediately benefit from a virtual reality interface with the current state of the technology. Such applications serve to demonstrate the effectiveness of virtual reality technology, as well as provide valuable scientific visualization tools.
In many ways the major impact of virtual reality technology on scientific visualization is in providing a “real-time” intuitive interface for the exploration of data, facilitating the use of scientific visualization in the research process. We shall now discuss how this capability fits into the history of scientific visualization. We wish to make two points: that “real time” exploration is a desirable capability with special demands and requirements, and that virtual reality interfaces greatly facilitate exploration capabilities.
Traditionally, scientific visualization has been used in two roles: exploration and presentation. In the early days of visualization, it was rather difficult for the researcher to produce visualizations beyond conventional drawings and plots. Familiarity with computer graphics programming was required to do more sophisticated visualization, a need that was addressed through the creation of “visualization shops”, where a visualization was produced “to order”. A researcher provided data to a visualization programmer, who then produced a high-quality image or animation. Thus there was a significant investment involved in the production of visualization. This served the purpose of visualization as a presentation medium, but I claim that it hindered the use of visualization as an exploratory medium.
In recent years, the problem of using visualization as a medium for exploration and discovery has been addressed through the development of visualization systems which attempt to present to the user significant visualization capability without requiring significant programming skills. There were fundamentally two approaches:
• data-flow systems, such as AVS and Iris Explorer, which used data-flow lan-guage concepts to allow the researcher to visually program a visualization. These systems include-ed a standard set of visualization tools.
• Specialized visualization programs, such as Plot3D and FAST [Bancroft et. al. 1990] for CFD visualization, which are systems dedicated to a particular type of visualization problem.
These systems put significant power in the hands of researchers who did not wish to spend their time writing complex graphics programs. Nonetheless, while these systems facilitated the development of quite sophisticated visualizations in a few minutes, they still required considerable effort. We shall now briefly state some of the obstructions to exploratory visualization in the above systems. We shall discuss these problems in greater depth, as well as their solutions, in the rest of this paper.
visualization:
• The time required to render the visualization given the user’ s input. • The interface which controls the visualization.
The first problem, the time required to render the visualization, produced a limit on the researcher’s ability to sample a data set, looking for interesting features. This time is due to several factors: the speed of the visualization system hardware, the algorithms used to compute the visualizations, and the size and complexity data to be visualized. We shall maintain in this paper that reducing the time to render a visualization significantly enhances the usefulness of visualization in the exploration and discovery process. This consideration naturally leads to systems that respond in “real time”. We shall make this precise below.
The second problem is that of the user interface. Many datasets fill three-dimensional volumes, often varying in time. Different locations in such data sets often exhibit strikingly and interestingly different features. Difficulty in specifying these locations will impede exploration of that data set. Early visualization systems specified the location of visualizations via command-line text input. Later systems, such as FAST and Iris Explorer, used Graphical User Interface (GUI) based concepts such as sliders to control the location of visualizations. Modern research visualization systems, such as SuperGlue [Hultquist and Raible 1992] allow the direct placement of visualizations using mouse-based input. This still leaves the problem of mapping the two degrees of freedom of the mouse to the three-dimensional volume containing the visualization. The natural solution is to provide a fully three-dimensional control and display interface, as we shall describe below.
To summarize, we claim that there are two capabilities that we feel are required in a system that wishes to support interactive exploration of complex data sets:
• real time interactive capability
• a natural, “anthropomorphic” three-dimensional interface
In section 2, we shall describe in more detail the motivation for allowing real-time interaction. Section 3 discusses the difficulties encountered in the development of real-time interactive response in interesting visualization problems. Section 4 addresses how these difficulties can be addressed. Section 5 considers the problem of natural, anthropomorphic interfaces which combined with real-time response leads naturally to the development of so-called virtual reality or virtual environment systems for visualization [Bryson 1992a].
2. The Advantages of Real-Time Visualization
Traditionally, scientific visualization has been based on static or animated two-dimensional images. These images have required a significant investment in time and
expertise to produce. While packages such as AVS and Iris Explorer have significantly reduced the time involved in producing an image or animation, a fair amount of time is still required to produce an image. For this reason scientific visualization has proved to be a difficult platform for the exploration of data, as the investment required to produce the image has limited the number of ways a data set can be practically explored. Typically the researcher produces a few images which highlight the expected regions of interest. While for simpler data sets a few images are sufficient to display all interesting aspects of a data set, for complex data sets such as those arising in simulated fluid flows there are many different interesting ways to view phenomena, each of which can require significant effort to produce.
Let us consider the example of computational fluid flow in somewhat more detail. Say a researcher wishes to understand a complex flow such as the powered lift simulation described below. A common method of understanding the geometrical structure of such a flow is to use streamlines (integral curves) of the fluid’s velocity vector field [Bryson and Levit 1991]. Streamlines require as input an initial position in the vector field. In conventional visualization systems, a streamline can require on the order of a second to draw. If a fluid flow data set is not well understood in advance, so that the researcher does not initially know where to place the streamline’s initial position in order to produce interesting streamlines, she must move that initial position about in the data set, watching for interesting resulting streamlines. If each move requires a second or more per streamline to display results, it requires patience and a fair amount of the researcher ’s time to get a good feel for all the phenomena in the volume containing the fluid. More often, the researcher has a sense of where the interesting streamlines will originate and will concentrate on placing the streamlines in those regions of expected interest. The investment required to place the streamlines effectively discourages the placing of streamlines in a way that would show unexpected activity.
For the reasons outlined above, the use of scientific visualization for exploration of unknown data sets has been a somewhat clumsy task. One speaks of producing “a visualization” or “a picture”, with each picture taking a fair amount of the researcher ’s time and effort. Under these circumstances it is to be expected that the researcher will hesitate to produce a picture that she expects to throw out. Thus scientific visualization has emphasized presentation, where a great deal of effort is put into a few visualizations. It has thus been natural to develop visualization groups, which produce high-quality visualizations according to the scientist’s specifications.
Everyone knows that faster visualization systems are desirable. Until recently, however, speed and performance were one of several values, along with accuracy, ease of use, access to very large data sets and elegance of program structure. We wish to argue in this paper, however, that for a wide range of scientific visualization applications performance is a significant value, enabling the rapid exploration of data. This exploration may potentially increase the use of scientific visualization in the research process. In accord with the colloquial usage, we shall call a system with sufficient performance to respond quickly and interactively with the user a “real time” visualization system. This usage of the phrase “real time” is distinct from the usage in the phrase “real time operating system”, where demands are far more stringent than those we shall discuss here.
There are two aspects to real-time interaction: interactive response and animation rate. Interactive response is how quickly the user sees the results of actions. If the interactive response is too slow, the user will experience difficulty in precisely placing visualization
tools [Sheridan and Ferrill 1974]. Animation rate is how fast images are presented to the user, which is particularly relevant with respect to viewing control and for time-varying data sets. If the rate is too slow, the images will be perceived as a series of still pictures rather than a continuous evolution or movement. We shall see below that there are reasons in virtual reality systems, to wish to separate the computation and visualization processes, so that they run asynchronously. In this case there may be several animation rates, i.e. the computation rate (the rate at which visualizations are computed), and the graphics animation rate (the rate at which pictures are presented to the user).
Experience has shown that in order to sustain usable interaction and to make the user feel that a series of pictures integrates into an insightful animation the following criteria must be met:
• The system must provide feedback regarding the effect of continuous user input to the user in less than 0.1 seconds.
• An animation must run at about 10 frames per second, particularly the graphics animation rate.
We shall call these criteria real time performance constraints. These performance demands are very loose and will vary from user to user. Some users have found five frames per second to be sufficient. Most users find 10 frames per second to be sufficient. Of course one wishes to do much better than these numbers. If the above demands are not met, however, then the system will loose many if not most of the advantages of real-time interaction.
In summary, we claim that there are several advantages of real-time visualization systems from the perspective of a researcher:
• The ability to quickly sample a data set’s volume without cluttering the visu-alization
• No penalty for investigating regions which are not expected to be of interest • The ability to see the relationship between data nearby in space or time
without cluttering up the visualization
These advantages can be summed up in the phrase “real-time interaction encourages exploration”.
Once real-time performance has been elevated to a dominant value, a visualization system must be designed so that other values such as accuracy, data access, programming elegance, and ease of use are not overly compromised. Nonetheless some compromise is often necessary to allow real time interactive performance. Strategies in designing such systems are the topic of this paper.
3. Demands of Real Time Visualization
a single value at every point. For such problems, a single well-designed picture can provide a great deal of insight into that data. For many problems, however, particularly modern engineering simulations, the data sets consist of a time series of three-dimensional volumes, with many parameters at each point. Further, one is often interested in behavior induced by that data (i.e. streamlines in a vector field) rather than the data values themselves. It is in these latter cases that we expect real-time interactive visualization to pay off, due to the complexity of phenomena that can occur in a three-dimensional volume. It is also in the latter cases, however, that real-time interactive visualization is most difficult, due to the size of the data sets and the complexity of the computations involved in the visualization process.
An instructive example is the visualization in computational fluid dynamics (CFD). As of early 1993, the data sets resulting from CFD computations typically provide five parameters (three components of velocity, pressure, and energy) at every point and are typically very large, with millions of points per timestep for hundreds or thousands of timesteps. These data sets are rapidly growing in size due to the increase in available computational power. The CFD visualization problem is further complicated by the fact that the data is provided on either multiple overlapping stretched computational grids, or on unstructured grids. In both cases position data for each point must be provided explicitly, further increasing the data size, particularly for simulations in which the computational grid points move in time. It is estimated that terabyte CFD data sets are not very far in the future. The computations involved in CFD visualization are computationally intensive. Widely used visualization techniques include particle advection techniques such as streamlines and particle paths, and scalar isosurfaces which involve marching through an entire timestep of data to determine the location of the isosurface.
To focus the discussion, let us consider a modest modern data set, that of a harrier jump-jet aircraft hovering above the ground [Smith et. al. 1991]. This data set contains in each timestep five floating point numbers at 2.8 million points distributed among 18 overlapping computational grids. There are 106 timesteps. The total data set size is 5.6 gigabytes, which breaks down to 56 megabytes per timestep, not including the single 42 megabyte data set which provides that positions of the grid points (and other necessary geometry data).
As an example, consider streamlines. This techniques often involves several hundred integrations in the computation of a single streamline. Each integration should use a high-order technique for accuracy, which increases the computational burden. Further, several hundred streamlines are often desirable. The net result is that streamlines can be very computationally expensive. To more precisely estimate the computational demands of a streamline computation, consider the second-order Runge-Kutta integration technique. When highly optimized by performing calculations in computational grid coordinates, so that the position directly provides the grid coordinates (assuming the initial position is provided in computational grid coordinates), a single integration of a streamline requires about 200 floating point operations and 24 accesses to the data set. A typical modern (c. 1993) high-performance workstation will have peak scalar performance ratings ranging from 10 to 100 megaflops, with perhaps 20 megaflops being typical. This system will be capable of computing 2 million floating point operations in 0.1 seconds, which implies an ability to perform just under 10,000 integrations. If each streamline involves 200 integrations, then this system will be able to compute at most 50 streamlines. In actual use,
most computer systems perform at about half their rated performance, implying that one could expect to compute about 25 streamlines. Similar analyses can be performed for the other visualization techniques.
The situation with respect to data management is similarly demanding. While the amount of data required in the above example computation is modest (3 trilinear interpolations each requiring 8 vectors with three components of four bytes each in IEEE 32 bit representation = 288 bytes), one wishes to perform many of these computations and these 288 bytes are scattered across the data set in 24 separate places (again, assuming that the components of each vector are stored contiguously). Consider the data as stored on disk: assuming that the data on the disk can be accessed at 3 megabyte/second, then bandwidth considerations alone impose a limit of about 1,000 integrations in 0.1 seconds. Further, as the data is scattered across disk, the disk must locate the data, which requires further time (called “seek time”). A typical workstation disk is capable of performing about 23,000 seeks/second, implying that only about 95 integrations can be performed off disk in 0.1 seconds based on seek time considerations alone, insufficient to compute even one of the above streamline. Thus both bandwidth and seek time imply a severe limit on the ability to do real-time computations from disk.
The obvious solution is to place the data in physical memory. Systems that can accommodate multi-gigabyte data sets are rare, however, and tend to be multiple user systems. While multi-gigabyte workstations will become available in the near future, the size of data sets will also be growing dramatically.
Finally, when the data set is rich and complex, it is often desirable to have many visualizations operating simultaneously. Thus graphical performance becomes an issue, particularly when the visualizations include objects like isosurfaces containing tens or hundreds of thousands of polygons.
Similar problems are encountered in the other visualization techniques. 4. Strategies for Meeting the Real-Time Performance Demands
To some extent, advances in technology will provide relief for the difficulties discussed in the last section. Processors and disks will be faster, and we will see workstations with more and more memory. It is clear, however, that the response to higher performance will be to make greater demands on the system, as we wish the frame rate to be as high as possible, and we will wish to have many visualization techniques in a single environment. Disk access times have to improve by two orders of magnitude to catch up with the current computational bottleneck. By the time this occurs the computational performance will have improved by as much as two orders of magnitude (based on the expected availability of 300 megaflop processors in multiple processor systems by 1994). Available memory is also taking an order of magnitude leap in the 1993-1994 period, with 16 gigabyte systems being expected in 1994.
The problems addressed in real-time scientific visualization will, however, far outstrip the advances in technology. Increased computational capability will be used to perform simulations that produce far larger data sets, and contain far more phenomena. There are several ways in which this will occur: the spatial and temporal resolution will increase for enhanced accuracy, increasing the number of points and timesteps; and multi-disciplinary simulations will be performed, including many phenomena in a single data set. An example currently under development is to include chemistry and airframe structure
dynamics and data into a CFD simulation. Thus the size of data sets will increase dramatically over the next few years. The complexity of the simulations will place further demands on the visualization system, implying a greater computational demand. Yet it is exactly the further complexity of these anticipated data sets which makes real-time interactive visualization capability more desirable.
There are no simple, general rules for meeting the performance requirements that can be applied across visualization applications. Sometimes precomputation of derived quantities is required for computational speed, and sometimes computation on the fly is forced by data management considerations. We shall consider several of the issues that arise.
4.1 Computation
At the time of this writing, graphics workstations with rated peak performance of a few gigaflops have been announced for 1994. This significant jump in computational power certainly alleviates the difficulty in providing real-time interactive computational performance, but as discussed above there will be circumstances when even this computational capability will not be sufficient for real-time response. An example is a CFD visualization environment which contains hundreds of streamlines, thousands of particles, and several isosurfaces for an unsteady data set. Another example would be ray-traced volumetric rendering of data.
The computational demands of a real-time interactive visualization system are determined by the particular visualization techniques used in that system. A visualization based on simply color-mapping an existing geometry (such as a face of the computational grid) required very little computation. A technique based on particle integration has the computational demands described above. Isosurface computation has computational demands which are dependant on the number of points in the data set, and so will be dependant on the size of the data set. Even something as seemingly innocuous as a color-mapped cutting plane has some computational demand, particularly for stretched curvilinear grids, as the spacing of vertices of the cutting plane must match the non-linear spacing of the data points, and the values of the data at those vertices must be obtained via tri-linear interpolation.
In spite of the dependence of the computational demands on the specific visualization techniques implemented, some general observations can be made. The following discussion is to be taken in the spirit of examples, and is not intended to be exhaustive.
There are several considerations in meeting the computational performance demands: • computational architecture: scalar vs. vector vs. parallel architectures
• computational algorithm: how is the visualization being computed? This consideration usually involves trade-offs between performance and accu-racy
• choice of data representation to facilitate computation: can interesting aspects of the data be precomputed, or is there a representation of the data which facilitates the computation?
• careful code optimization
We shall discuss each of these in turn.
Computational Architectures: The most basic computational architecture is called a scalar architecture, which essentially performs one operation on one number at a time. There are several flavors of scalar architectures, such as cached, which moves numbers into special high-speed access areas, and pipelined which moves the numbers through the load/operate/store process in an optimized way. Scalar processors are very versatile.
Much of the performance of modern supercomputers is due to special processor architectures. An example is the vector architecture, which can load a collection of floating point numbers into a “vector register” and perform the same operation on these numbers simultaneously. For certain problems, particularly large matrix manipulations, vector architectures are very fast. Some visualization techniques, such as the marching cubes computation of an isosurface, can take full advantage of vector processors. To use vector architectures to their full potential, however, one must perform several operations per load of the numbers into vector registers, to balance the overhead of the initial load of these vector registers. If the register needs to be reloaded for every operation, the advantage of vector processing is lost. Thus vectorization of code (the writing of code so that it will indeed use the power of a vector processor) is a delicate and sometimes difficult art. Function calls, accesses to memory, or any other operation which causes the registers to be reloaded will significantly impact performance. An example of a visualization technique which is not well suited to vector architectures is the computation of streamlines. Streamline computation is inherently iterative, with each computation requiring the results of the last computation. Thus a single streamline cannot be vectorized, and it is found that non-vectorized performance on a supercomputer can be comparable or slower than a fast scalar workstation processor. If one has multiple streamlines, one can attempt to vectorize across the streamlines. This indeed improves performance somewhat (but not dramatically) in the case where the streamlines are computed in parallel. The full performance of the vector processors is not reached, however, due to the many accesses to memory involved in the computation of a streamline. In the case of streamlines on multiple-grid data sets, however, the advantage of vectorization will be lost completely if any of the streamlines were to fall off of one grid onto another while others did not. In this case the streamline computations would no longer be exactly in step and the computation would no longer vectorize.
Another architecture used in modern supercomputers is the use of many small (usually) scalar processors in parallel, called massively parallel processing (MPP). One may, for example, compute many streamlines in parallel by assigning a streamline to a processor. There are several models of MPP, each with its own advantages and disadvantages. On some MPP systems, each processor has a small area of memory and data is distributed among the processors. When a processor requires data which is in another processor’s memory, messages must be passed increasing the data access overhead. Thus such systems are not suited for high-performance computation with large data sets. Some MPP systems have all the processors sharing a single memory area. In this case, the main limitation on performance is the speed of the individual processors. When these processors are small, for example, they may not be capable of computing individual streamlines within the 0.1 second constraint. Thus as a single streamline computation
cannot be parallelized for the same reasons that it cannot be vectorized, it does not matter how many processors you have as it will still take more than 0.1 seconds to compute one or many streamlines. Using the full performance potential of MPP systems is fraught with difficulty, though as experience with programming these system increases they will presumably become easier to program.
The current trend in workstation architectures towards several very fast scalar processors running in parallel sharing the same physical memory space provides a very good computational architecture for high-performance visualization. As it is based on scalar architectures, this type of architecture provides performance in the most versatile way. Presumably the near future will see this architecture scaled up to MPP range (hundreds to thousands of processors). While it remains to be seen if performance actually scales in such a system, I think we have reason to be optimistic.
Computational Algorithm: One of the most important considerations in scientific visualization is that the graphics rendered accurately represents actual phenomena in the data. This value is at odds with computational performance, however, as there tends to be an inverse relationship between the computational accuracy and computational performance of algorithms. Consider again the issue of streamlines of a vector field, which are computed by numerically integrating an ordinary differential equation dx(s)/dt = f(x, s), where f(x, s) is the vector field, x a point on the streamline, and s the integration parameter (not necessarily time), given an initial condition x(t0). There are many algorithms for integrating such equations, which are of varying accuracy. The example discussed above used a second-order Runge-Kutta integration, which has an accuracy proportional to the square of the size of the integration step. Higher-order algorithms are available, but they typically take more time to compute. A more accurate method is to use an adaptive integration algorithm, which chooses its integration step size based on various criteria such as the magnitude of the vector at that point. In these methods, one obtains accuracy by either selecting a more accurate algorithm, thereby slowing down the computation, or selecting a smaller integration step, which results in either shorter streamlines in the same time or a longer time to compute a streamline of the same length.
The compromise between accuracy and performance is a painful one, and one that, I feel, should not be made by the system developer. The user should be given run-time, interactive control over both the choice of algorithm and the parameters which determine the performance of the computation, such as the integration step size in the example of a streamline. With this type of control, the researcher can do quick exploration in a low-accuracy mode (i.e. moving many long streamlines), then verify any discovered phenomena using higher-accuracy algorithms. The most important feature in this area is that the researcher know and understand the accuracy of the tool currently being used.
One can, of course, implement some kind of architecture where only those visualizations which have actually changed are being recomputed. The advantages of such an architecture are lost, however, when the visualization is of a time-varying vector field in which, for example, all streamlines must be recomputed all the time. Another optimizing move is to store all visualizations when they are computed, so that when the time cycles around (in a periodic simulation) the visualizations which have not changed can be read from memory rather than recomputed.
visualization can be represented in ways that can significantly enhance the performance of computations associated with visualizations. This strategy often involves choosing opportunistic coordinates or precomputing data in a way appropriate to the desired visualization technique. The specifics of such representations will, of course, depend on the details of the both the data to be visualized and on the desired method of visualization. We will use the example of the streamline computation described in section 3 to illustrate the point. In that example the data (the velocity vector field) was in computational grid coordinates, so that the vector values encode, at each point in the grid, where in the grid a particle at that point would move to in terms of the grid coordinates (indices). Thus given a particle at point (x, y, z), the vector (∆x, ∆y, ∆z) at (x, y, z) indicate that the particle would move to the point (x+∆x, y+∆y, z+∆z). These coordinates directly provide the grid indices (i, j, k) = integer part of (x+∆x, y+∆y, z+∆z) for the new particle position, so the vector at the new position can be immediately read and the process iterated. Given the position of the particle in grid coordinates the position of the particle in world coordinates is looked up in the position data set, which is provided as part of the stretched grid data. For particles which do not fall exactly on grid points some kind of interpolation (trilinear in the example of section 3) between the grid points is necessary. The alternative (and perhaps more intuitive) method would be to perform the streamline computation in physical space, so that the coordinates of the point provides the actual position of the particle (in world coordinates). In this case the vectors are in physical coordinates. In order to determine the vector at the current particle position, however, it is necessary to determine the indices of the nearest grid points to the current position. As grids in CFD are highly stretched and irregular, locating the nearest grid point involves nearest-point searches of the grid. While one can use information such as the location of the last grid point and estimates of where the particle should have moved to to optimize the computation of the new nearest grid point, (sometimes significant) grid searching is usually involved. This grid searching can involve more time than the computation of the new particle position once the vector is obtained. Thus as the vector data is usually provided physical coordinates there is considerable advantage in pre-converting it to computational coordinates.
Another class of data representations which can increase actual computational performance is the visualization of derived quantities. Examples in CFD include quantities such as the velocity magnitude and vorticity. Computation of such quantities can be done off-line, producing new data files which can be directly visualized. Another example is the computation of geodesics for the visualization of numerical spacetimes in simulations of general relativity [Bryson 1992b]. These geodesics are computed in terms of quantities which are derived from the basic variables output by the numerical calculation. The computation of these derived quantities is complex, involving several derivatives per quantity so considerable computational savings are achieved by computing these pre-quantities. They are considerably larger in data storage than the original variables, however, so pre-computation aggravates the data management problem described below.
Careful code optimization: It is obvious that when performance is paramount code should be carefully optimized. This requires a good, low-level understanding of both the computational hardware that one is using and the programming language or development environment in which one is programming. We shall make some fairly obvious observations which are often neglected in the rush of actual development. Pay the greatest
attention to the code that gets executed most often. Understand how an array reference is decoded to access memory, and what alternatives may be faster. While compilers will often provide significant optimization (learn about the options!), there are often things the programmer can do to further increase performance.
Be aware of the impact of memory accesses or function calls on special computational architectures such as vector processors. Perform benchmarks to be sure that optimized code is really optimized. When parallelizing code, benchmark to be sure that the parallelized parts of the code has really sped up by the expected factor.
4.2 Data Management
As described in section 3, data management problems in modern scientific visualization can be quite severe. Given the current data access overheads encountered in mass storage systems discussed in section 3, the only viable data management strategy seems to be to have all data accessed in the course of the visualization session stored in physical memory. This implies that real-time interactive visualization systems rely on hardware platforms with large amounts of physical memory. Happily, 1993-4 is seeing the introduction of graphics workstations supporting as much as a 16 gigabytes of physical memory. Such systems are quite expensive in the short term, however, and as pointed out in section 3 interesting data sets can be larger than this and are growing rapidly. Thus large physical memory alone is not sufficient to solve all data management problems. There are a number of obvious strategies that can be implemented.
One data management compromise is inspired by the observation that one is often interested only in data within some small subvolume of the entire data set. The researcher may be allowed to select such a subvolume, load the data for only that volume, and perform interactive visualization for that subvolume. Such a strategy has been implemented by Michael Gerald-Yamasaki in the context of unsteady CFD visualization.
One may also subsample the data, choosing for example only every other data point. With respect to spatial subsampling, this strategy should be implemented with great care, and is not usually advisable. Numerical simulations are often performed with the smallest number of points consistent with the accuracy of the simulation, with grids stretched in such a way that the density of points is just right for the phenomena being modeled. Thus further reducing the number of points can result in the introduction of spurious artifacts and incorrect visualizations. Experience in CFD visualization has indicated that subsampling by only a factor of two (retaining every other point) results in phenomena that are qualitatively incorrect. Unsteady data sets are, however, often computed by simulating each timestep independently. In these cases subsampling in time does not compromise the accuracy of the simulation at each timestep, though one may miss phenomena which occur only in individual timesteps. Thus subsampling unsteady data sets in time can be a viable option.
The development of workstations with considerable computational capability suggests compression strategies may be used to reduce the amount of data that need be stored on disk or other mass storage devices. In this strategy one decompresses the data as needed, accepting increased computational overhead for the sake of increased data capacity. Great care must be exercised when choosing such a strategy, however. Many compression strategies rely on patterns in the data to be compressed, but numerical data is typically composed of floating-point numbers which contain essentially no pattern, particularly in
the bits representing high accuracy. In these cases, pattern-based compression techniques are probably inappropriate, and can often output “compressed” data files which are larger than the “uncompressed” input. Another compression strategy is to treat the data as a field and transform it in come functional sense, i.e. through a fourier, wavelet, or eigenfunction expansion. For example, using the Fourier transform the data is expressed in terms of sums of constituent frequencies. In some cases, determined by the values of the data, this sum of frequencies is short. In other cases compression is achieved by truncating the frequencies above a certain value. The problem with these methods is that they are “lossy” in the sense that any compression implies loss of data, i.e. in the case of fourier-based compression schemes by the loss of high-frequency information. Thus they can be expected to introduce artifacts into the visualization process and so are usually unsuitable for scientific visualization applications.
Another approach to data management is to arrange the data on disk in such a way that it can be loaded “as needed” at interactive rates. When the desired visualization technique is isosurfaces of a scalar field, for example, the field values may be re-ordered on disk by the data field value so that only the data with values near the desired field value would be accessed. The data would be “binned” so that points with data between two values would appear sequentially on disk, requiring only one seek for a load. The size of the bins (resolution of data values) would be chosen so that all the values in the bin could be loaded sufficiently quickly to allow interactive isosurfaces. The isosurface computation is also optimized by this strategy, as only the points in the bin (or nearby bins) need be searched. This type of strategy can potentially be generalized to other data types and visualization techniques.
4.3 Network Overhead in Distributed Systems
A strategy which addresses both computational and data management requirements is to distribute the visualization system across several hardware platforms. In this way resources may be allocated in ways that make the best use of their capabilities. For example, a computer system may be available which has very fast disk access but slow computational and graphics performance. Another system may have very large physical memory while still another may have very fast processing capability. Or one may wish to perform different computations on different platforms: a vector computer may be used for scalar data to compute isosurfaces, while a fast scalar machine would be used for the vector field, computing streamlines. The resulting geometries could be collected by a graphics workstation for rendering.
Distributed systems introduce into the above performance considerations the consideration of the network connecting the system components. There are three concerns that appear when a system is distributed: network bandwidth, network latencies, and synchronization of the various components. Cutting-edge, high-performance networks can also significantly raise the amount of effort required in the development of a visualization system, requiring the assistance of network specialists and support from potentially all hardware vendors involved.
Network bandwidth is the rate at which data can be shipped across a network. This is typically a marketing, and is not always well defined in practice. As an example, consider the UltraNet, a gigabit network which when moving large packets of data (> 10 megabytes) actually performed as advertised in the 90 megabyte/second range. When
moving small packets, however, it was considerably slower, down to 1 megabyte/second, indicating that this network is optimized for large packets. Other high-performance networks such as the High-Performance Parallel Packet Interface (HIPPI) are becoming available for point-to-point networks, but gigabit local or wide area networks are as of this writing only promises. The next best networks are T1 and T3 lines, which deliver bandwidths in the 10 megabyte/second range. Finally there is ethernet, which is standard and therefore convenient but delivers a bandwidth of only a megabyte/second.
The bandwidth requirements of a visualization application of course depend on what the application does. These demands are determined by the amount of information that must be transmitted over the network. A typical CFD visualization environment can have hundreds of thousands of points contained in streamlines, with 12 bytes/point (assuming four-byte floating point representations). This implies over a megabyte per timestep in unsteady CFD visualization environments for the streamlines. Isosurfaces can contain thousands of triangles each, requiring hundreds of thousands of bytes (assuming the triangles are unconnected, implying 3*12 = 36 bytes/triangle). Thus the network demands of a distributed visualization can be considerable.
There are several ways to partition a visualization system. For simplicity let us consider a two-component system, which uses a remote supercomputer for data management and computation communicating over a network with a graphics workstation which provides the display. This is the same distributed architecture used in the virtual windtunnel [Bryson and Gerald-Yamasaki 1992]. It is clearly optimal to perform the computations associated with the visualizations on the remote supercomputer, which results in geometry which we shall assume in this example is three dimensional. The questions are: which system performs the projection of this geometry onto the graphics; what is shipped over the network; and how is the interactive control distributed? The answers to these questions will depend on many details of the application. If the network has low bandwidth and the geometry contains a large number of three-dimensional points, it may be optimal to perform the projection from three-dimensional world coordinates to two-dimensional screen coordinates on the remote supercomputer, reducing the amount of data sent over the network (two-dimensional arrays of integers rather than three-dimensional arrays of floating point numbers). The disadvantage of this partition is that the supercomputer is presumably busy computing the three-dimensional geometry while the graphics workstation presumably is optimized to perform the graphics projection in hardware. The advantage is the reduction in network traffic. If the display is stereo, two projections are required for each point, implying four integers per point. Thus unless one uses two-byte integer representations, in the stereo case sending the three-dimensional geometry is optimal with respect to network traffic as well as which machine performs the projection. Each case should be carefully examined to determine the optimal partition. 4.4 Graphics
Computer graphics is at the heart of the visualization process. In broad terms, scientific visualization is the mapping of abstract quantities into graphical representations. The specifics of this map in a real time interactive system are to a large extent determined by the real-time performance constraints described in section 2.
There are two performance considerations in modern three-dimensional graphics workstations: vertex processing time and pixel scan conversion time. The vertex
processing time is the time required for a three-dimensional point to be projected onto a two-dimensional screen. This is the parameter usually quoted in the performance literature as “n vectors per second” or “n triangles per second” (the triangles are typically assumed to be defined in a connected mesh, so that a new triangle requires only one new vertex). The pixel scan conversion time is the time required to fill in a projected polygon on a screen. This performance parameter depends on several aspects of the rendering, such as the area of the projected polygon, the shading algorithm (flat vs. garaud, etc.), lighting and so on. A general rule of thumb is that for a particular style of rendering the pixel scan time is directly proportional to the area of the projected polygon to be rendered.
Luckily, for many scientific visualization applications the graphical rendering is, to some extent, arbitrary and can be chosen specifically to meet the performance constraints. This situation is in contrast to high-fidelity visual simulation of the real world, which can require time consuming photo-realistic rendering techniques. Consider again the example of streamlines in CFD. A streamline is simply an array of points in three dimensions. This array can be simply rendered as lines, which allows very fast rendering. In this case the performance limitation from streamlines arises almost entirely from the projection of the points in the array defining the streamlines. Such issues as the number of pixels that must be set (the scan conversion overhead) in this case do not enter. A method of rendering streamlines which perhaps provides a more pleasing image is to render the streamline as a shaded tube. This involves many more points and pixels, however, and may not be optimal for real-time interactive systems.
This example can be generalized to the following principle for the development of real-time interactive visualization systems: keep the graphical rendering as simple as possible. Use points instead of spheres, use lines instead of tubes, and so on. While it is true that, for example, using lighted tubes rather than lines provides more depth cues to show three-dimensional structure, the ability to provide real-time interaction can also provide strong depth cues, either through allowing interactive rotations or through the use of head-tracked rendering (see section 5). Also remember that while a technique may not be very time consuming when done once, one may desire many instances of this technique in an environment.
There are, of course, visualization techniques which will demand considerable graphics performance, such as isosurfaces and volumetric rendering. In a large data set a typical isosurface may contain tens of thousands of (logically) disconnected polygons. This large number of polygons will be difficult to render within the real-time performance constraints. Ways of meeting this problem for isosurfaces include subsampling the surface (potentially masking interesting features), rearranging the polygons so that they are optimized for the graphics hardware, hashing the polygon list to replace coplanar neighboring polygons with a single polygon, and limiting the spatial extent of the isosurface. Many or all of these techniques may be necessary to achieve the real-time performance demands, and even then a very limited number of isosurfaces may be practical. Time intensive methods such as ray-traced or volumetric rendering may be impractical in real time applications.
5. Virtual Reality Interfaces
The discussion has, up to this point, primarily addressed the issue of real-time performance. In section 1, however, we suggest that real-time performance is only one of
two requirements of a scientific exploration environment. The other requirement is a natural, “anthropomorphic” interface. By this we mean an interface which requires as little as possible of the user’s attention, which is used in as natural a way as possible, and provides as unambiguous a three-dimensional display as possible. This is to be contrasted with the current interaction paradigm in scientific visualization, which is based upon text or two-dimensional input via graphical user interfaces and two-dimensional projections of three-dimensional scenes. Conventional interfaces make it difficult to specify positions in three dimensions and do not provide unambiguous display of three-dimensional structure. Virtual reality interfaces attempt to provide the most anthropomorphic interfaces possible. There are two components to be considered in virtual reality interfaces: display and user control. We shall consider them separately, discussing the concepts involved as well as the difficulties encountered in use.
5.1 Three Dimensional Display
Scientific visualization makes particular demands upon virtual reality displays. The phenomena to be displayed in a scientific visualization application often involves delicate and detailed structure, requiring high-quality high-resolution full color displays. Experience has shown that displays with 1000x1000 pixel resolution are sufficient for many applications. In addition, a wide field of view is often desirable, as a wide field allows the researcher to view how detailed structures are related to larger, more global phenomena.
The combined demands of wide field of view and high display quality and resolution is, as of this writing, extremely difficult to put into a head-mounted display. The alternatives of the BOOM displays and CAVE-type displays are proving popular for many scientific visualization applications. Both types of displays are, however, high cost and have several disadvantages.
Another factor to be considered in selecting virtual reality visual displays for scientific visualization is the issue of user acceptance. There is reason to doubt that many researchers would be willing to invest the time required to don and doff head-mounted displays available at the time of this writing. Further, many researchers have expressed distaste for donning helmets or strapping displays onto their heads. Both the BOOM-type and CAVE-type displays avoid many of these problems.
5.2 Interaction
The main point of this paper is that real-time interactive environments can enhance the role of scientific visualization in the scientific discovery process. Such a system will clearly require interactive capabilities which allow intuitive control of the data visualization displays, avoiding as much as possible difficulties due to the arbitrariness of the interface. The most obvious need is for the ability to rapidly and precisely select a location in three dimensions. There are three components to this ability: the ability to specify a location, an action at that location, and the ability to provide feedback as to the location selected. This feedback would typically be provided by the three-dimensional display discussed above. This section concerns the specification of the location and the action to occur at that location.
position in three dimensions, using one of a variety of available tracking technologies [Bryson 1992c]. All of these technologies suffer from various problems, including lack of accuracy and cost. Nonetheless, the advantages gained seem to significantly outweigh the disadvantages.
In a conventional workstation/mouse environment, there is currently no standard way of specifying a location in three dimensions. A mapping from the two-dimensional mouse position to three-dimensional space must be defined, usually using various button-press states. These mappings are necessarily non-intuitive. In this case some kind of visual feedback as to the position selected is critical to the ability to select a position in three-dimensional space. This is an area which may benefit from further research, but it is hard to imagine that the use of a two-dimensional mouse could be as fast and easy as the use of a three-dimensional tracker.
The specification of the action to occur at the selected location can be accomplished with conventional hand-held button devices (or keypresses). Buttons have the advantage that they are unambiguous and easy to train. They have the disadvantage that they must be held in the hand and provide an arbitrary method of indicating commands. Particularly in immersive environments, where one cannot see one’s real hands, there may be fatigue problem with the long-term use of hand-held button devices. An alternative is to use a glove device, which tracks the angle of bend of the user’s fingers. Using this device such intuitive gestures as “fist” and “point” can be used to provide intuitive control of the environment. Examples include using the “fist” gesture to grab and move objects, and using the “point” gesture to indicate an object. The problems with glove devices include inaccuracies in measurement and the lack of a standard gestural vocabulary. The use of gloves must be implemented with great care to obtain good results. It remains to be seen if gloves or button devices are preferred by the user community.
The use of natural, three-dimensional interaction raises the issue of how that interaction is used in the environment. Some tasks, such as grabbing and moving a visualization object, are obvious. There are other tasks, however, such as selecting or changing the state of objects, which do not have an obvious intuitive interface. Methods of addressing these issues include the extension of conventional graphical user interface techniques such as menus and sliders to objects in the three-dimensional visualization environment. Much of the problem is a lack of understanding of the human factors issues involved in the design of these types of environments. This is clearly an area which could use more research and experimentation.
5.3 Sound
The use of three-dimensional sound in scientific visualization has been relatively unexplored but holds promise. Aside from the conventional uses of sound to provide user feedback as to the state of the environment, sound can be used as an additional data display channel. Scalar quantities can be mapped to the frequency, timbre, or amplitude of a sound.
5.4 Haptics
Haptics has been applied to scientific visualization in virtual reality contexts in a few laboratory systems [Brooks et.al. 1990][Taylor et. al. 1993][Iwata et. al. 1993]. The
effectiveness of the use of haptics in these research projects have indicated that haptics is a potentially useful data display channel.
6. Looking Towards the Future
Scientific visualization is potentially a very fruitful application area for virtual reality and should be aggressively pursued. Unlike some application areas, significant applications can be developed with existing technology. Critical to this capability are the existence of high-performance three-dimensional graphics workstations with multiple scalar processors with total floating point power of hundreds of megaflops, operating systems which support regularly scheduled pre-emptive lightweight threads, and very large (> 10 gigabytes) physical memory capability. This trend in workstation design towards faster graphics, higher computational power, operating systems supporting concurrent execution, and very large memory should be encouraged and accelerated.
While many significant applications of virtual reality to scientific visualization may be implemented with existing technology, the current state of the technology imposes significant limits. Areas in need of improvement to eliminate these limits are:
Data management--high-bandwidth low latency mass storage systems are required to handle modern visualization problems containing many gigabytes to terabytes of data in a virtual reality application.
Rapid prototyping capability--systems which provide capabilities similar to AVS or IRIS Explorer tailored towards the virtual reality interface. This capability will greatly facilitate the development of applications.
Networks--High bandwidth, low latency networks. Such networks already exist in the form of gigabit local area nets. Such high performance needs to be implemented on long-haul networks to facilitate shared visualization by researchers in widely scattered locations.
Architectures--Software architectures need to be developed which support access to data and computations resident on both local and remote computers. This architecture should scale and make transparent the location of the data or computation, while satisfying the real-time performance constraints. Further, software architectures which facilitate the development of highly concurrent yet synchronized environments is needed. Existing operating systems which provide multi-processing capability only partly address this need.
Improved visual displays--higher resolution, wider field of view, lower levels of optical distortion and improved form-factors. It is possible that head-mounted displays will not be widely accepted by researchers until they are similar in form factor to sunglasses.
Improved input devices--higher accuracy and longer range trackers, as well as devices for sensing user intentions. These devices can include improved gloves, better button devices, speech recognition and so on. Which device is appropriate for which context is yet to be determined.
In addition to the immature state of the technology, there is also an immaturity in our knowledge of how to build useful virtual reality applications. Scientific visualization provides an opportunity to experiment with virtual reality applications in a setting that make reasonable demands of the technology yet requires interesting and imaginative
approaches to system design. For this reason I feel that virtual reality and scientific visualization will provide a rich and fruitful interplay for many years to come.
8. References
[Bajura et. al. 1992] M. Bajura, H. Fuchs, and R. Ohbuchi, “Merging Virtual Objects with the Real World: Seeing Ultrasound Imagery Within the Patient”, Computer Graph-ics: Proceedings of SIGGRAPH ‘92, Volume 26, Number 2, July 1992.
[Bancroft et. al. 1990] G. V. Bancroft, F. J. Merritt, T. C. Plessel, P. G. Kelaita, R. K. McCabe, and A. Globus, “FAST: A Multi-Processed Environment for Visualiza-tion of ComputaVisualiza-tional Fluid Dynamics”, Proceedings of IEEE Visualization ‘90, San Francisco, Ca. Oct. 1990
[Brooks et. al. 1990] F. P. Brooks Jr., M. Ouh-Young, J. J. Blatter, and P. J. Kilpatrick, “Project GROPE - Haptic Displays for Scientific Visualization”, Computer Graph-ics: Proceedings of SIGGRAPH ‘90, Volume 24, Number 4, August 1990.
[Bryson 1992a] S. Bryson, “Virtual Reality Takes on Real Physics Applications”, Comput-ers in Physics, Volume 6, Number 4, July/August 1992
[Bryson 1992b] S. Bryson, “Virtual Spacetime: An Environment for the Visualization of Curved Spacetimes via Geodesic Flows”, Proceedings of IEEE Visualization ‘92, Boston, Ma. 1992
[Bryson 1992c] S. Bryson, “Virtual Environments for Scientific Visualization”, Course Notes, IEEE Visualization ‘92, Boston, Ma. 1992 and “The Implementation of Immersive Virtual Environments”, Course Notes, SIGGRAPH ‘92, July 1992. [Bryson and Levit 1991] Bryson, S. and Levit, C., “The Virtual Wind Tunnel: An
Environ-ment for the Exploration of Three Dimensional Unsteady Flows”, Proceedings of Visualization '91 San Diego, Ca, Oct. 1991,also Computer Graphics and Applica-tions July 1992
[Bryson and Gerald-Yamasaki 1992] Bryson, S. and Gerald-Yamasaki, M., “The Distrib-uted Virtual Wind Tunnel”, Proceedings of Supercomputing '92 Minneapolis, Minn, Nov. 1992
[Cruz-Neira et. al. 1993a] C. Cruz-Neira, D. J. Sandin, and T. A. DeFanti, “Surround-screen Projection-based Virtual Reality: The Design and Implementation of the CAVE”, Computer Graphics: Proceedings of SIGGRAPH ‘93, August 1993. [Cruz-Neira et. al. 1993b] C. Cruz-Neira, J. Leigh, C. Barnes, S. Cohen, S. Das, R.
Engle-mann, R. Hudson, M. Papka, L. Siegel, C. Vasilakis, D. J. Sandin, and T. A. DeFanti, “Scientists in Wonderland: A Report on Visualization Applications in the CAVE Virtual Reality Environment”, Proceedings of IEEE Symposium on
[Deering 1992] M. Deering, “High Resolution Virtual Reality”, Computer Graphics: Pro-ceedings of SIGGRAPH ‘92, Volume 26, Number 2, July 1992.
[Hultquist and Raible 1992] J. P. Hultquist and E. L. Raible, “SuperGlue: A Programming Environment for Scientific Visualization”, Proceedings of IEEE Visualization ‘92, Boston, Ma. 1992
[Iwata et. al. 1993] Iwata, H., “Volume Haptization”, Proceedings of IEEE Symposium on Research Frontiers in Virtual Reality, Oct. 1993
[McCormick et. al. 1987] B. McCormick, T. A. DeFanti, and M. D. Brown, Eds., “Visual-ization in Scientific Computing” Computer Graphics Volume 21 Number 6, 1987 [Sheridan and Ferrill 1974] Sheridan, T. B. and Ferrill, W. R., Man Machine Systems, MIT
Press, Cambridge, Ma. 1974
[Smith et. al. 1991] Smith, M., Chawla, K., and Van Dalsem, W., “Numerical Simulation of a Complete STOVL Aircraft in Ground Effect”, paper AIAA-91-3293, Ameri-can Institute of Aeronautics 9th Aerodynamics Conference, Baltimore Md. 1991 [Song and Norman 1993] D. Song and M. L. Norman, “Cosmic Explorer: A Virtual
Real-ity Environment for Exploring Cosmic Data”, Proceedings of IEEE Symposium on Research Frontiers in Virtual Reality, Oct. 1993
[Taylor et. al. 1993] R. M. Taylor, W. Robinett, V. L. Chi, F. P. Brooks Jr., and W. Wright, “The Nanomanipulator: A Virtual Reality Interface for a Scanning Tunnelling Microscope”, to appear in Computer Graphics: Proceedings of SIGGRAPH ‘93, August 1993.
