Virtual Reality Techniques for the
Visualization of Biomedical Imaging Data
M. A. Shaw, w.
B.
Spiliman Jr.,
K. E. Meissnera, Gabbardc
aThe Optical Sciences and Engineering Research Center, Virginia
Polytechnic Institute and State University;
of Physics,
cResearch Graduate Studies, Virginia Polytechnic Institute and State
University
Abstract
The Optical Sciences & Engineering Research Center (OSER) at Virginia Polytechnic and
State University investigates advanced laser surgery optics, biocompatible material for
implants, and diagnostic patches and other diagnostic and drug delivery tools. The Center employs optics to provide new biological research tools for visualization, measurement, analysis and manipulation. The Center' s Research into Multispectral Medical Analysis and Visualization techniques will allow human and veterinary medical professionals to diagnose various conditions of the body in much the same way that satellite information is used to study earth resources. Each pixel in the image has an associated spectra. Advanced image analysis techniques are combined with cross-correlation of the spectra with signatures of known conditions, allowing automated diagnostic assistance to physicians. The analysis and visualization system consists of five components: data acquisition, data storage, data
standardization, data analysis, and data visualization. OSER research efforts will be
directed toward investigations of these system components as an integrated tool for next
generation medical diagnosis. OSER will research critical data quality and data storage
issues, mult-spectral sensor technologies, data analysis techniques, and diagnostic
visualization systems including the VT-CAVE,(www.cave.vt.edu). The VT-CAVE isVirginia Tech's configuration of Fakespace Systems, Inc Virtual Reality system.
Introduction
The Optical Sciences and Engineering Research Center (OSER) was recently formed at Virginia Tech. The center is a collaborative effort between Virginia Tech and the Carilion
Biomedical Institute.
OSER is specifically tasked with conducting research and
engineering activities involving optics and other disciplines to create knowledge andtechnology to benefit the medical, biomedical and veterinary fields, while supporting the practical goals of improving services and reducing the costs of health care.
Multispectral medical analysis and visualization can allow human and veterinary medical
professionals to diagnose various conditions of the body in much the same way that information from satellites can be studied to yield information about the earth's surface
(vegetation, pollution, and mineral resources, for example). Each pixel in the image has an
associated spectra. Advanced image analysis techniques are combined with cross-correlation of the spectra with signatures of known conditions, allowing automated
Multispectral Medical Vizualization
Advances in sensor technology using multispectral and hyperspectral data acquistion systems can be combined with immersive data visualization techniques to produce
enhanced reconstructions of a patient's condition for diagnostic purposes. These emerging
systems will require developments in sensors, data standards, data management,
computational data analysis, and improved data visualization systems.
Advances in multispectral and hyperspectral technology produce data rich images. These
images can be combined using computational techniques to produce information rich
products. Data compression and transmittal standards and processes for these images must be developed to ensure quality is maintained. Furthermore, the computational algorithms
combined with immersive stereoscopic visualization techniques can result in enhanced
diagnostic capabilities. A rigorous information system is required to support this research
and development effort. The system must be modifiable, flexible, robust, scalable,
maintainable, and have a set of supporting process in place to ensure data integrity, quality control, and availability.
The critical challenge is the development of techniques that will provide the clinician an enhanced diagnostic space. This will require development of new methods for information portrayal derived from new sensor technologies. Display of enhanced three dimensional image elements within an immersive environment and interacive analysis techniques must
be developed. Finally, this new diagnostic space must provide access to all available
patient and other critical information within the work area to provide a complete decision support capability.
Medical Visualization System Components
The goal of a multispectral data visualization system is to provide enhanced diagnosis
capabilities for use by the medical practitioner. The system consists of five components: 1. multispectral data acquisition; 2. data management; 3. data reduction; 4. data analysis; and
5. stereoscopic visualization, (Figure 1.). The data acquisition and visualization systems will provide enhanced capabilities for portraying multispectral data abstractions within a natural three-dimensional stereoscopic immersive display system.
Multispectral Data Acquisition
Multispectral imaging systems acquire full spectral data at each pixel of an image. Thus, a
three-dimensional data cube is built with the axes being x, y and wavelength (X). The
multispectral data acquisition system used at OSER was custom built by OKSI (Torrance, CA). It consists of a TE-cooled, blue enhanced Silicon CCD array, a liquid crystal tunable
filter (LCTF), camera optics, and a laptop computer for control and storage. Either a
visible LCTF (400nm —720nm)or a near-infrared LCTF (600nm —lO5Onm)may be fitted
on the camera system. The image acquisition time depends on the integration time at each wavelength band, but is generally on the order of one minute. An image cube is 512x 512 x 33 (39) for the visible (NIR) and occupies approximately 17MB (20MB).
Data Management
The massive amounts of data generated by the enhanced sensor systems must be managed and stored in a manner that preserves the quality and unique identifying characteristics of each data set. The meta data describing the sensor data must include machine calibration
Figure 1 . ComponentDiagram of a multispectral medical data acquisition, analysis, and Visualization system.
record of data reduction and/or computational modification to the original data. This information is critical to ensuring data quality is maintained by preserving the chain of
events and processes that result in the analytical or visualization product being evaluated by the clinician.
The OSER system will support the research effort and therefore data provenance is critical
when evaluating data anomalies and determination of the efficacy of computational
algorithms. The data maintained by the OSER data management system will be invaluable for evaluating new diagnostic tolls as techniques are developed by the research team.
Also included in the data structure will be a complete set of patient records. This
information will be available to the analyst and diagnostician during computational analysis and also within the virtual environment to provide an effective and completevirtual diagnostic space.
The availability of this information is critical to the proper analysis of factors that may contribute to the variances in the sensing system. Changes in tissue response due to the
physiological state of the patient due must be available to the development team.
Data Reduction
Data reduction transforms raw multispectral sensor derived data into more useful forms. The techniques used during transformation will be preserved along with the resulting data set to ensure data provenance is preserved for quality control. The data reduction products will include data transformation into industry standard formats for data analysis, image visualization, and three-dimension virtual reconstruction.
Data Analysis
The data analysis component will consist of integrated commercial-off-the-shelf (COTS), open-source, and custom software to be used for computational analysis. The analytical
component will be tightly coupled with the visualization system to allow interactive
stereoscopic visualization. This capability will be available both at desktop workstation and with the VT-CAVE immersive environment.
The data analysis toolkit will include both COTS and custom computer applications.
Appropriate software applications used by the remote sensing industry will be applied (and customized as necessary) to support the multispectral diagnostic development effort. It is
expected that research into the efficacy of applying these reliable remote sensing
techniques to medical multispectral data is a promising research area.
Visualization
Data visualization is the science of turning sets of complex data into high-density visual information and understanding. Current and emerging technologies allow the generation of
virtual environments (VEs) that provide access to traditional data visualization analysis techniques and the ability to combine these capabilities with scaled digital models, video, and other multimedia within an interactive immersive, stereographic environment.
Advanced visualization has become a mainstream tool in applications where complex data
sets need to be studied and manipulated. The ability to provide human-scale display,
immersion, visual databases, spatial integration, and collaboration solutions will have a
significant impact on clinical and diagnostic procedures. The technology provides
immediate sensation of the spatial relationships between data. Data can be portrayed using
techniques that can either abstract complex data into recognizable patterns or portray
information in a natural and therefore more understandable context.
The use of VEs for data fusion is an area of active research. The technology currently
provides capabilities for:
.
interactivework environments with access to traditional data analysis techniques,S any
combination of spatially related data (or abstractions of the data),
computational products, or data classifications,
.
3Ddata models from other sources,.
3Dmodels produced by voxelation of image slices.
video,S images,
.
audioand 3D audio,.
scalemodels of environments, structures, sensors, and devices,.
interactivityat any scale or from any viewpoint,.
textuallabels, visual, and audio information that would help investigate,
understand, and communicate a medical investigation,
.
interactivedevices that can be developed specifically to aid in the dissection ofdiscrete data images,
•
interactive tools to manipulate and navigate the VE scene graph, and,•
interactive collaborative work sessions with remote sites.glasses. As a viewer wearing a position sensor moves within its display boundaries, the
correct perspective and stereo projections of the environment are updated by a
supercomputer, and the images move with and surround the viewer. Hence stereo
projections create 3D images that appear to have a presence both inside and outside the projection-room continuously. To the viewer with stereo glasses the projection screensbecome transparent and the 3D image space appears to extend to infinity
Specifically, the CAVE(tm) is a theater lOxlOx9 feet, made up of three rear-projection screens for the front, right and left walls and a down-projection screen for the floor. Electrohome Marquis 8000 projectors throw full-color workstation fields (1024x768
stereo) at 96 Hz onto the screens, giving approximately 2,000 linear pixel resolution to the
surrounding composite image. Computer-controlled audio provides a sonification
capability to multiple speakers. A user's head and hand are tracked with Ascension tethered electro magnetic sensors. Stereographics' LCD stereo shutter glasses are used to separate the alternate fields going to the eyes. A Silicon Graphics Power Onyx with three Infinite
Reality Engines is used to create the imagery that is projected onto the walls and floor.
(http://www.sv.vt.edu/future/vt-cave/whatis/)
Conclusion
The goal of a multispectral data visualization system is to provide enhanced diagnosis
capabilities for use by the medical practitioner. The system consists of five components: 1. multispectral data acquisition; 2.datamanagement; 3. data reduction; 4. data analysis; and 5. stereoscopicvisualization. The full spectra imaging system is combined with the data
analysis and interactive visualization to provide means for evaluating sensor response,
computational alogorithms, and efficacy of the portrayal techniques as a diagnostic tool. The data management component provides a platform for storage of raw and computational
data and the resulting visualization products. This component also provides the patient information necessary to a comprehensive diagnostic space. Finally the data reduction component provides the required procedures for standardizing and reducing the data acquired from the sensor systems. The stereoscopic workstations and the VT-CAVE, a
multi-person, room-sized, high-resolution, 3D video and audio environment, will be used for medical data visualizations and research into diagnostic procedures.
In conjuction with this data visualization work, we investigate an intriguing method for
analyzing hyperspectral data in talk 4259-10, "Cellular Automata for the Analysis of
Biomedical Hyperspectral Images", later in this session. Here, cellular automata are used
to rapidly scan hyperspectral images and quantify the extent of conditions of medical interest. This technique can lead to a large reduction in the computational time spent
