6.1! Summary and Conclusions
A new volume rendering engine was developed to support multiple platforms through a unified interface. This provides developers with a global volume rendering solution for deploying applications on desktops, laptops, immersive clusters and mobile devices. Before a multi-platform engine could be created, an investigation of the complexities and challenges of performing volume rendering on each platform was required along with a common architecture for performing the core volume rendering. This led to the development of three unique sandbox applications, each of which provided major contributions to the final volume rendering engine.
The desktop sandbox application was developed to build the core volume rendering algorithms for the engine such as resampling, coloring, shading and compositing. The resulting application included all of the core volume rendering algorithms as well as some unique features. Real-time windowing was built directly into fragment shader uniforms to dynamically modify the opacity transfer function. The GPU compositing algorithm was implemented using only the OpenGL specification and not any specific extensions allowing the library to render on any commodity graphics card. Finally, a convex clipping plane algorithm was designed to allow any number of clipping planes to be used to clip the volume in all directions.
The immersive sandbox application was built directly on top of the core volume rendering logic developed in the desktop application. The main focus of the immersive investigation was whether the core volume rendering logic could be
extrapolated to and immersive environment. Based on the current research reviewed, the resulting immersive application is the first application capable of performing GPU-based volume raycasting in large immersive clustered environments. Additionally, the core volume rendering logic did not have to be customized at all to accommodate the immersive platform.
The mobile sandbox application began as an investigation into the capabilities of today’s mobile device hardware. The initial results proved that raycasting cannot yet be performed in real-time due to the lack of 3D texture support. This shortcoming led to the development of an orthogonal texture slicing implementation. To create an interactive volume rendering solution, many performance enhancing features were built into the mobile application, each of which could be utilized by both rendering algorithms in the engine. These features include a method for dynamically modifying the render resolution, an incremental rendering loop for canceling high resolution renders, a shader-based clipping algorithm for OpenGL ES 2.0 and an internal backface culling algorithm to perform depth sorted rendering with alpha blending.
The completion of the sandbox applications verified that the common architecture used could support multi-platform volume rendering. This led to the development of VIPRE, or the Volume Image Processing and Rendering Engine. VIPRE contains the following features:
• Two volume rendering algorithms: raycasting and orthogonal texture slicing • Three rendering modes: composite, MIP and MinIP
• Four preset opacity transfer functions and eight color transfers functions • Real-time windowing using a dynamic shader-based opacity controls
• Bilinear and trilinear interpolation for 2D and 3D fragment shader textures • Custom CPU and GPU-based clipping algorithms
• Accurate depth sorted rendering with alpha blending
• Dynamic render quality modification using multiple render passes • Early ray termination and empty space skipping
• Phong illumination supporting multiple gradient operators • Multi-pass rendering for backface depth rasterization
• Support for all commodity desktop graphics cards in addition to iOS devices VIPRE was designed from the beginning to provide a unified solution for performing volume rendering on multiple platforms. This produced a robust volume rendering core which is able to be extended to support more complex forms of volume rendering. To help enable the extension of VIPRE, simplified versions of the sandbox applications as well as a robust set of documentation are included in the library to provide a cohesive starting point for novice and intermediate developers. Additionally, VIPRE is going to be released under licensing terms to allow it to be used in both the academic and commercial communities. The intention of this is to provide researchers and developers the ability to create new inventive methods of interaction with volumetric data without having to build their own volume renderer. Today’s technology is ever advancing, and VIPRE is an attempt to lower the barrier to entry for volume rendering to usher in a new generation of volume rendering applications. Competition fosters innovation, and by making volume rendering more accessible to researchers and developers, everyone can benefit.
6.2! Future Work
For the future development of VIPRE, there are several areas of focus. The first focus area is surgical planning. Surgical planning requires inserting instruments into the volume. VIPRE currently only supports proper depth rendering for geometry located outside of the volume. To render geometry inside the volume properly, an additional render pass needs to be implemented to build a depth map of the internal volume geometry. This depth map can then be integrated into the compositing algorithm in the main rendering pass to stop the ray traversal at the proper location.
Another area of focus is segmentation. By integrating a segmentation library into VIPRE, new methods of interaction for training segmentation routines could be studied in different platforms such as immersive clustered displays or mobile devices. Additionally, VIPRE will be extended to support multiple volumes and independent volume clipping to help visualize the internal structures of a segmented dataset such as a tumor. By reusing textures generated by previous render passes, more interactive user interfaces could be designed for immersive applications.
6.3! Acknowledgements
I would like to take this opportunity to thank everyone involved with the work presented in this dissertation as well as those who supported me throughout my graduate career. First and foremost, I would like to thank my fiancé, Rachael Gonzales. Without you I never would have had the strength to pull through the difficult times. I would also like to thank my family for their love, support and understanding through all the years.
Next I would like to thank my advisor Dr. Eliot Winer. Not only did your guidance help make me the person I am today, but without you I don’t know if I would have ever attended graduate school.
To the faculty and staff of the Virtual Reality Applications Center at Iowa State University, thank you for all the assistance you have given me and for making my graduate school experience as meaningful and worthwhile as possible. It has truly been a wonderful experience that I will never forget.
And last but not least, my research colleagues Dr. Eric Foo and Brandon Newendorp, your time and input throughout many design discussions and late night debugging sessions was essential to the development of VIPRE.
BIBLIOGRAPHY
1.! Kaufman, A. and K. Mueller, Overview of volume rendering. Imaging. Vol. d. 2005: Elsevier.
2.! Goodenough, D. and K. Weaver, Overview of Computed Tomography. Nuclear Science, IEEE Transactions on, 1979. 26: p. 1661-1667. 3.! Bottomley, P.A., Nuclear Magnetic Resonance: Beyond Physical
Imaging1983: IEEE Spectrum. 32-38.
4.! Lichtenbelt, B., et al., Introduction to volume rendering1998, Upper Saddle River, NJ: Prentice Hall PTR Upper Saddle River, NJ. 103-120.
5.! Elboth, T. and A. Helgeland, Hurricane visualization using anisotropic diffusion and volume rendering. 5th Annual Gathering on High Performance Computing in Norway, 2005: p. 3-4.
6.! Robertson, D., Virtual frog dissection: interactive 3D graphics via the Web. Computer Networks and ISDN Systems, 1995. 28: p. 155-160.
7.! Jansen, R.J., et al., High-resolution spiral computed tomography with
multiplanar reformatting, 3D surface- and volume rendering: a non-destructive method to visualize ancient Egyptian mummification techniques.
Computerized Medical Imaging and Graphics, 2002. 26: p. 211-216.
8.! Ketcham, R., Acquisition, optimization and interpretation of X-ray computed tomographic imagery: applications to the geosciences. Computers &
Geosciences, 2001. 27: p. 381-400.
9.! Seo, K.H. and J.F. Frank, Attachment of Escherichia coli O157:H7 to lettuce leaf surface and bacterial viability in response to chlorine treatment as demonstrated by using confocal scanning laser microscopy. Journal of food protection, 1999. 62: p. 3-9.
10.! Krüeger, A., et al., Sinus endoscopy--application of advanced GPU volume rendering for virtual endoscopy. IEEE transactions on visualization and computer graphics, 2008. 14: p. 1491-8.
11.! Patel, D., et al., A virtual reality solution for evaluation of radiotherapy plans. Radiotherapy and oncology : journal of the European Society for Therapeutic Radiology and Oncology, 2007. 82: p. 218-21.
12.! Sande, S., 1,800 iPads on the way to Ottawa Hospital, in The Unofficial Apple Weblog, 2011.
13.! Washuk, B., 5 kindergarten teachers to pilot Auburn iPad 2 program, in Sun Journal, 2011: Lewiston, Maine.
14.! Shreiner, D., OpenGL Programming Guide 7th Edition. The KIPS TransactionsPartA. Vol. 16A. 2009: Addison-Wesley. 1019 Pages. 15.! Pham, D.L., C. Xu, and J.L. Prince, Current Methods in Medical Image
Segmentation. Annual review of biomedical engineering, 2000. 2: p. 315-337. 16.! McInerney, T. and D. Terzopoulos, A Survey in Deformable Models in Medical
Analysis. Medical image analysis, 1996. 1: p. 91-108.
17.! Owens, J.D., et al. A Survey of General-Purpose Computation on Graphics Hardware. in Computer graphics forum. 2007. Wiley Online Library.
18.! Levoy, M., Display of surfaces from volume data. Computer Graphics and Applications, IEEE, 1988. 8: p. 29-37.
19.! Pommert, A., et al. Surface shading in tomographic volume visualization: A comparative study. in Visualization in Biomedical Computing, 1990.,
Proceedings of the First Conference on. 1990. IEEE.
20.! Moller, T., et al. Classification and local error estimation of interpolation and derivative filters for volume rendering. in Proceedings of 1996 Symposium on Volume Visualization. 1996. Acm.
21.! He, T. and A.E. Kaufman. Virtual Input Devices for 3D Systems. in
Visualization, 1993. Visualization'93, Proceedings., IEEE Conference on. 1993. IEEE.
22.! Keys, R., Cubic Convolution Interpolation for Digital Image Processing.
Acoustics, Speech and Signal Processing, IEEE Transactions on, 1981. 29: p. 1153-1160.
23.! Drebin, R.A., L. Carpenter, and P. Hanrahan. Volume Rendering. in ACM Siggraph Computer Graphics. 1988. ACM.
24.! He, T., et al., Generation of Transfer Functions with Stochastic Search Techniques. New York, 1996.
25.! Phong, B.T., Illumination for computer generated pictures. Communications of the ACM, 1975. 18: p. 317.
26.! Gouraud, H., Continuous shading of curved surfaces. IEEE transactions on computers, 1971. C-20: p. 623-629.
27.! Blinn, J.F., Light Reflection Functions for Simulation of Clouds and Dusty Surfaces. ACM SIGGRAPH Computer Graphics, 1982. 16: p. 21-29. 28.! Porter, T. and T. Duff, Compositing digital images. ACM SIGGRAPH
Computer Graphics, 1984. 18: p. 253-259.
29.! Sabella, P., A Rendering Algorithm for Visualizing 3D Scalar Fields. ACM SIGGRAPH computer graphics, 1988. 22: p. 51-58.
30.! Upson, C., Visible Volume Rendering. Computer Graphics, 1988. 22: p. 59-64.
31.! Lorensen, W.E. and H.E. Cline, Marching Cubes: A High Resolution 3D
Surface Construction Algorithm. ACM Siggraph Computer Graphics, 1987. 21: p. 163-169.
32.! Westover, L., Footprint evaluation for volume rendering. ACM SIGGRAPH Computer Graphics, 1990. 24: p. 367-376.
33.! Botsch, M. and L. Kobbelt, High-quality point-based rendering on modern GPUs. 11th Pacific Conference onComputer Graphics and Applications, 2003. Proceedings., 2003: p. 335-343.
34.! Zwicker, M. and H. Pfister, Hardware-Accelerated Adaptive EWA Volume Splatting. IEEE Visualization 2004, 2004: p. 67-74.
35.! Neophytou, N. and K. Mueller. GPU accelerated image aligned splatting. in Fourth International Workshop on Volume Graphics, 2005. 2005. IEEE.
36.! Pfister, H., et al. The VolumePro real-time ray-casting system. in Proceedings of the 26th annual conference on Computer graphics and interactive
techniques. 1999. New York, New York, USA: ACM Press/Addison-Wesley Publishing Co.
37.! Wu, Y., et al. Shear-image order ray casting volume rendering. in Proceedings of the 2003 symposium on Interactive 3D graphics. 2003. New York, New York, USA: ACM.
38.! Rezk-Salama, C., et al. Interactive volume on standard PC graphics hardware using multi-textures and multi-stage rasterization. in Proceedings of the ACM SIGGRAPH/EUROGRAPHICS workshop on Graphics hardware. 2000. New York, New York, USA: ACM.
39.! Engel, K., M. Kraus, and T. Ertl. High-quality pre-integrated volume rendering using hardware-accelerated pixel shading. in Proceedings of the ACM
SIGGRAPH/EUROGRAPHICS workshop on Graphics hardware - HWWS '01. 2001. New York, New York, USA: ACM Press.
40.! Roettger, S., et al. Smart hardware-accelerated volume rendering. in Proceedings of the symposium on Data visualisation 2003. 2003. Eurographics Association.
41.! Hsu, W.M. Segmented Ray Casting for Data Parallel Volume Rendering. in Proceedings of the 1993 symposium on Parallel rendering. 1993. ACM.
42.! Ma, K.L., Parallel volume ray-casting for unstructured-grid data on distributed- memory architectures, 1995.
43.! Ma, K.L. and T.W. Crockett, A Scalable Parallel Cell-Projection Volume Rendering Algorithm for Three-Dimensional Unstructured Data. prs, 1997: p. 95.
44.! Heng, Y. and L. Gu. GPU-based Volume Rendering for Medical Image
Visualization. in Conference proceedings : ... Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE Engineering in Medicine and Biology Society. Conference. 2005. Ieee.
45.! Hadwiger, M., et al. Advanced Illumination Techniques for GPU-Based Volume Raycasting. in ACM SIGGRAPH 2009 Courses. 2009. ACM. 46.! Whitted, T., An improved illumination model for shaded display.
Communications of the ACM, 1980. 23: p. 343-349.
47.! Levoy, M., Efficient ray tracing of volume data. ACM Transactions on Graphics, 1990. 9: p. 245-261.
48.! Weiler, M., et al., Hardware-based ray casting for tetrahedral meshes. IEEE Transactions on Ultrasonics, Ferroelectrics and Frequency Control, 2003: p. 333-340.
49.! Meagher, D., Geometric modeling using octree encoding. Computer Graphics and Image Processing, 1982. 19: p. 129-147.
50.! Junyaprasert, K., et al., Interactive volume rendering for virtual colonoscopy. Proceedings. Visualization '97 (Cat. No. 97CB36155), 1997: p. 433-436,. 51.! Wan, M., et al. Volume Rendering based interactive navigation within the
human colon. in Visualization'99. Proceedings. 1999. IEEE.
52.! Wan, M., A. Sadiq, and A. Kaufman. Fast and reliable space leaping for interactive volume rendering. in Proceedings of the conference on Visualization'02. 2002. IEEE Computer Society.
53.! Kruger, J. and R. Westermann, Acceleration techniques for GPU-based volume rendering. IEEE Transactions on Ultrasonics, Ferroelectrics and Frequency Control, 2003: p. 287-292.
54.! Lee, T.-H., et al., Fast perspective volume ray casting method using GPU- based acceleration techniques for translucency rendering in 3D endoluminal CT colonography. Computers in biology and medicine, 2009. 39: p. 657-66. 55.! Mueller, K. and A. Kaufman, Empty space skipping and occlusion clipping for
texture-based volume rendering. IEEE Transactions on Ultrasonics, Ferroelectrics and Frequency Control, 2003. 4400: p. 317-324.
56.! Li, W. and A. Kaufman, Texture partitioning and packing for accelerating texture-based volume rendering. Proceedings: Graphics Interface 2003: Halifax, Nova Scotia, 11-13 June 2003, 2003: p. 81.
57.! Fuchs, H., Z.M. Kedem, and B.F. Naylor, On visible surface generation by a priori tree structures. ACM Siggraph Computer Graphics, 1980. 14: p. 124-133.
58.! Bhaniramka, P. and Y. Demange. OpenGL Volumizer: A toolkit for high quality volume rendering of large data sets. in Proceedings of the 2002 IEEE
symposium on Volume visualization and graphics. 2002. IEEE Press. 59.! Ruijters, D. and A. Vilanova, Optimizing GPU volume rendering. Journal of
WSCG, 2006. 14: p. 9-16.
60.! Volz, W.R., Gigabyte Volume Viewing Using Split Software/Hardware Interpolation. 2000 IEEE Symposium on Volume Visualization (VV 2000), 2000. Vi: p. 15-22.
61.! Weiler, M., et al., Level-of-detail volume rendering via 3D textures.
Proceedings of the 2000 IEEE symposium on Volume visualization - VVS '00, 2000: p. 7-13.
62.! Grimm, S., et al., Memory efficient acceleration structures and techniques for CPU-based volume raycasting of large data. 2004 IEEE Symposium on Volume Visualization and Graphics, 2004. Vi: p. 1-8.
63.! Beyer, J., et al., High-quality multimodal volume rendering for preoperative planning of neurosurgical interventions. IEEE transactions on visualization and computer graphics, 2007. 13: p. 1696-703.
64.! LaMar, E., Multiresolution techniques for interactive texture-based volume visualization. Proceedings of SPIE, 2000. vi: p. 365-374.
65.! Boada, I., I. Navazo, and R. Scopigno, Multiresolution volume visualization with a texture-based octree. The Visual Computer, 2001. 17: p. 185-197. 66.! Thali, M.J., et al., Forensic microradiology: Micro-computed tomography (Micro-CT) and analysis of patterned injuries inside of bone. Journal of forensic sciences, 2003. 48: p. 1336-1342.
67.! Guthe, S., et al., Interactive rendering of large volume data sets. IEEE Visualization, 2002. VIS 2002., 2002. D: p. 53-60.
68.! Ljung, P., C. Lundström, and A. Ynnerman. Multiresolution interblock
interpolation in direct volume rendering. in Proceedings Eurographics/IEEE Symposium on Visualization 2006. 2006. Citeseer.
69.! Gobbetti, E., F. Marton, and J.A. Iglesias Guitián, A single-pass GPU ray casting framework for interactive out-of-core rendering of massive volumetric datasets. The Visual Computer, 2008. 24: p. 797-806.
70.! Crassin, C., et al. Gigavoxels: Ray-guided streaming for efficient and detailed voxel rendering. in Proceedings of the 2009 symposium on Interactive 3D graphics and games. 2009. ACM.
71.! Ljung, P., et al., Full body virtual autopsies using a state-of-the-art volume rendering pipeline. IEEE transactions on visualization and computer graphics, 2006. 12: p. 869-76.
72.! Hege, H., et al., Smooth mixed-resolution GPU volume rendering. vrvis.at, 2008.
73.! Guthe, S., Advanced techniques for high-quality multi-resolution volume rendering. Computers & Graphics, 2004. 28: p. 51-58.
74.! Nguyen, K.G. and D. Saupe, Rapid High Quality Compression of Volume Data for Visualization. Computer Graphics Forum, 2001. 20: p. 49-57.
75.! Ihm, I. and S. Park, Wavelet-Based 3D Compression Scheme for Interactive Visualization of Very Large Volume Data. Computer Graphics Forum, 1999.
18: p. 3-15.
76.! Kim, T. and Y. Shin. An Efficient Wavelet-Based compression method for volume rendering. in Computer Graphics and Applications, 1999.
Proceedings. Seventh Pacific Conference on. 1999. IEEE.
77.! Rodler, F.F., Wavelet based 3D compression with fast random access for very large volume data. Proceedings. Seventh Pacific Conference on Computer Graphics and Applications (Cat. No.PR00293), 1999. D: p. 108-117.
78.! Möller, T., et al. Employing Complex GPU Data Structures for the Interactive Visualization of Adaptive Mesh Refinement Data. in Volume Graphics 2006: Eurographics/IEEE VGTC workshop proceedings. 2006. Citeseer.
79.! Kraus, M. and T. Ertl. Adaptive texture maps. in Proceedings of the ACM SIGGRAPH/EUROGRAPHICS conference on Graphics hardware. 2002. Eurographics Association.
80.! Langer, M.S. and H.H. Bulthoff, Depth discrimination from shading under diffuse lighting. Perception, 2000. 29: p. 649.
81.! Sattler, M., et al., Exploitation of human shadow perception for fast shadow rendering. Proceedings of the 2nd symposium on Appied perception in graphics and visualization - APGV '05, 2005: p. 131.
82.! Carson, G.S., Introduction to the computer graphics reference model. ACM SIGGRAPH Computer Graphics, 1993. 27: p. 108-119.
83.! Behrens, U. and R. Ratering. Adding shadows to a texture-based volume renderer. in Proceedings of the 1998 IEEE symposium on Volume
visualization. 1998. ACM.
84.! Kniss, J., et al., Interactive translucent volume rendering and procedural modeling. IEEE Visualization, 2002. VIS 2002., 2002. di: p. 109-116.
85.! Hernell, F., P. Ljung, and A. Ynnerman, Interactive Global Light Propagation in Direct Volume Rendering using Local Piecewise Integration. Volume
86.! Wald, I., et al., Faster isosurface ray tracing using implicit KD-trees. IEEE transactions on visualization and computer graphics, 2005. 11: p. 562-72. 87.! Marmitt, G., H. Friedrich, and P. Slusallek, Interactive volume rendering with
ray tracing. Eurographics State of the Art Reports, 2006: p. 115-136.
88.! Williams, L. Casting curved shadows on curved surfaces. in ACM SIGGRAPH Computer Graphics. 1978. ACM.
89.! Reeves, W.T., D.H. Salesin, and R.L. Cook. Rendering antialiased shadows with depth maps. in ACM SIGGRAPH Computer Graphics. 1987. ACM.
90.! Kim, T.Y. and U. Neumann. Opacity shadow maps. in Proceedings of the 12th Eurographics Workshop on Rendering Techniques. 2001. Citeseer.
91.! Lokovic, T. and E. Veach. Deep shadow maps. in Proceedings of the 27th annual conference on Computer graphics and interactive techniques. 2000. New York, New York, USA: ACM Press/Addison-Wesley Publishing Co. 92.! Hadwiger, M., et al. GPU-accelerated deep shadow maps for direct volume
rendering. in Proceedings of the 21st ACM SIGGRAPH/EUROGRAPHICS symposium on Graphics hardware, September. 2006. Citeseer.
93.! Stewart, A.J. Vicinity shading for enhanced perception of volumetric data. in Proceedings of the 14th IEEE Visualization 2003 (VIS'03). 2003. IEEE Computer Society.
94.! Desgranges, P. and K. Engel, Fast ambient occlusion for direct volume rendering, in US Patent App. 11/455,627, 2006, Google Patents.
95.! Hernell, F., P. Ljung, and A. Ynnerman. Efficient ambient and emissive tissue illumination using local occlusion in multiresolution volume rendering. in Eurographics/IEEE-VGTC symposium on volume graphics. 2007.
96.! Ropinski, T., et al. Interactive volume rendering with dynamic ambient occlusion and color bleeding. in Computer Graphics Forum. 2008. Wiley Online Library.
97.! Khanduja, G. and B.B. Karki. A systematic approach to multiple datasets visualization of scalar volume data. in International Conference on Computer Graphics Theory and Applications, (GRAPP'06). 2006. Citeseer.
98.! Dietrich, C.A., et al. Real-time interactive visualization and manipulation of the volumetric data using GPU-based methods. in SPIE medical imaging. 2004. 99.! Mcinerney, T. and S. Broughton, HingeSlicer : Interactive Exploration of
Volume Images Using Extended 3D Slice Plane Widgets. Science, 2006. Im: p. 171-178.
100.! Wang, S.W. and A.E. Kaufman, Volume sculpting. Proceedings of the 1995 symposium on Interactive 3D graphics - SI3D '95, 1995: p. 151-ff.
101.! Weiskopf, D., K. Engel, and T. Ertl, Interactive clipping techniques for texture- based volume visualization and volume shading. IEEE Transactions on Visualization and Computer Graphics, 2003. 9: p. 298-312.
102.! Konrad-Verse, O., B. Preim, and A. Littmann. Virtual resection with a deformable cutting plane. in Proceedings of simulation und visualisierung. 2004. Citeseer.
103.! Stegmaier, S., et al., A simple and flexible volume rendering framework for graphics-hardware-based raycasting. Fourth International Workshop on Volume Graphics, 2005., 2005: p. 187-241.
104.! Brecheisen, R., et al., Flexible GPU-based multi-volume ray-casting. Vision, modeling, and visualization 2008: proceedings, October 8-10, 2008,
Konstanz, Germany, 2008: p. 303.
105.! Niederauer, C., et al., Non-invasive interactive visualization of dynamic architectural environments. Proceedings of the 2003 symposium on Interactive 3D graphics - SI3D '03, 2003: p. 55.
106.! Chen, M., et al. Spatial transfer functions: a unified approach to specifying deformation in volume modeling and animation. in Proceedings of the 2003 Eurographics/IEEE TVCG Workshop on Volume graphics. 2003. ACM. 107.! Islam, S., et al., Spatial and temporal splitting of scalar fields in volume
graphics. Image (Rochester, N.Y.), 2004. i: p. 87-94.
108.! McGuffin, M.J., L. Tancau, and R. Balakrishnan. Using deformations for browsing volumetric data. in Proceedings of the 14th IEEE Visualization 2003