Significant Facet Retrieval for Real-Time 3D Sound Rendering in Complex
Virtual Environments
Chris Joslin
MIRALab – University of Geneva
24 Rue du General Dufour
CH1211, Geneva-4, Switzerland
+41 22 705 7768
Nadia Magnenat-Thalmann
MIRALab – University of Geneva
24 Rue du General Dufour
CH1211, Geneva-4, Switzerland
+41 22 705 7769
ABSTRACT
Sound rendering requires that many different aspects are considered, especially when rendering a real-time virtual environment. In sound rendering, much the same as for graphics, one of the major influencing factors is the number of reflective polygons in a scene and due to the increase in the ability of most common graphics cards; this number can now be very high, even when designers create the optimum scene using other optimizing tools such as Polygon Cruncher or Rational Reducer. In addition, the use of programs such as LightscapeTM [20], which is used to provide realistic lighting, by using per vertex shading, increases the number of polygons in a scene by several factors. Therefore a strong, pre-processing method is proposed that dramatically reduces the number of polygons in the scene to a suitable level for real-time sound rendering. The method can also be combined with other methods (e.g. scene partitioning) for even lower CPU usage.
Categories and Subject Descriptors
I.3.6 [Computer Graphics] Three Dimensional Graphics and Realism – virtual reality
General Terms
AlgorithmsKeywords
Bounding-Box, Sound Rendering, Scene Segmentation, Virtual Environments.
1.
INTRODUCTION
3D Sound Rendering is becoming more and more common in so many areas of virtual reality and games; however dynamic systems are not always considered, as they require a lot of computational power to cover all the necessary aspects in order to
correctly produce an accurate rendering of sound; the following computations are generally required:
• Transmission – This takes into account the absorption of the sound by the air, energy distribution, and also how sound is affected by surfaces between the source and the listener either by direct path, or via any of the reflections.
• Reflections – This takes account of the sound propagation around an environment by multiple early reflections possible due to reflective surfaces.
• Reverberation – Reverberations are later reflections that are not uniquely distinguishable; reverberation is generally only possible in closed environments (such as a room).
• Head Related Transfer Function (HRTF) – This computation calculates the sound propagation/interference changes due to the listener’s head and the ear’s pinnae.
• Sound Diffraction – This deals with how a sound is diffracted when striking either a very irregular surface, or edges of surfaces.
The main focus of this paper is the consideration for the computation of the initial reflections, and also to a lesser extent the later reverberation effect, which is normally calculated using the facets and facet structure of an environment; this can be done using a variety of methods:
• Ray Tracing [19, 29] is one method, related to graphics, which basically computes all the paths between the source and the destination. This is done by generating all the rays emanating from the source and continuing through the environment, reflecting off objects, until it reaches the destination (in graphics the screen, in sound the user).
• Image Source methods [2, 5] use mirrored points on each facet in order to determine the paths from source to destination. This method has a high complexity order, defined as No + N, where N is the number of facets, and O is the order of reflections; however it does
present the most robust method in computing reflections.
• Beam Tracing methods [15, 7, 10] classify propagation paths from a source by recursively tracing pyramidal beams (i.e. sets of rays) through the environment. These generally rely on Binary Space Partitioning (BSP) [12] in order to progress through the polygons using a front to back order, such that no polygon is considered until all others (at least partially occluded) have been considered
Overall the key aspect is to reduce the heavy calculation of reflection and reverberation to a real-time value whereby the other computations present can also be done in real-time (i.e. as sound is part of a bigger system requiring rendering, navigation, virtual human animation etc and therefore allowances for this must be taken into account). This is mainly achieved by the use of a reduced number of facets in the scene necessary for a sound-rendering engine.
In this paper we first consider other approaches for achieving real-time sound rendering and then look more specifically at models used for reducing the overall number of polygons in the scene that could be applied to sound rendering, and their advantages and disadvantages. In Section 2 we layout step-by-step our approach to the problem, and show the variables that can be changed to increase or decrease the resolution; which also relate to the sound itself. Section 3 demonstrates the results of the described methodology, and also exemplifies differences when changing variables for the resolution. Section 4 concludes the paper with concluding remarks and future work.
1.1
Dynamic Real-Time Models
Whilst many methods exist for the off-line computation of sound dynamics in complex spaces such as those from Naylor et al [22], many systems are turning towards the complex, yet realistic simulation of 3D spatial audio in real-time; especially in the context of game simulation, and collaborative virtual environments. Several models have been proposed to allow for the dynamic calculation of part and even all the aforementioned computations. Funkhouser et al [13] propose a “Beam Tracing” method for interactive virtual environments; this works by recursively tracing pyramidal beams through the environment. The system builds a spatial subdivision to accelerate the beam tracing and is done by using a Binary Space Partition (or BSP) [12]; however it is not able to cope with interactively changing scenes without incurring large overhead. In addition, since cones and triangles represent approximations of the propagating sound field, overlaps and the regularly occurring missing reflections must be compensated for statistically. Another method proposed by Savioja et al [26] includes the rendering of room acoustics by integrating the impulse response of the room into the HRTF calculation; this method works well for simple room designs but not for complex dynamic environments, in addition this particular method is a design for a Digital Signal Processing (DSP) chip, and although it could be applied to a normal environment it has not been verified. The system from Foster et al [11] only calculates the first and second order reflections for simple volume environments. Hein et al [16] use statistical methods in order to generate diffraction and reverberation, which obviously suffers from complexity limitations. Whilst Herder [17] proposes a clustering method to optimize the computation in spatial sound
environments, but this is based on the grouping of multiple sources and does not take into account the scene complexity.
1.2
Polygon-Reduction Methods
Many methods for reducing the number of essential polygons to a reasonable number are different when considering the visual/interactive aspects and sound rendering. When considering collision detection, several methods consider only a local set of polygons [31]; this already significantly reduces the overall number of active polygons, although they can also ignore significant polygons crucial for sound rendering. Eckstein et al [9] consider using bounding volumes to dramatically reduce the number of collision computations necessary.
Cohen et al [6] use a two-level approach, the first step segmenting a scene using Voronoi regions [24]. A Voronoi region associated with a feature (a vertex, an edge, or a face) is a set of points closer to that feature than to any others. Ehmann et al [10] define a unified approach to perform a set of proximity queries seamlessly for general, rigid polyhedral objects with boundary representations which are orientable 2-manifolds. The proposed method involves a hierarchical data structure built upon a surface decomposition of the models.
Looking more towards other methods for reducing the number of polygons in a scene either by optimization or approximation, Polygon Cruncher [23] and Rational Reducer [25] are commercial plugins for Discreet’s 3D Studio Max [8] and Alias|Wavefront’s Maya [1] respectively; they work by determining adjacent polygons and the angular separation between them, if the angle is considered small enough then the vertex points are combined to form one single vertex. This is performed as a recursive search through the polygonal model’s structure, so that polygons can be recursively combined should their neighbor have a small angular separation (although steps are used to avoid flattening a sphere). Both of these methods are generally off-line solutions, and also reduce to a specific factor (i.e. towards a preset angular separation). Barequet et al introduce a preprocessing method using sub-tree bounding box hierarchies, which this works on general objects already placed in a hierachical structure, it is not proven to work for large single polygonal sets; in addition it is not really possible to control the facet acceptance process based on a specific size.
1.3
Main Contribution
In this paper, we present a method that reduces the number of significant polygons to a level more suitable for spatial sound rendering, whilst maintaining a control method that allows the system to determine the sound wavelength propagation. We make emphasis on environments that have not been pre-conditioned by scene designers other than perhaps already some optimization of polygonal structures. The only necessary condition, much like any interactive environment, is that objects that can be moved, or interacted with, are defined under a separate node hierarchy.
2.
METHODOLOGY
2.1
Introduction
As specified in the introduction, sound rendering involves several key elements; however not all are directly related to the size of the virtual environment. The computation of the
HRTF
is not directly linked to the number of facets in the room; reverberation is partially linked to the number of facets but does not require such a heavy calculation. The calculation for the reflection isdirectly linked to the number of facets in a scene, and more importantly it has a significant impact depending on the number of reflections that are computed.
As each path needs to be test for occlusions and reflection for each facet the number of function calls (existing for the same number of cycles) is approximated to the following equation:
∑
= ==
i R i iN
1χ
(1)Where N is the number of polygons in the scene, and R is the number of orders that is required (e.g. an order of 2 is a reflection off two objects). For example, a scene with 500 usable facets, and a third order computation, equates to a value for χ of 125 x 106 function calls, which is quite large and not possible to push through as a real-time solution (due to the fact that this is a function call and not an explicit clock cycle, and in addition there are other functions running concurrently).
Therefore the overall purpose of this scheme is to reduce the overall number of facets and leave only the facets significant for the reflection computation in rendering sound. Using traditional methods for finding bounding boxes for a scene, in order to reduce the complexity, requires that each facet in an Indexed Face Set (IFS) has its own node. However many scenes are designed without this kind of large separation, or hierarchy, and in fact in many cases scenes consist of one IFS branching from just one group node. This is because the objectives of a scene designer are often towards functionality and visual effect, and not towards real-time sound rendering; it is also often the case where the two do not even mix. Figure 1a and 1b show the required facet segmentation of a scene, and the scene resulting from a single bounding box pass when all the facets are on one node.
Figure 1a. Required Scene Bounding Box
Figure 1b. Resulting from IFS with only one root node
In Figure 1b, the resulting bounding box would provide an approximation that would be useful for picking, for example, but it would not allow for the correct rendering of sound due to the enclosure caused by the bounding box; i.e. the sound would not be able to propagate properly.
Therefore a method is required that segments the scene into sections, whilst respecting both significant gaps and holes, and also small objects/facets that do not provide much change with relation to the sound rendering. This entire method is done as a pre-processing step in order to output a set of significant bounding boxes that can then be attached to the node; this means that if the node is transformed, the bounding boxes will be transformed also and therefore the there is no need to go through this processing step unless a new IFS is added to the scene.
2.2
Initial Scene Requirements
Virtual environments are becoming more and more complex using a higher number of polygons as graphics cards are increasing in ability. Sound however does not need to have such a high resolution to present an accurate model, this is because the wavelength of sound is a very large 17m for low frequency sound (20Hz), and a moderate 17x10-3m for high frequency sounds (20KHz); speech generally being in the range of 3-4KHz (0.1m). This means that only a simple representation is required in order that the sound is rendered accurately.
As the sound is rendered directly from the scene-graph there are several nodes that are also not required. Therefore these can be discarded directly (these include color, each normal to a facet, and texture coordinates). Once the scene has been stripped of all this information, it can then be segmented.
2.3
Initial Scene Segmentation
Once all the additional scene information has been removed, the scene can be segmented. For each node, a bounding box is taken in order to determine the extremities of the geometry on the node (providing Xmin, Xmax, Ymin etc). Using these points, each dimension is segmented using the same fixed distance δ for each of the planes (this fixed distance is used in place of a percentage due to the fact that it is necessary to use a value relative to the average wavelength, or larger). The scene segmentation basically creates a set of grid lines, which are used to test for intersection (see Section 2.5); these grid segmentations can be seen in Figure 2 where δ was set to 1.2m.
Figure 2. Scene Segmentation
The value for δ can be adjusted to approximately the maximum wavelength desired. This does not mean that the frequency range will be limited, just that the reflection propagation of a room will be limited to a specific frequency range (and therefore when comparing to a normal room this will be an influential point).
2.4
Planar Categorization
Before anything else, each facet needs to be classified in order to determine the closest plane that it rests in. The number of planes that are used as a reference for matching can be set at as a variable in order to set a value for the resolution.
The minimum number of planes is three, whereby the angle between the planes is
π
/2 (i.e. planes XY, YZ, and XZ), the number of planes generated according to the angle separation is (360°/Angle Separation) – 1 (e.g. an angular separation of 22.5° results in 15 planes).In order to determine which plane a facet is closest to, the normal of the facet and each separating plane are compared using a dot product and the angle (if less than π/2, if larger then π/2 is subtracted – due to planes being orthogonal) that is the smallest angle is taken as being the closest match (and therefore the corresponding plane is used); this is demonstrated in Equation (2).
F f P p P f p f p
N
N
N
N
= = = −=
⋅
⋅
max, 0 min 1(
)
cos
θ
(2)Where Np is the normal to the plane P, and Nf is the normal to the
facet being tested, and θmin is the minimum angle of all the planes. Pmax represents the maximum number of planes in being used. This is then registered for each “Intersection Point” as it is found, which is then used when performing clustering as described in Section 2.6; this method is performed for the total number of facets in the scene, F, to determine assign their closest plane.
2.5
Intersection Points
Instead of working directly with facets, intersection points are used due to two key reasons:
• Extraction of Signification Facets – The main reason for the use of Intersection Points is actually part of a two-part process to determine significant facets in a scene.
• Reducing Clustering Computation – by using intersection points instead of facets, the number of facets compared in the clustering (described in Section 2.6) is significantly reduced.
Each facet is tested to determine which of the grid lines intersects it and the intersection point is recorded. This is done for all facets until a complete set of intersection points has been obtained. Duplicate points are discarded, i.e. points generated from intersections from gridlines from the three main planes (all facets are tested for intersection; however, two points may lie in the same position, but come from different planar facets; therefore these are retained in the points database). This point sorting, according to their plane, can be seen in Figure 3.
This results a number of groups, each group containing points lying in a specific plane.
2.6
Clustering of Intersection Points
Once the intersection points, for each plane, have been calculated they need to be clustered together to form significant groups. These groups represent the significant surfaces in the scene, whereby determining that several points are within close proximity to each other, they can be used to form the contents of facet group and thus by using a bounding box form a significant surface.
Figure 3. Planar Intersection Points
The key to defining separation between major surfaces (such as walls) and gaps (such as corridors) is the use of cubic clustering; this is done by searching for clustered points that only make up a cubic structure. By shifting down through the database of intersection points, as determined in Section 2.5 – for each plane, the distances between the point being tested and all following points (from that position in the database) are compared with a threshold value, β (which is set to 1.3m), and then grouped accordingly. If a point is not already in a group, it represents the new point in a group. If a point is already in a group and has close proximity to another the two groups are joined, all the points are relabeled to be with the primary group, and the secondary group is discarded. Once the search through the database has been completed all the points should be in a group consisting of one or more points. The cubic structure of the clustered points is verified
if a segmentation point cannot be accounted for in a cluster; therefore a new cluster is created; therefore as the cluster points are searched, if the points cannot continue a cubic structure, then a new cluster is started once the full dimensions have been explored.
Once all the points have been processed and a group reference allocated to each cluster group, a normal bounding box function [4] is then passed over the points that exist within each cluster. This produces a single bounding box for the cluster.
This kind of reduction produces an excellent approximation of surfaces that are set close together and basically clusters them into one bounding box object.
This threshold value β is always set to a value slightly greater than the segmentation distance δ (usually about 10%); this allows for two points to be set so that they may be clustered, whilst removing obscure/unimportant facets. It also closes gaps smaller than the threshold value (and hence the wavelength). Space represents a key element in the scene (such as a door), then this is added as a separate node (as it would be) and therefore treated as a separate bounding box, for more information see Section 2.8.
2.7
Bounding Box Reduction
Once all the bounding boxes have been found, they can then be processed again in order to reduce the number of overall facets. First, the bounding boxes are tested to determine if one side is less than a percentage (this is usually 10%, which removes relatively small boxes, without unnecessarily destroying them) of the two sides dimensions (i.e. each side is tested against the other to determine size ratio). Therefore if a bounding box is relatively flat then it can be reduced to a flat surface and therefore a box of 12 facets is reduced to a surface of 2 facets (taking the mean distances for the smaller lengths).
2.8
Scene-Graph Association
Due to the fact that the method progresses through the scene-graph node by node, each bounding box can be attached to a specific node. For instance, a 50-polygon door can be simplified into one bounding box, and when that door is opened the attached bounding box moves in line with the node; this removes the need to recompute the bounding box again.
This means that the sound-rendering engine only needs to access the bounding box nodes attached to the scene-graph. In addition, if a new object is added to the scene (for instance introduced as part of a collaborative environment session) as it is a separate node, only this new object is passed through the pre-processing step and the resulting bounding box attached to the scene-graph. The results are highly significant; a 50K polygon model can be reduced to 14 significant facets, in approximately 10 seconds. This reduces the overall complexity (using Equation 1) from a 2.5x109 function calls, to a more manageable 210 function calls. Obviously this brings the kind of sound rendering into the real-time domain.
3.
RESULTS
The significance of this kind of reduction, produced from a single node IFS, can be used sequentially on further nodes if necessary, but the strength of this method is that the nodal structure does not need to be predefined by a scene designer; especially as applications such as LightScapeTM are usually performed as part of the post processing step to the architectural design. Figure 4
shows the resulting 14 facets calculated from a 43K facet model, using a planar separation of π/2 radians.
Figure 4 – Resulting Reduced Facet Set (π/2 Planar Separation)
If a planar separation of π/4 is used, which results in a model consisting of 34 facets calculated from the same 43K facet model; this is shown in Figure 5.
Figure 5. Resulting Reduced Facet Set (π/4 Planar Separation)
As can be seen, the dome roof on top of the building is better defined, and therefore produces a more realistic sound rendering for this kind of structure; however it results in more facets. In addition the back wall, which has a convex shape to accommodate a door, is also truer to the real scene; providing segmentation into five defined bounding volumes, in place of three.
4.
CONCLUSION
As can be seen this method presents a rapid pre-processing method that only obtains the significant facets in complex environments using scene segmentation, 3D point clustering and bounding box regrouping. The method can be considered more significant for the reduction of overly complicated scenes, generated by widely used commercial programs such as LightScapeTM where each surface may have multiple facets in order to represent a shading effect from one or more light sources; whilst retaining the high levels of detail for rendering realistic and life-like scenes.
The method also allows quite fine-grain control over the overall gap and facet size in the environment, therefore it can be used to control the propagation of sound according to its frequency range. In addition, the resulting reduced facet set can still be used in conjunction with other real-time dynamic methods, such as scene partitioning and localization (based on the relative position of the source/destination); thereby reducing the overall CPU usage even more. In addition this method is completely scalable, meaning that as processing units become more capable in the future, this can be scaled up to account for it, by adding more planes. It is evident that this methodology can remove more facets than might be necessary when making a non-real-time computation of sound propagation in a virtual environment; however this method provides value adjustment of the reduction of the facet set according to a desired level.
It can be observed from that the resulting facets are approximately matched to the significant facets in the scene; however as future work the authors would like to compare the paths generated from the original scene and the ones generated from the optimized scene in order to obtain an error ratio. In addition, in order to provide a significant control mechanism (important in today’s autonomous world) the authors are pursuing methods to relate performance to planar segmentation and scene polygon density.
5.
ACKNOWLEDGEMENTS
This research has been funded through the European Project STAR (IST-2000-28764) by the Swiss Federal Office for Education and Science (OFES). The authors would also like to thank Sophie Jarlier, Derrick Sonamzi, and Siviwe Sisusa, for implementing this working into their applications.
6.
REFERENCES
[1] Alias|Wavefront: http://www.aliaswavefront.com
[2] Allen, J., Berkley, D. Image Method for Efficiently simulating small room acoustics, Journal Acoustical Society of America, Vol. 65, No. 4, 1979, pp. 943-951.
[3] Barequet, G., Chazelle, Guibas L., Mitchell J., Tal A., BOXTREE: Hierarchical Representation for Surfaces in 3D, Computer Graphics Forum, 1996.
[4] Barequet G., Har-Peled S., Efficiently Approximating the Minimum Bounding Box of a Point Set in Three Dimensions, Proceedings of 10th Annual ACM-SIAM Symposium on Discrete Algorithms, 1999, pp. 82-91.
[5] Borish J., Extension of the Image Model to arbitrary polyhedra, Journal Acoustical Society of America, Vol. 75, No. 6, 1984, pp. 1827-1836.
[6] Cohen J., Lin M., Manocha D., Ponamgi M., I-COLLIDE: An Interactive and Exact Collision Detection System for Large-Scale Environments, Symposium on Interactive 3D Graphics, 1995, pp. 189–196.
[7] Dadoun N., Kirkpatrick D., Walsh J., The Geometry of Beam Tracing, Proc. Computational Geometry, 1985, pp. 55-71.
[8] Discreet (Autodesk): http://www.discreet.com
[9] Eckstein J., Schomer E., Dynamic Collision Detection in Virtual Reality Applications, International Conference on Computer Graphics and Visualisation, EUROGRAPHICS and IFIP WG 5.10, 1999.
[10] Ehmann S. and Lin M., Accurate and Fast Proximity Queries Between Polyhedra Using Surface Decomposition, Computer Graphics Forum (Proc. of EUROGRAPHICS), 2001.
[11] Foster S., Wenzel E., Taylor R., Real-Time Synthesis of Complex Acoustic Environments, Proceedings of the IEEE workshop on Applications of Signal Processing to Audio and Acoustics, New Platz, NY, 1991, pp. 365 – 374.
[12] Fuchs H., Kedem Z., Naylor B., On Visible Surface Generation by a Priori Tree Structures, ACM SIGGRAPH, 1980, pp. 124-133.
[13] Funkhouser T., Carlbon I., Elko G., Pingali G., Sonhi M., West J., A Beam Tracing Approach to Acoustic Modeling for Interactive Virtual Environments, ACM SIGGRAPH, 1998, pp. 21-32.
[14] Funkhouser T., Min P., Carlbom I., Real-Time Acoustic Modeling for Distributed Virtual Environments, Proceedings of SIGGRAPH, 1999, pp. 365 – 374.
[15] Heckbert P., Hanrahan P., Beam Tracing Polygonal Objects, ACM Computer Graphics SIGGRAPH, 1984, Vol. 18, No. 3, pp. 119-127.
[16] Hein R., Binaural Room Simulation Based on an Image Source Model with Addition of Statistical Methods to Include the Diffuse Sound Scattering of Walls and to Predict the Reverberant Tail, Journal of Applied Acoustics, Vol. 38, No. 2-4, 1993, pp. 145 – 160.
[17] Herder J., “Optimization of Sound Spatialization Resource Management through Clustering”, IEEE Signal Processing, Jan 2000, pp. 564 – 571.
[18] Klosowski J., Held M., Mitchell J., Sowizral H., Zikan K., Efficient Collision Detection Using Bounding Volume Hierarchies of k-DOPs, IEEE Transactions on Visualization and Computer Graphics, Vol. 4, No. 1, 1998, pp. 21-26. [19] Krokstad A., Strom S., Sorsdal S., Calculating the Acoustical
Room Response by Use of a Ray Tracing Technique, Journal of Sound and Vibration, Vol. 8, No. 1, 1968, pp. 118-125. [20] Lightscape (Autodesk): http://www.lightscape.com
[21] Min P., Funkhouser T., Priority-Driven Acoustic Modeling for Virtual Environments, EUROGRAPHICS, Vol. 19, No. 3, 2000, pp. 179 – 188.
[22] Naylor G., “ODEON – Another Hybrid Room Acoustical Model”, Journal of Applied Acoustics, Vol. 38, No. 2-4, 1993, pp. 131 – 144.
[23] Polygon Cruncher: http://www.mootools.com
[24] Preparata F, Shamos M., Computational Geometry, Springer-Verlag, New York, 1985
[25] Rational Reducer: http://www.rational-reducer.com [26] Savioja L., Huopaniemi J., Huotilainen T., Takala T.,
Real-Time Virtual Audio Reality, Proceedings of the ICMC, Hong Kong, 1996, pp. 107 – 110.
[27] Tsingos N., Funkhouser T., Ngan A., Carlbom I., Modeling Acoustics in Virtual Environments Using the Uniform Theory of Diffraction, ACM SIGGRAPH, 2001, pp. 59–68. [28] Tsingos N., Gascuel J., A General Model for Simulation of
Room Acoustics Based on Hierarchical Radiosity, Technical Sketches, SIGGRAPH Visual Proceedings, 1997.
[29] Vian J., Van Maercke D., Calculation of Room Response Using a Ray Tracing Method, Proceedings of the ICA Symposium on Acoustics and Theatre Planning for Performing Arts, Vancouver, CA, 1986, pp. 74 – 78.
[30] Whitted T., An Improved Illumination Model for Shaded Display, Communications of the ACM, Vol. 23, No. 6, 1980, pp. 343-349.
[31] Wilson A., Larson E., Manocha D., Lin M., Partitioning And Handling Massive Models for Interactive Collision Detection, EUROGRAPHICS, Vol. 18, No. 3, 1999
