SCENE SAMPLING

When using more than two cameras, the resolvable scene resolution will vary some- what irregularly throughout the scene depending on which cameras can observe a partic- ular region. Consequently, it is impossible to select a regular set of points and associated convolution kernels that match the resolvable scene resolution. In most situations it is also impossible to position the samples so that they correspond exactly with pixel rays from each camera. Therefore, some form of interpolation must be used to map between the scene samples and the image data.

For planar camera configurations, two common sampling schemes are used. These both involve uniform sampling under a disparity coordinate system. In the first ap- proach, sample points are positioned so as to correspond to an integer disparity shift be- tween pixels in neighbouring cameras. Such a sampling scheme is shown in Fig. 2.12(a). If the cameras are evenly spaced on a regular grid then the sample points will corre- spond exactly with pixel ray intersections. As with integer disparity sampling for two cameras, this allows the discrete mapping points for each image to correspond exactly with the set of scene sample points. Unfortunately, to prevent aliasing, the convolution kernel, Wi(x, y, Z, u, v, w), must be elongated in the Z direction so as to match the

sample spacing. This results in large variations in the kernel shape between images. If a narrower kernel in theZ direction is used instead to reduce variation between images, then the scene will be inadequately sampled. Another consequence of this sampling scheme is that the resolution in the Z direction is significantly less than the resolvable limits of the system in many places.

An alternative approach is to arrange the samples so as to correspond to an integer disparity shift between the pixels in the two outermost cameras. This results in a finer spacing between samples in the Z direction, as shown in Fig. 2.12(b). However, some form of interpolation is required to map between the sample points and the discrete mapping points, _{(x, y, Zik(x, y)) :x, y, k ∈Z}. Within the region that is visible to all

cameras, the resulting sampling resolution is equal to the maximum resolvable scene resolution. Outside this region, the sample spacing is closer than what can be resolved. In most situations this is not a problem, except that more samples are used than is necessary. With a number of stereo algorithms, this region is outside the defined scene volume and so can be ignored anyway.

To deal with more general camera systems or provide an arbitrary scene resolution, a variety of other sampling schemes can be used. These can be useful in certain situations, but usually result in a more complex mapping between the scene and image parameters. One such approach is to position samples on a regularX, Y, Zgrid throughout the scene volume [Seitz and Dyer 1999, Culbertson et al. 1999, Slabaugh et al. 2000b]. This is useful in some applications where the scene is constrained to lie within a known finite volume, and must modelled at a fixed resolution. An alternative approach presented by Slabaugh et al. [2000a] is to warp voxels based on their position within some user-defined voxel space. This allows the scene to be sampled independently of camera position or

Camera N Camera 1 ... Scene space Virtual image plane Principal point Sample points Camera N Camera 1 ... Scene space Fully visible region

(a) (b)

Figure 2.12 Planar multiple camera sampling schemes. (a)Nearest neighbour integer disparity sam- pling, where sample points are positioned so as to correspond with integer disparity shifts between neighbouring cameras. (b) Furthest neighbour integer disparity sampling, where sample points are positioned so as to correspond with integer disparity shifts between the two outermost cameras.

resolution and enables the reconstruction volume to accommodate a semi-infinite or infinite region. Environment mapping [Greene 1986] can also be used to deal with large or infinite scenes. This approach is commonly used in the computer graphics domain, where background or distant objects are represented by a texture mapped sphere or cube that surrounds the foreground scene. Although convincing synthetic images can be produced, this method is inappropriate for most scene reconstruction applications as the three-dimensionality of the background is lost. It also requires separate modelling of the foreground and background, leading to difficulties in the reconstruction process.

2.6 PRIOR KNOWLEDGE

To improve the estimation process, prior knowledge about a scene can be incorporated into the reconstruction process. This allows additional bits of information to be used that are not available from the camera images. The most common way of doing this is to apply hard constraints to the set of model parameters. This is the simplest approach. It limits the range of possible scene estimates, hopefully reducing the probability of a poor reconstruction. Such constraints can be applied to individual parameters or on the allowable combination of parameter values. For example, the continuity constraint enforces opaque surfaces to be linked together so that they form a continuous surface.

In addition to imposing hard constraints, prior statistical information relating to the scene parameters can be applied. This is a more general and flexible approach, as both hard constraints and arbitrary probabilistic information can be used. For in- stance, the scene will usually consist of cohesive opaque and transparent regions rather than a random cloud of points. Therefore, preference should be given to neighbouring