The 2-plane “uv-st” parameterisation

There are four most popular parameterisation of a ray (of light) depending on the representation used[Camahort et al., 2009], [Levoy, 2006]. The “2-plane” parameter- isation, more commonly known as “uv-st” parameterisation uses two 2-dimensional planes, respectively the u-v plane and the s-t plane to represent the origin and the direction of a ray. It is not necessary for the planes to be parallel and in many light field applications it is often at different angles to each other. However for the scope of Lumigraph, UV and ST are assumed to have constant distance between them. Figure 2.12a shows the two planes UV (blue) and ST (grey) parallel to each other at a distance (d). The rayR1 is intersecting UV plane at (u,v) and the ST plane at (s,t). Together the ray can be represented as (u,v,s,t) as shown in the Figure 2.12a.

(a) The 2 Plane UVST parameterisation (b) A Lumigraph cube around an object Figure 2.12: The Lumigraph and its 2-plane parameterisation

2.8.2 Discretisation

The Lumigraph is a 4 dimensional continuous function where the plenoptic func- tion is represented on the surface of a 3-D cube surrounding an object or a scene (Figure 2.12b). For ease of computation, the plenoptic function is discretised in

the Lumigraph. There are M subdivisions indexed as (i,j) for the (s,t) plane and N subdivisions indexed as (p,q) for the (u,v) plane. The data value at the each grid point is referred asxi,j,p,q. A basis function Bi,j,p,q is associated to reconstruct the

continuous Lumigraph (L) as a linear sum. The choice of the basis function can be a constant as unity or it can accept quadrilinear basis function where at the grid point the value is 1 and falls to 0 at all neighboring points.

xi,j,p,q =< L, Bi,j,p,q >

2.8.3 Capture

The camera is a virtual pin hole camera placed at each grid point (i,j) centered at the point (si.tj) focused at the hyper-focal distance and looking straight along the

z-axis. The pixel values in the images are used as he values xi,j,p,q. The capturing

system of the Lumigraph is a regular camera with precise calibration mechanism. The capture positions respect to the object is calculated by the special arrangement of the “capture stage” and a few “markers”. The capture stage is built with two cyan coloured walls joined with each other at 90 s. The base is also a cyan coloured square which can be rotated in 90 s increment. The walls and the base have numerous concentric circles in deeper shade of cyan to act as markers (Figure 2.13a). At a time 8 or more markers need to be visible for each frame of the Lumigraph. A lot of these images are to be taken in variable sampling rates throughout the stage to capture the Lumigraph (Figure 2.13b).

(a) Capture stage. (b) Camera positions of the cap- tures on the viewing sphere. Figure 2.13: Capturing a Lumigraph. Image Courtesy Gortler et al.

2.8.4 Limitations

Lumigraph technique is highly effective for capturing the entire view of objects and scenes which later can be reconstructed from arbitrary camera position and even

approximate the 3D volumetric shame of the object by capturing picture samples around the object/scene on a specially constructed capture stage. Unfortunately, this remains Lumigraph’s biggest limitation as well.

Identifying the capture points in a large 3D scene is not trivial and poses many real-world difficulties. Lumigraph’s usage of the capture stage is an effective way of capture point identification but this makes it unviable for large objects or real-world scenes.

A very similar approach to Lumigraph was taken by Hanrahan et al. which represents outgoing light rays from an object or a scene into a 4 dimensional field [Levoy and Hanrahan, 1996]. The next section will describe the “light field” technique to represent and later reconstruct a real-world object or a scene.

2.9 Light Fields

Since the texture mapping technique [Blinn and Newell, 1976] there has been efforts for making realistic image based renderings and reconstructions with arbitrary views of real environment. One of the first such approaches warped 2D images of real scenes to give the rendered scene the perception of depth and projection [Chen, 1995] independent of the view angle. Apple incorporated this technique in their Quicktime VR software in the early nineties.

The major drawback of these methods was their viewpoints were fixed. Al- though it was possible to interpolate the images to fit the new viewpoints, the pro- cedure required depth data which was difficult to provide from 2D images of the environment. The introduction of Lumigraph [Gortler et al., 1996] described previ- ously, provided ways to reconstruct scenes and objects from arbitrary view points. The plenoptic function based parameterisation of captured light rays enabled Lu- migraph to represent, store and reconstruct efficiently, albeit with a complicated capturing method.

Light Field [Levoy and Hanrahan, 1996] captures its samples along a camera plane rather than samples around the subject like Lumigraph.

2.9.1 Representation

The Light Field is defined as the amount of radiance from a particular point towards a particular direction. It is represented by a 4 Dimensional plenoptic function (given the sampling is in 2D surface) P having 4 parameters defining the positions of the point(Orix, Oriy) and the direction(Dirx, Diry) of the ray.

(a) Light slab representation (b) Viewing geometry to create a light slab Figure 2.14: The parameterisation and schematic of the light slab generation in Light Fields. Image courtesy Hanrahan et al.

The parameterisation of the light ray space is based on a light slab repre- sentation (figure 2.14a) where there are two planes; one for origin of the ray and one serves as the exit plane of the ray. The cartesian co-ordinates of the two planes (s,t and u,v) are used as the 4 parameters to represent the plenoptic function.The light slab representation is good for efficient calculation and it also enables uniform sampling of the images of the scene. Figure 2.14b depicts the sampling row (at the camera plane) of a scene (at the focal point) to create a light slab out of the sampled images.

2.9.2 Methodology

The light field is constructed from a number of 2D images of the scene to be rendered from varying points in a plane (camera plane). Each point of the images represents one light ray origin and the its direction is determined by the relative position of the point in the image and the same point in the actual scene. The image plane here serves as the entry plane and the focal plane of the images serves as the exit plane. The rays can be easily constructed from these two planes and a collection of these rays can construct alight slab with illuminance data for the actual rendering.

After creating the light slab, it is necessary to store it after a compression procedure to (1) get rid of data redundancy, (2) enable easier random access to the ray database and (3) making it computationally less expensive to store and retrieve data during rendering. The light slab data is compressed in two steps. A lossy vector quantization where decoding the compressed data is very fast because of reproduction vector called codewords and efficient organization of a set of codewords into codebooks. The second step is to employ an entropy coding like Lempel-Ziv [Ziv and Lempel, 1978] to compress high probability codeword indices.

Reconstructing images from the Light Field is the final step of the entire ren- dering process. It is done in two steps: (1) calculating theu, vands, tparameters to construct the ray; (2) reconstruct the radiance sample from the ray. Given the entry

and the exit plane, it is quite trivial to compute the (u,v,s,t) parameters from the image co-ordinate (x,y) as a projective map via texture mapping. The re-sampling of the radiance procedure first constructs the function from the original samples and then applies a bandpass filter to reduce any aliasing effect. The re-sampling process is approximated by either a nearest neighbor or a bilinear interpolation.

The entire process of the Light Field started a new direction in image based models where it supported arbitrary camera positions for the first time in the history of realistic rendering. The Incident Light Field has also has been developed on a similar technology where the rays are constructed from images of the scene, com- pressed, stored into light slabs and re rendered efficiently. There is a very similar technology to Light Field that was there independently at the same time. We will now briefly describe The Lumigraph in the next section.

2.9.3 Limitations

Despite its many capabilities and prospects, Light Fields have a few limitations. 1. Static focal plane: The focal plane of LF is fixed and thus can only be focused

on a single plane. If an object spans considerable are, the entire object can not be focused at once.

2. Restrictions of the aperture: The image plane in the LF is fixed and the apertures are fixed at the capture time depending on the camera positions and image density. These can not be modified afterwards.

3. Can’t render synthetic scenes: LF only reconstructs the captured scene. It can not be used directly to render synthetic objects with the light available in the captured scene.

In order to mitigate these limitations, there are two prominent LF based technologies that offers much more flexibility into LF. The first one is Dynamically reparameteri- sation of LF which is discussed in the following section. The other technology is the Incident Light Field (ILF) which has been described in detail in the next chapter since the work described in this thesis has been based on ILF, eventually augmenting it to develop the novel Temporal ILF.