Pilot experiments

Pilot experiments (2.4, E0) provided the means by which several candidate shape- from-silhouette approaches and algorithms were implemented and evaluated. All the pilot experiments were conducted using the simulated setting described in Section 2.3.5.

Volumetric reconstruction

An implementation of volumetric reconstruction based on the space carving algo-

rithm [64] was tested. In this algorithm, 3D space is divided into a grid of equal

sided cubes, known as voxels. Each voxel maps to a pixel, or group of pixels in every camera’s image plane. Voxels that fall inside the silhouette in every camera image plane form part of the object being reconstructed, whereas those that fall inside the silhouette for some or none of the camera images do not. Iterating over every voxel defining the 3D reconstruction volume forms a volumetric represen- tation of the object being reconstructed where set voxels fall inside the volume of the object and unset voxels do not. The resulting data can either be displayed us- ing volumetric techniques, or the surface can be extracted using an algorithm such

as marching cubes [74]. Example reconstructions from the implemented volumet-

ric algorithm are shown in Figure 2.3 showing reconstruction of the entire human body, and Figure 3.2a a human head.

In applications where the cameras remain fixed, volumetric reconstruction can be accelerated by building tables mapping each voxel to the pixels representing it in the camera image planes. However, this becomes memory intensive as the camera count or resolution is increased, and therefore limits the scalability of such approaches. The necessity to extract a surface from the volumetric data to enable polygonal model genesis and texturing adds a further processing burden to the

CHAPTER 3. BACKGROUND AND RELATED WORK 57 workload. Whilst the volumetric implementation continued to be used for our initial synchronisation experiment (2.4, E2), alternative methods better suited to our requirements were being sought.

Voxel contour intersection

Figure 3.1: Point clouds formed using the Voxel Contour Intersection method in a simulated setting, from synthetic human model using virtual cameras.

One way in which volumetric reconstruction was a poor fit to our requirements was that it models object volumes rather than the surface between object and space. This can be remedied, as previously mentioned, by a post processing al- gorithm that steps around the boundary between set and unset voxels forming a polygonal surface. With this in mind we conceived our own novel algorithm for evaluating only those voxels that fall at the edge of the object. This was named ”Voxel Contour Intersection” and worked using contours defining the edge of sil- houettes as well as the silhouette images from the conventional space carving approach. Voxels in the final set were determined to fall at the surface of the ob- ject being reconstructed if they fell on the path of the silhouette contour in at least one image and within the silhouette in all other images. In this way we were able

to generate a point cloud representing voxels at the surface of the object, as shown in Figure 3.1.

This approach was no more computationally efficient than calculating the voxel occupancy in the space carving approach, but appeared to get a step closer to a polygonal representation without using marching cubes. Deriving a polygonal mesh from the point cloud was the next problem to be solved along this line of research. In fact marching cubes would have been no use here, because the result- ing point cloud was sparse compared with a voxel set, due to the discreet nature of pixels falling along the silhouette contour. Approaches such as delaunay triangu-

lation [31], [29] seemed an appropriate fit, and software libraries offering it were

readily available. However, whilst delaunay triangulation of the sparse point cloud formed a cleaner surface than marching cubes, the performance was no better.

Direct surface reconstruction

Literature revealed that researchers had used ”hybrid” methods [14] in which the

form is roughly calculated volumetrically before refinement by following contour edges. However, it was becoming clear that an approach able to resolve the sur- face of the object directly from silhouettes without first performing a volumetric reconstruction was required. Such an approach would certainly reduce memory overhead, and could possibly be computationally more efficient.

One such technique, the Adaptive Dandelion [73], was implemented and tested.

The approach begins by constructing the triangles forming the surface of an icosa- hedron such that the centre of the icosahedron’s volume falls within the silhouettes in all images. Rays are then traced out from this central point along the line to each of the icosahedron vertices. The rays are mapped to every camera image and trimmed to the shortest length that falls within all silhouettes. Icosahedron ver- tices are adjusted to the 3D position at the end of the trimmed ray. The adaptive aspect of the algorithm is that each triangle can be recursively subdivided as re- quired to increase surface detail. The algorithm was found to be very efficient, and

CHAPTER 3. BACKGROUND AND RELATED WORK 59 lacked the memory overhead of volumetric approaches. However the ability of the approach to only model spherical terrain like (STL) objects meant that whilst it was reasonable for modelling the human head, it was unable to model limbs. Fig- ure 3.2b shows a human head reconstructed using the adaptive dandelion method. Notice how in comparison to the same head reconstructed by other methods in that diagram the adaptive dandelion head lacks the top of the ears, this is due to their geometry not conforming to the STL requirements of adaptive dandelion. A more general purpose approach, capable of reconstructing the visual hull was

found in the Exact Polyhedral Visual Hulls (EPVH) algorithm [39]. The method

works by projecting camera silhouette contours into 3D space to form viewing rays, consecutive pairs of which from a particular camera form an infinite trian- gle. The rays are projected into every other camera and intersections with contour edges along the ray recorded. Using these intersections the minimal spans pass- ing across the surface of the visual hull can be determined. From these initial spans, each defined by two vertices, the algorithm describes a method by which the exact connectivity to neighbouring span vertices can be determined to com- plete a watertight polyhedral mesh. Finally the surface polygons can be extracted by following the connected vertices of the mesh and referring back to the infinite triangles formed at the start.