The importance of the shape recovery from 2D images is underscored by the difficulties faced by 2D shape analysis methods aimed at overcoming the 3D features of human face. Such methods require the use of multiple images of each subject which are carefully acquired and thus it is difficult to apply these methods outside the research domain. The 3D shapes acquired so far by such methods are suitable for specific applications such as face tracking and are not suitable for realistic synthesis of novel subjects.
A 3D face shape can be acquired in a number of ways. One of these methods is shape recovery from 2D images. For shape recovery from 2D image (s), the intensity variation is a major clue. The relationship between 3D shape and shading is affected by a number of factors including texture, illumination and pose variations and is therefore not reliable relationship in the context of shape estimation. In this chapter, it was shown that the shape from shading techniques are limited by their generic nature and simplifying assumptions.
A trend which is emphasized in this thesis is that a variety of methods have overcome the shortcomings of the generic nature of shape from shading methods by incorporating class specific shape information such as 3D face models. This is evident from improvements obtained in the performance of the shape from shading and shape from silhouettes methods for face recognition.
Morphable models have provided a major improvement over 2D shape analysis by using a 3D shape model and a 2D texture model in an analysis by synthesis framework. However morphable models require a complicated scheme involving optimization over shape and texture parameters simultaneously which affect the performance of this method.
It can be seen from the literature review that most of the 3D reconstruction methods use high quality 3D laser scanned models which are not publicly available. Non-availability of data is seen as a key factor limiting research in this area.
This chapter has also presented research where low resolution synthetic models have been used to generate photo-realistic images by warping texture onto those models. This indicates that
2.6. Summary and Conclusions 35
non-availability of high resolution 3D scans such as Cyberware need not limit the research in this area. Photorealistic images can be generated by simply mapping texture from the given image onto a 3D model. Texture synthesis for generating photorealistic images from the given models is not essential.
Photorealism of images rendered from reconstructed 3D models is often perceived as the cri- terion for the accurate reconstruction. Since photorealistic images can be generated even if the actual shape has not been recovered accurately, photorealism is not a valid criterion for accurate shape recovery. By a similar argument 2D face recognition results obtained using such methods are also not a valid criterion for accurate shape recovery.
The discrimination ability of a feature space is critical to the success of model based shape reconstruction approaches. This chapter has identified features that have been used for shape reconstruction in various model based approaches. These features include pixel intensities, edges, specularities, and anatomical landmarks (feature points).
At the heart of model based reconstruction approaches, such as morphable models, there is an optimization process which in the case of morphable models has been stochastic Newton opti- mization (SNO). The optimization involved in reconstruction has been simplified and speeded up by improving the feature space. Recent research has revealed the shortcomings of stochas- tic Newton optimization and used other optimizers such as Levenberg Marquardt, and the multi-dimensional simplex or Amoeba optimizers have been used for shape reconstruction.
The ability of statistical methods to encode shape variations is dependent on the quality of the point to point correspondences between 3D shape models. Morphable models are built using 3D correspondence established using a modified optical flow algorithm. The modified optical flow algorithm establishes 3D correspondence using pixel wise correspondences between 2D images to establish correspondences between 3D scans. Since 2D images can vary a lot in terms of size, pose, skin color and illumination, such correspondences can often be incorrect.
Some of the 3D reconstruction approaches treat 3D shape recovery as a subtask of 2D face recognition with the objective of establishing independence from various artifacts of 2D face
images. Thus the efficiency of the reconstruction process is given high priority while the 3D accuracy of reconstructed models has been neglected. This approach has been facilitated by the fact that 3D face geometries between different subjects does not differ dramatically, and most shape differences can be hidden by texture mapping.
This proves that 2D face recognition results are essentially qualitative proof of reconstruction accuracy. The comparison of the reconstruction techniques on the basis of recognition rate is also not possible unless these are not available for the same dataset and are represented in the same fashion. Ideally a quantitative method of determining reconstruction accuracy would be based on comparison of a 3D shape acquired using an image based reconstruction technique with the ground truth 3D shape acquired using dedicated laser scanner.
The feasibility of the shape reconstruction approach from a single face image is demonstrated by the success of reconstruction techniques that do not assume precise knowledge of intrinsic and extrinsic camera parameters. This research also shows that the impact of intrinsic camera parameters on reconstruction accuracy is limited. Thus it should be possible to recover 3D shape from 2D images using appropriate features and optimization strategies under reasonable assumptions about various factors.
Chapter 3
Processing 3D Shape Data
3.1
Introduction
The literature review in chapter 2 established the dependence of model based shape reconstruc- tion systems on the accurate representation of 3D shape variation. 3D shape analysis depends on the scanning technology as well as the methods used for processing 3D shape data. This chapter explores the advantages and limitations of various scanning technologies, the need for pre-processing and the analysis of the accuracy of 3D scans.
The 3D shape databases used in this research are described along with the method used to process them. The raw data acquired using scanners has a number of defects, moreover the raw scans are randomly aligned and there is no known point to point correspondence. The raw scans therefore need to be processed, before they can be used for shape analysis.