4.3
Selection of Landmarks
Annotation of image sets is often the first step in statistical shape modeling. A landmark point is defined as a point of correspondence on each subject that matches between and within populations [DM98]. The key issues involved in landmarking are selection of landmark points, and the choice between manual and automated landmarking.
The landmarks are chosen considering their potential usage and ease of manual or automated selection for the given application. The applications for landmarks in this research are align- ment, texture mapping and shape reconstruction. The potential landmark points therefore should help define the overall face shape and should also be easily and reliably identifiable. This implies that landmarks should be able to capture local as well as global shape variation. However to be able to do so a large number of landmarks will be needed, in practice selection of such a large number of landmarks using either automated or manual technique is not feasible. For facial images, the landmark points typically chosen are anatomically significant such as edge and corner points around the eyes, nose and lips. The anthropometric research done by Farkas et al [Far94] described a set of facial landmarks and related measurements which are widely used in medicine as well as various computer graphics applications. Figure 4.2 shows the anthropometric landmarks used frequently for facial reconstructive surgery. DeCarlo et al [DMS98] built a face model from a parameterized surface based on anthropometric facial measurements obtained from 47 landmark points.
Larger areas of the face such as cheek and forehead are smooth; therefore it is difficult to reliably select landmarks around these areas. For 2D face images the facial pose variation is also a constraint, however 3D models are not affected because of pose variation. Keeping in view the above constraints, only a few landmarks can be reliably placed on the human face. Manual placing of the landmarks is time consuming and tedious. To overcome these difficul- ties, some semi-automatic landmarking methods have been developed. However, mostly these methods assume simple shapes, or a sequence of images of the same object [WCT99]. It is still a challenge to automatically annotate different objects of the same class for example face
Figure 4.2: Anthropometric landmarks used frequently for reconstructive surgery [KS96]. images of different people across large variations in pose and expression.
The global shape variation can be captured using a small number of facial landmarks. For example, Hutton et al [Hut04] demonstrated construction of dense surface models through manual annotation of only 9 facial points. These surface models were then successfully employed for classification of various medical syndromes as well as for the automatic reconstruction of raw 3D scans using statistical shape information.
A key issue regarding the use of manually placed landmarks is reliability. It has been shown by Gwilliam et al [GCH06] that up to 20 landmarks can be reliably placed on the face surface. In another piece of research, J. Shi et al [SSM06] used 29 manually placed landmarks for 2D face recognition. These landmarks were chosen on the basis of relevant research in anthropology, aesthetic surgery and art; this research motivated the authors to explore the role of ratios of the distances between these landmarks for face recognition. They used a mixed ANONA model to prove that the placement of landmarks was not affected by the persons doing landmarking or repetition of landmarking by the same person.
In our approach, anatomical landmarks need to be placed on 2D images as well on 3D face models. An important criteria for selection of landmarks is that these should be visible in all the images which are included in the dataset. This criteria is satisfied by assuming that
4.3. Selection of Landmarks 69
only frontal face images would be considered for shape reconstruction. In the domain of 3D shapes, landmarking often encounters certain problems caused by visualization, defects due to shape acquisition and the representation of the geometry. It is therefore recommended that 3D surfaces should be texture mapped to visually aid the landmarking process [Hut04]. To be able to perform shape analysis with landmark data, landmarks must be labeled consistently. For example left corner of the eye should always be labeled as landmark 1, and so on.
(a) (b)
Figure 4.3: Selection of landmarks,(a). Used for shape reconstruction, (b). Used for alignment.
Given the constraints in reliably identifying landmark points on the face surface, only 20 land- mark points have been selected (see figure 4.3). It has been shown that most of the landmarks chosen can be reproduced reliably across subjects for frontal face images [GCH06, SSM06]. The list of landmark given below defines left or right with respect to image rather than viewer and is given as:
1. Left edge point on the face boundary parallel to left corner of the left eye 2. Left corner of the left eye
3. Right corner of the left eye 4. Left corner of the right eye 5. Right corner of the right eye
7. Face boundary on the lower tip of the left ear 8. Lower point of left nostril
9. Lower point of right nostril
10. Face boundary on the lower tip of the right ear 11. Face edge point parallel to the left edge of the mouth 12. Left edge of the mouth
13. Right edge of the mouth
14. Face edge point parallel to the right edge of the mouth 15. Central point of the chin
16. Tip of the nose 17. Upper lip center 18. Bottom lip center 19. Left eye center 20. Right eye center
The landmarking operation is implemented using the visualization library VTK which supports the annotation through simple mouse click operations [SML96]. The landmark points have been concentrated around eyes, nose and lips because of the ease of landmarking. Another reason for the concentration of landmarks points around certain areas is to allow these facial features to map correctly to corresponding features on the 3D face model during inverse camera projection from 2D to 3D or during 3D to 2D camera projection. The landmarks on the inner parts of the face are also useful for accurate reconstruction of the nose, eyes and lips. Some landmark points have also been placed on the face boundary in order to explore the reconstruction of the face boundary.