Existing Databases

Several databases have been proposed which contain facial images for computer vision purposes, including [LAKG98,DGLG11a]. These databases consist either of single images or image sequences, mainly captured in laboratory conditions. The information on the databases content is usually provided by the database authors. Depending on the purpose of the database, this is called "meta-data" and includes the person’s identity, facial expression, action unit activation, annotated facial components or landmarks.

To be able to validate the facial expression based on image-data under harsh lighting conditions that will be discussed in this thesis, we need a suitable set of data. There have been a number of freely available databases in the public domain, none of which addresses successfully all the FER problems intended to be solved in this thesis. Table 2.2 lists a number of important characteristics of the databases. In the table we have included information on 10 facial expression databases that have been used in the literature on FER. A number of important characteristics are also listed: the number of subjects, videos and images contained in the database as well as whether the database is publicly available. In the following Sections, the most popular and those used in this thesis will be discussed. Two publicly available databases: the Japanese female facial expression (JAFFE) and the Static facial expression in the wild (SFEW) were chosen. The other database used is the HDR dataset, which was created in this thesis for the purpose of providing the facial expression recognition community with a dataset for developing and validating algorithms for facial expression analysis. This will be discussed in Chapter 3.

Table 2.2: Characteristics of facial expression databases

Name Expression description No._Subjects Characteristics Views

AR

[Martinez & Benavente, 1998] 3 basic emotions 126

Glasses, scarf,

different illumination Frontal AT&T (formerly ORL

[Samaria & Harter, 1994] 2 basic emotions 30 Glasses, varying lighting Frontal BU-3DFE [Yin et al., 2006] 6 basic emotions 100 4 levels of intensity (3D) Frontal, 45, -45 Cohn-Kanade

[Lucey, Patrick et al., 2000] 7 basic emotions 97 Uniform lighting Frontal, 30 degrees

JAFFE [J. et al., 1998] 6 basic emotions 10 Uniform lighting Frontal

MMI [Pantic et al., 2005] 6 basic emotions 90 Glass, facial hair Frontal, profile

SFEW 6 basic emotions 95 Non-uniform lighting Frontal

PIE [Sim et al., 2002] 4 expressions 68 43 lighting conditions Frontal

NimStim

[Tottenham et al 2009] 6 basic emotion 31 Uniform lighting Frontal

MultiPIE [Gross et al., 2007] 6 basic emotions 43 Frontal

mJAFFE: The JAFFE [LAKG98] facial expression database has been extensively used for facial expression analysis. Each female has two to four samples for each expression, totalling 213 greyscale facial expressions images from 10 Japanese female actresses. It consists of six basic emotions (anger, disgust, fear, happiness, sadness, surprise and neutral). The expressions expressed by each picture were subjectively tested on 60 Japanese volunteers. Each image is of size 256×256. Figure 2.7 shows a sample image from the JAFFE database.

Figure 2.7: Sample Six Basic Emotions from the JAFFE database.Starting from top, we have angry, disgust, fear, happy, sad and surprise

Since the JAFFE database is a standard database that has been accepted and used on a number of occasions for validating facial expression experiments. We decided to create a modified version of JAFFE called mJAFFE (modified JAFFE) with a Photoshop script by making: shadow, high contrast, low contrast, overexpose and underexpose lighting conditions. This is in order to make an artificial lighting conditions close to the HDR database, for the purpose of validation the FER experiments. Figure 2.8 shows a sample image from the mJAFFE database.

Figure 2.8: Sample Six Basic Emotions from the mJAFFE database. Starting from top, we have angry, disgust, fear, happy, sad and surprise

SFEW:The SFEW [DGLG11a] dataset was extracted from frames from Acted Facial Expressions in the Wild (AFEW) database. The database was collected with close to real world lighting conditions, with different head poses, large age ranges, different face resolutions, occlusions and different focus. There are 95 subjects and a total of 663 well-labelled usable images. Figure 2.9 shows a sample image from the SFEW database.

Figure 2.9: Sample Six Basic Emotions from the SFEW database. Starting from top, we have angry, disgust, fear, happy, sad and surprise

Image Acquisition

In image processing, image acquisition usually involves retrieving images from a source that is automatically capturing images [CB03]. For example, real-time image acquisition creates a stream of files that can be automatically processed, queued for later work, or stitched into a single media format. There are methods of image acquisitions in image processing that actually uses acquisition devices, for example, using digital cameras to acquire images to build accurate models of different scenes.

Ultimately, in an image processing task, image acquisition is always the first step because, without an image, no processing is possible. The image that is acquired is completely unprocessed and is the result of whatever hardware was used to generate it, which can be very important in facial expression process to have a consistent baseline from which to work. One of the purposes of an image acquisition process is to have a source of input that operates within such controlled and measured guidelines that the same image can, if necessary, be nearly perfectly

reproduced under the same conditions so that inconsistency factors are easier to locate and eliminate.

In this thesis, due to the type of images used (those under harsh lighting conditions), it is important to relate an image to the scene light from which it was captured. In practice, often, facial images are subjected to changes in viewpoints, lighting conditions and expressions, wherefore providing clear, highly detailed images still needs more improvement. For instance, one of the key elements in getting a good image is how well the camera can cope with the scene changes in lighting and weather conditions. Even with the best camera on the market, image variability is still an issue. The first issue is how to choose the best features to represent the face in order to deal with the facial variability.