Although stimuli were relatively small (9◦×12◦ of visual angle) and briefly displayed (200 ms),
thus discouraging both the need for and possibility of eye movements despite the experimental instructions to stay fixated at the center, naturally different subjects would occasionally move their eyes or covertly attend to different facial properties. In this section, we investigate how facial features, both holistic and localized, weigh differently in the perception of familiar faces relative to unfamiliar ones. We do this by modeling dissimilarity judgments between faces as driven by the inter-stimulus distances between single features, such as eye position or attractiveness. Refer to Section 2.10.5.2 for a detailed description of how a single feature is assigned a corrected significance level in modeling the dissimilarity judgments.
2.7.1 Independently-rated face properties and dissimilarity
An entirely separate set of 10 healthy adult subjects (7 male) participated in an online experiment,
FaceRate, in which each of the 40 face stimuli in our data set were rated for 18 holistic and localized features such as femininity, largeness of the eyes relative to average, and attractiveness. The face features were rated by subjects on a computer, in their own home, taking as much time as they wanted for each face, while being presented with the face stimulus (inner features only) on the screen, and a sliding-bar interface like the one shown in Figure 2.25.
Figure 2.25: A snapshot of part of sliding bar interface used by subjects in FaceRate. This was rendered in a browser and used by 10 participants to provide holistic and featural judgments about the 40 face stimuli.
Each feature was to be rated between 0 (much less than average) and 100 (much greater than average), and was initialized at 50. These ratings were then individually Z-scored (across stimuli) for each rater. Finally, the 10 raters’ data were averaged together to form a single averaged Z-score
Figure 2.26: Face dissimilarity judgments, between pairs of entirely unfamiliar faces, can be statistically significantly modeled using distances between single features. The significance values are written in the table and the color corresponds to −log10(p) . The values in the “All” row are
determined by modeling the subject-averaged dissimilarities.
metric for each of the face features. The rated face features are described in detail in Section 2.10.5.1, but they have mostly self-explanatory names.
Figures 2.26 and 2.27 show the result of modeling dissimilarity judgments in SimRate using the 18 face features provided by participants in FaceRate. Whereas in Figure 2.26 we model only those dissimilarity judgments between pairs of unfamiliar faces, in Figure 2.27 we model only the dissimilarity judgments between pairs in which at least one face is familiar. We see that for both kinds of comparisons, the size of the eyes relative to the face is an important: if one face has relatively smaller eyes, and the other relatively larger, this will on average be associated with a higher dissimilarity. This is likely in part explained by the fact that subjects were asked to fixate between the eyes.
The weights (−log10p)of features for the unfamiliar face pairs are on the whole positively cor-
related with the weights of features for familiar face pairs: there is an average individual subject correlation across conditions of 0.23, and the subject-averaged weights correlate across conditions at 0.28. However, there are also some important differences. Comparisons involving familiar faces tend to use a broader range of face features than those involving only unfamiliar faces: this is seen
Figure 2.27: Face dissimilarity judgments, between half- or both-familiar face pairs, can be statistically significantly modeled using distances between single features. The significance values are written in the table and the color corresponds to −log10(p) . The values in the “All” row are
determined by modeling the subject-averaged dissimilarities (among those who are familiar). qualitatively by examining the relatively higher scattering of brightness values on the right side of table showing weights for familiar comparisons. Also, whereas happiness does not significantly model unfamiliar comparisons (p <0.64), it does for familiar comparisons (p <0.0002). Although
the faces were all supposed to be emotionally neutral, inevitably subtle micro-expressions leaked through conveying some kind of nonzero emotional valence. The increased weighing of happiness in familiar comparisons is consistent with the hypothesis that we are more sensitive to detecting the emotional state of familiar faces.
2.7.2 Individual facial keypoints and dissimilarity
Using the locations of the annotated keypoints in the scale-normalized faces (i.e., the stimuli the sub- jects viewed), we performed an analysis exactly analogous to the one on facial features above, except with scalar distances replaced with Euclidean distances in 2D combining the x- and y-coordinates of the keypoints. We used inter-stimulus distances between keypoints to model the dissimilarity judgments of subjects. The results are shown on the average face (equally blended morph of all 40 stimuli) in Figures 2.28 and 2.29.
Figure 2.28: Face dissimilarity judgments, between pairs of entirely unfamiliar faces, can be statistically significantly modeled using distances between single features. The locations of keypoints which significantly model (viz., havingp≤0.01) dissimilarity judgments betweenunfamiliar faces
are highlighted with a circular Gaussian kernel for illustration purposes. The standard deviation of this kernel (0.5◦of visual angle) was chosen to allow some fuzziness in the location of the underlying
Figure 2.29: Face dissimilarity judgments, between half- or both-familiar face pairs, can be statistically significantly modeled using distances between single features, shown highlighted where
p ≤ 0.01. However, compared to judgments between unfamiliar faces, there are fewer significant keypoint locations, possibly due to more holistic processing of familiar faces.
For both comparisons involving familiar faces, and those which do not, we find individual facial keypoints which can statistically significantly model dissimilarity judgments. However, we find on average many more of these when modeling unfamiliar face comparisons. One possible simple reason might be that we have fewer data samples of dissimilarity judgments involving familiar pairs, so the average ratings are noisier and the individual ratings are sparse, making a significant fit more difficult to achieve. However, this is unlikely because there were many examples of face features, rated in FaceRate and shown above, which significantly modeled only the familiar comparisons, not the unfamiliar ones, such as “nordicness” for subject S9, or “blemishedness” for subject S4, or happiness for several others. A more plausible explanation is that familiar faces are perceived and compared more holistically than unfamiliar faces, thereby eliminating a significant effect of any one single keypoint location as our analysis here attempts to find for each individually. A related finding was reported by Heisz et al. [40], in which it was found that familiar faces (though not
personally familiar) under free viewing (3 seconds, compared to our≤0.2 seconds) were scanned by
eye movement less in some contexts (identity recall but not recognition of familiarity). If we suppose that eye movements are more difficult to suppress or are otherwise more frequent for unfamiliar faces, this might mean that locally salient visual information plays a larger role in unfamiliar faces than for familiar faces.
The facial keypoint weightings in this subsection may be related to the independently-rated face features weightings, shown in the previous subsection; we list here several examples of consistency between the two: (1) subject S7 heavily weighs both nose-upturned-ness above, and nose keypoints here, in the case of familiar face pairs, (2) subject S6 weighs “archiness” (extent to which eyebrows are arched) heavily above, and eyebrow keypoints here, in the case of unfamiliar face pairs, (3) for subject S0, eye size is significant in unfamiliar face pairs above, but it is not for familiar ones, and the corresponding results are found with respect to eye keypoints here, (4) subject S8 heavily weighs face wideness above, and here keypoints along the side of the face which would cue face wideness, for unfamiliar face pairs, (5) the most significant rated feature above for subject S1 in the case of familiar face pairs is eyebrow archiness, and the only keypoint location we find significance for here is the tip of one of the eyebrows.