Offline Deformable Face Tracking in Arbitrary Videos
Grigorios G. Chrysos
Epameinondas Antonakos
∗Stefanos Zafeiriou
∗Patrick Snape
Department of Computing, Imperial College London
180 Queens Gate, SW7 2AZ, London, U.K.
{g.chrysos, e.antonakos, s.zafeiriou, p.snape}@imperial.ac.uk
Abstract
Generic face detection and facial landmark localization in static imagery are among the most mature and well-studied problems in machine learning and computer vision. Currently, the top performing face detectors achieve a true positive rate of around 75-80% whilst maintaining low false positive rates. Furthermore, the top performing facial land-mark localization algorithms obtain low point-to-point er-rors for more than 70% of commonly benchmarked images captured under unconstrained conditions. The task of facial landmark tracking in videos, however, has attracted much less attention. Generally, a tracking-by-detection frame-work is applied, where face detection and landmark local-ization are employed in every frame in order to avoid drift-ing. Thus, this solution is equivalent to landmark detec-tion in static imagery. Empirically, a straightforward ap-plication of such a framework cannot achieve higher per-formance, on average, than the one reported for static im-agery1. In this paper, we show for the first time, to the best of our knowledge, that the results of generic face detection and landmark localization can be used to recursively train powerful and accurate person-specific face detectors and landmark localization methods for offline deformable track-ing. The proposed pipeline can track landmarks in very challenging long-term sequences captured under arbitrary conditions. The pipeline was used as a semi-automatic tool to annotate the majority of the videos of the 300-VW Chal-lenge2.
1. Introduction
Generic face detection is widely regarded as a mature field and has been successfully integrated into a number of consumer-grade electronics including digital cameras and
∗The authors had equal contribution.
1We found that, in practice, the performance is dramatically lower than
the one reported for static imagery due to the extremely challenging record-ing conditions of arbitrary videos.
2http://ibug.doc.ic.ac.uk/resources/300-VW/
(a) Our proposed pipeline.
(b) State-of-the-art detector [23] + landmark localization [22].
(c) State-of-the-art tracker [16] + landmark localization [22].
Figure 1: We propose a pipeline for robust and accurate offline deformable face tracking in long-term sequences. Our system significantly outperforms current tracking-by-detection techniques using either state-of-the-art face detec-tors (1b) or rigid object trackers (1c).
modern smartphones [45, 48, 39, 28]. Modern face de-tectors are robust under a large variety of arbitrary, often termed “in-the-wild”, conditions. Recently, due to (i) the ef-forts made by the community to collect and annotate facial data with regards to a consistent set of facial landmarks [19, 11, 25, 48, 32, 33, 30], and (ii) the development of robust deformable models [37, 41, 8, 4, 5, 2, 38, 22, 29, 6, 10, 3], significant progress has been made towards accurate and ef-ficient facial landmark localization under both controlled and unconstrained conditions. This progress is, arguably, one of the most important steps towards high performance face recognition and verification [36], as well as facial ex-pression recognition [34].
Despite the fact that face detection and facial landmark localization in static imagery has received considerable at-tention, facial landmark tracking in lengthy videos (also
re-ferred to as deformable face tracking) has attracted much less research effort. Currently, there are no existing meth-ods that are able to reliably track the landmarks of a human face over lengthy periods without the need of human inter-vention and re-initialization. This challenging problem is often referred to, in literature, as “tracking in long-term se-quences” or simply “long-term tracking” [21], even though those terms are used for tracking by means of a bounding box. The main reasons behind this lack of attention for de-formable face tracking are:
• The creation of a meticulously designed benchmark for deformable face tracking in lengthy videos requires the manual annotation of a set of landmarks in each frame of each video. This is a laborious and very ex-pensive task that has yet to be tackled by any research group. Furthermore, the sequences that are currently used for demonstrating qualitative performance of de-formable face tracking algorithms are very short (in the range of 3-5 seconds long [40]) and are chosen so that standard face detection algorithms, such as Viola Jones [39], perform well [41, 31, 9, 12].
• There is a trend towards approaching facial landmark tracking as a by-product of generic face detection and landmark localization (i.e., tracking is performed by applying face detection followed by landmark local-ization at each frame) [41, 38, 29]. This is motivated also by the recent literature in object tracking where tracking-by-detection is usually applied in order to cir-cumvent the drifting issues [21, 47].
We believe that deformable face tracking cannot be ef-fectively solved by using the standard tracking-by-detection procedure of applying face detection followed by landmark localization at each frame [41, 38, 29]. By inspecting the results of coupling state-of-the-art face detection and land-mark localization algorithms [32, 30, 28, 2, 38, 13], we note a significant gap in performance. The current state-of-the-art generic face detection algorithms [28, 13] report a true positive rate of about 75-80% in the popular FDDB benchmark [20] whilst allowing for a very small number of false positives. Hence, approximately 20-25% of faces have not been successfully detected. Similarly, the re-sults of the latest landmark localization competition (i.e., 300W [32, 30]), as well as the reported performance of recent methods [2, 38, 29], indicate that the best perform-ing landmark localization methods manage to successfully align no more than 70% of the images of databases with challenging capture conditions. In practice, we empirically found that the performance of state-of-the-art generic face detection and landmark localization algorithms on arbitrary videos collected from YouTube and other sources can be much lower than reported on static images. Figure 1 shows a characteristic example in which neither a state-of-the-art
tracker [16], or state-of-the-art face detector [23], followed by landmark localization, successfully track the face in an arbitrary video. In contrast, our pipeline is both accurate and robust in these sequences.
In this paper we show that even though state-of-the-art generic methodologies for face detection and landmark lo-calization cannot solve the facial landmark tracking prob-lem in unconstrained videos, they provide an excellent base on which to build effective procedures. That is, we pro-pose a pipeline for joint automatic construction of person-specific deformable face detection and landmark localiza-tion in an offline manner. Automatic construclocaliza-tion of person-specific statistical facial deformable models from videos by exploiting a generic model [14, 31] or in an unsupervised manner [7, 44] has only very recently received attention. Nevertheless, all these methods assume that correct bound-ing boxes are provided by a robust face detector for all the images, which is rarely the case for arbitrary videos. The aforementioned methods all fail if false positive detections are provided. On the contrary, our method jointly solves the problem of automatic construction of deformable face detection and facial landmark localization. We show that the proposed pipeline is extremely effective for robust long-term offline deformable face tracking. We show quantita-tive experiments in 16 lengthy videos (1-2 mins, 25 fps), which have been annotated with regards to 68 facial land-marks (more than 30,000 annotated frames) and qualitative results in more than 100 long-term sequences.
In summary, the contributions of this paper are:
• We present an accurate and efficient pipeline for of-fline deformable facial landmark tracking in long-term sequences. The proposed system is the first, to the best of our knowledge, to return accurate results for all the frames of arbitrary length videos without false posi-tives or drifting issues.
• The proposed technique iteratively updates a person-specific face detector and facial landmark localization model which gradually improves accuracy.
• Our experiments show that the proposed pipeline sig-nificantly outperforms existing tracking methods that employ state-of-the-art face detection [48, 39, 28] and landmark localization [41, 8, 38, 22, 29] techniques in a tracking-by-detection manner on challenging videos. • The proposed pipeline was used as a semi-automatic tool to annotate the videos of the 300 Videos in-the-wild (300-VW) Challenge [35]2.
2. Deformable Face Tracking Pipeline
The proposed pipeline, presented in Fig. 2, aims to per-form accurate facial landmark tracking on all the frames
Figure 2: Overview of the pipeline. After applying a state-of-the-art generic deformable face detector (Step 1) to obtain an initial estimation of the shapes with minimal false positive detections, we iteratively (Step 4) train and fit a person-specific face detector (Step 2) and a person-specific generative deformable model (Step 3) that gradually improves the results.
of lengthy video sequences3. Let us denote the shape in-stance of a frame j as sj = [ℓj1, . . . , ℓjn]T, that consists of the Cartesian coordinates of n landmark points, denoted by ℓj
i = [x j i, y
j
i]T, ∀i = 1, . . . , n. Thus, given an input long-term sequence of N0 frames, i.e. IN0 = {I1, . . . , IN0}, the pipeline aims to estimate the corresponding set of land-marks per frame, i.e. {s1, . . . , sN0}. This involves four
discrete steps solved in an iterative manner which are de-scribed in detail in the following sections:
Step 1: Acquire an initial estimation of the landmarks per frame using state-of-the-art generic face detection and landmark localization techniques.
Step 2: Train and fit a person-specific face detector. Step 3: Train, fit and iteratively update a person-specific
generative deformable model for landmark local-ization.
Step 4: Update the person-specific detector and re-apply Steps (2) and (3).
Step 1: Generic Deformable Face Detection
The first step is to acquire an initial estimation of the landmarks per frame. We take advantage of recent state-of-the-art generic face detection and landmark localization methods in order to obtain acceptable initial estimations for as many frames as possible. There have also been pro-posals to perform face detection and landmark localization jointly [48, 13]. Nevertheless, the best results are achieved by using different methods for face detection and land-mark localization (e.g. by combining [48] with [2, 38] or with [41]).
The results reported in the most well-known face de-tection benchmark (FDDB [20]) show that there are many face detection systems that achieve impressive true positive vs. false positive rates; some with publicly available imple-mentations [28, 23, 43, 48, 26], others without [13, 42]. The biggest advantage of most of these methods is that they can 3We make the assumption that the person may leave the frame or that
the camera will move, hence in many frames the face may not be visible.
be adjusted not to return any false positive detections, gen-erally at the cost of decreasing the true positive rate. Since the detected faces will be used as initializations when train-ing person-specific models, we require a face detector that returns a minimal number of false positives and is highly efficient. We have experimented with the majority of the top performing publicly available detectors and we empir-ically found that the implementation of [23] had the best ratio of performance and speed4. The decision threshold of the method in [23] was set so that virtually no false posi-tives were returned whilst retaining a true positive rate of approximately 65%.
Existing state-of-the-art generic facial landmark local-ization methods can be separated in two categories: genera-tive [37, 2, 38, 5, 6] and discriminagenera-tive [41, 8, 22, 29]. Most methods are accompanied by publicly available implemen-tations pre-trained on multiple databases which makes them easy to access and use. Based on the results reported on standard datasets of static images, we chose to use the open-source implementation4 of the discriminative tech-nique in [22], which utilizes an ensemble of regression trees and shows accurate real-time performance.
We wish to reiterate that any reliable face detection and landmark localization technique is suitable for this step, as long as the number of false positives is low. Moreover, as aforementioned, this step is equivalent to the current state-of-the-art for deformable face tracking in the litera-ture. However, as we show in our experiments and in Fig. 1, it is inadequate in the case of arbitrary videos, mainly due to the small true positive rate of the detectors and the lim-ited accuracy demonstrated by a generic deformable model as opposed to a person-specific one.
4The DLib C++ Library provides open-source implementations of [22,
23, 16] in http://dlib.net/. The Menpo Project [1] (http:// www.menpo.org/) provides open-source implementations of various deformable models, such as [37, 38, 5, 2, 41, 6], and interfaces easily with the DLib library. Furthermore, the deformable models of [28] and [48] show similar performance but they are more computationally expensive.
Step 2: Person-Specific Face Detection
We assume that the first step of the pipeline (Step 1) re-turns the detected landmarks{s1, . . . , sN1} for a subset of
the original framesIN1 = {I1, . . . , IN1} ⊆ IN0 for which
we obtained a correct detection. In the second step of the pipeline, we train a person-specific face detector using the previously returned detections by building a Deformable Part Model (DPM) [17, 48], which is the discriminative counterpart to the generative Pictorial Structures (PS) [18]. Let us define a tree structure G = (V, E), where V = {v1, . . . , vn} is a set of vertices that correspond to n land-marks (parts) and there exists an edge(vi, vj) ∈ E for each pair of connected landmarks. DPMs aim to learn a mix-ture of M different tree models (Gm = (Em, Vm), m = 1, . . . , M ), each one consisting of a set of parameters that correspond to the appearance of each part and a set of pa-rameters that describe the deformation of the part connec-tions. The purpose of this mixture is to cover a range of different facial poses. DPMs learn the appearance and de-formation parameters using a discriminative training proce-dure, typically via Support Vector Machines (SVM). Then, given an image I, the cost function for the m-th mixture component is expressed as
C(s|I, m) = A(s|I, m) + S(s|I, m) + am A(s|I, m) = n X i=1 wmi TF(ℓi|I) S(s|I, m) = X (vi,vj)∈Em amijdx2ij+ bm ijdxij+ +cm ijdy 2 ij+ dmijdyij (1) where
− A(s|I, m) is the appearance cost of placing each part vm
i ∈ Vmat the image location ℓi, measured as the mismatch between the learnt template (filter) wmi and the extracted image appearanceF(ℓi|I). F denotes a feature vector extraction function (e.g. HoG [15], SIFT [27]) from the neighbourhood around location ℓi. − S(s|I, m) denotes the deformation cost when all the
adjacent parts(vm
i , vjm) : i, j ∈ Emare placed in lo-cations ℓi and ℓj respectively. dxij = xi− xj and dyij = yi − yj are the relative locations (displace-ments) of the i-th part with respect to the j-th one. − αmis a scalar bias (prior) per mixture component. The cost function of Eq. 1 can be expressed in a more con-venient form using a dot product as
C(s|I, m) = wmTy (2) where wmis the vector of the concatenated appearance and deformation parameters
wm= [w1m, . . . , wnm, . . . , amij, bmij, cmij, dmij, . . . , αm] (3)
The final landmark locations are obtained by maximizing Eq. 2 with respect to s and m, as
C∗(I) = max
m,s C(s|I, m) (4) This problem is solved in linear time with respect to the number of parts n, number of components M and image size, by employing an efficient dynamic programming al-gorithm based on the Generalized Distance Transform [18]. There are two main annotation settings for estimating the parameters of DPMs: weakly and strongly supervised.
Weakly Supervised Setting: In this setting, only the bounding boxes of the positive examples{s1, . . . , sN1} and
a set of negative examples are available. The number of parts n and mixtures M are defined a priori, while the part locations and the mixtures in the training set are considered as hidden (latent) information revealed during training [17]. By defining z = [m, ℓm1, . . . , ℓmn] to be a latent vari-able vector, the goal is to learn a vector of parameters wm = [wm1, . . . , wmn]. Since only one of the mixture tree models Gm can be activated, we define a general sparse feature vector y(z) = [0, . . . , y(˜z), . . . , 0], which is the score for the hypothesis ˜z = [ℓm1, . . . , ℓmn]. By denot-ing the space of possible latent values for an example q as Z(q), the classifier that scores this example has the form fwm(q) = maxz∈Z(q)w
T
my(q, z). In order to find the parameters wm, given the set of N1training examples {(q1, y1), . . . , (qN1, yN1)} and yi ∈ {−1, 1}, which are
images with a bounding box annotation, we minimize the SVM objective function using the standard hinge loss
C(q) = 1 2||wm|| 2+ C N1 X j=1 max(0, 1 − yjfwm(q)) (5)
which can be reformulated as arg min wm 1 2||wm|| 2+ C N1 X j=1 max(0, 1 − yjwTmy(q, z∗)) s.t. z∗= max z∈Z(q)w T my(q, z) (6) The minimization of the above cost function is highly non-convex, but becomes convex once the latent information is specified for the positive training examples. In [17], the au-thors propose an alternating optimization procedure, called latent-SVM. Specifically, by fixing wm, the highest scoring latent value for each positive example is determined. Then, by fixing the latent values for the positive set of examples, wmis updated by minimizing the SVM cost of Eq. 5, using stochastic gradient descent.
Strongly Supervised Setting: Under a strongly su-pervised setting, we assume that (i) the mixture compo-nents of the training set are labelled, and (ii) the training set consists of images IN1 with annotated landmarks, i.e.
{s1, . . . , sN1}. Consequently, there are no hidden latent
variables zjto be estimated for each training sample qjand only the model parameters wmneed to be learnt. That is
arg min wm,{ξj} 1 2w T mwm+ C N1 X j=1 ξj s.t. ∀ qj∈ C+, wTmy(qj, zj) ≥ 1 − ξj ∀ qj∈ C−, wTmy(qj, zj) ≤ −1 + ξj ∀ k ∈ K, wk≤ 0 (7)
where K is the set of indices of wm that correspond to the quadratic terms of the shape cost am
ij and cmij. The above constraints state that the score of the positive exam-ples should be larger than1 (minus the small slack variable value ξj), while the score of the negative examples should be less than−1 (adding the small slack variable value ξj), for all part configurations and components. The last set of constraints guarantees that the deformation cost is a proper metric. The family of optimization problems that have the form of Eq. 7 are referred to as structural SVMs. Similar to [48], we employ a modified coordinate descent method, which allows the incorporation of the last set of negativity constraints. This method iterates between finding wmin the dual space and mining for violated constraints according to the current estimate of wmuntil convergence is met.
Output Detections: In our experiments, we employ a strongly supervised DPM trained on the shapes that are re-turned from the generic deformable face detector of the pre-vious step, i.e. {s1, . . . , sN1}. Of course, a weakly
super-vised DPM can also be applied, however, in this case, the final loop of the pipeline (Step 4) that updates the person-specific DPM would have no effect. This is because the bounding boxes cannot be improved from the generative landmark localization procedure of Step 3. Finally, the person-specific nature of the DPM ensures that the returned true positive rate will reamin extremely high, while keeping the false positive rate close to zero. This means that the set of N2 output shapes for which the DPM returns a correct detection will be close to the number of the initial frames, i.e. N1< N2≈ N0.
Step 3: Person-Specific Generative Facial
Land-mark Localization
Due to the use of a highly flexible tree-structure, the strongly supervised DPMs do not achieve state-of-the-art facial landmark localization performance. Furthermore, since the method is discriminative, it is very sensitive to inaccurate landmark detections. For this step, we employ a state-of-the-art generative deformable model that is able to correct inaccurately localized landmarks. Motivated by re-cent developments in automatic construction of generative deformable models [14, 31, 44, 7], we use the state-of-the-art pstate-of-the-art-based Active Appearance Model (AAM) of [38],
referred to as the Gauss-Newton DPM (GN-DPM), and it-eratively improve the appearance model.
GN-DPM [38] is a generative statistical model of shape and appearance that is able to recover a parametric descrip-tion of a face via Gauss-Newton optimizadescrip-tion. By apply-ing Generalized Procrustes Analysis to align the shapes {s1, . . . , sN2} obtained from Step 2 and using Principal
Component Analysis (PCA), we build an orthonormal ba-sis of nS eigenvectors US ∈ R2n×nS plus a mean shape¯s. This linear shape model can be used to generate shape in-stances as s(p) = ¯s + USp, where p = [p1, . . . , pnS]
T is the vector of shape parameters. Similarly, in order to build the appearance model, we first sample all the training images IN2 = {I1, . . . , IN2} in patches
cen-tred around each landmark location using the function F(ℓji|Ij), ∀j = 1, . . . , N
2, ∀i = 1, . . . , n defined for Eq. 1 and concatenate these vectors in order to acquire an na× 1 vectorized part-based appearance representation a(s|I) = [F(ℓ1|I)T,F(ℓ2|I)T, . . . ,F(ℓn|I)]T for all im-ages. Then, the linear appearance model of GN-DPM is trained by performing PCA on the set of part-based appear-ance vectors of all training images that results in a sub-space of nAeigenvectors UA ∈ Rna×nA and the mean ap-pearance ¯a. This model can be used to synthesize shape-free appearance instances, as a(λ) = ¯a+ UAλ, where λ = [λ1, . . . , λN
A]
T is the vector of appearance param-eters. Given a test image I, the optimization problem of GN-DPM employs an ℓ22norm and is expressed as
arg min p,λ
ka(s(p)|I) − ¯a− UAλk2 (8) This can be efficiently solved using the Gauss-Newton al-gorithm. Due to limited space, the optimization process is not included. For more details, please refer to [38].
The goal of this step of the pipeline is to construct an op-timal generative appearance model that best describes the appearance variation that is present in the video sequence. For that purpose, we formulate an iterative optimization problem that aims to minimize the mean GN-DPM fitting ℓ22norm of Eq. 8 over all the frames of the video. In order to facilitate this procedure, we assume that the initial shape and appearance models that are utilized are trained using a database with static images of generic faces. Let us denote the generic shape basis as US, the generic appearance basis as UAand a person-specific appearance basis as BA. Then, the minimization problem is expressed as
arg min ¯ a,[UABA],pi,λi 1 N2 N2 X i=1 ka(si(pi)|Ii) − ¯a− [U ABA]λik2 s.t. [UABA]T[UABA] = Ieye (9) where Ieyedenotes the identity matrix. This procedure it-eratively trains a new person-specific appearance basis BA
based on the current estimate of the N2shapes, combines the generic appearance model UA with BA so that they are orthogonal ([UABA]T[UA BA] = Ieye) and then re-estimates the parameters{pi, λi}, i = 1, . . . , N
2by mini-mizing the ℓ22norm for each frame. Thus, the optimization is solved in an alternating manner in two steps:
(1) Fix{pi, λi} and minimize for {¯a,[U
ABA]}: Hav-ing estimated the current shapes for each frame i = 1, . . . , N2, we build a person-specific appearance sub-space BA from {a(si(pi)|Ii)} and combine it with the generic appearance model UA in order to create an up-dated orthogonal subspace, thus satisfy the constraint [UABA]T[UABA] = Ieye.
(2) Fix {¯a,[UA BA]} and minimize for {pi, λi}: Having created the orthogonal basis that combines the generic and person-specific appearance variation, we now aim to estimate the shape and appearance parameters per frame. Based on Eq. 8, this is done by solving arg minpi,λikai(si(pi)|Ii) − ¯a − [UA BA]λik2, ∀i =
1, . . . , N2using the Gauss-Newton optimization [38]. The above iterative procedure gradually improves the ap-pearance model and therefore also improves the detected landmarks. Note that other state-of-the-art generative meth-ods [37, 2, 5] could also be used. Our experiments showed that, given the person-specific nature of the generative model, only a few iterations are adequate in order to obtain accurate results.
Step 4: Person-Specific Face Detection Update
The output of Steps 2 and 3 is a set of tracked shapes {s1, . . . , sN2} that correspond to each of the frames in
IN2 = {I1
, . . . , IN2}. These frames are the ones for which
a correct detection was returned from Step 2. As explained in Step 2, due to the person-specific setting of the DPM, N2 must be very close if not equal to N0. The next step is to further improve the fitting accuracy and increase the number of true positive detections, if required. This is done by updating the person-specific DPM with the ac-quired set of tracked shapes, which makes the generative deformable model more expressive. Given the fitted shapes {s1, . . . , sN2}, the DPM can be updated in two ways: (i)
re-train it under a strongly supervised setting (Step 2), or (ii) update the parameters of the existing model using the passive-aggressive algorithm of [47]. We have experimen-tally verified that both have similar performance with the latter being considerably faster.
This update step dramatically improves the true posi-tive rate of the detector, as well as the accuracy of the de-formable model. Our experiments showed that one such iteration is enough for the vast majority of videos. The av-erage recall (i.e., true positive rate) we achieve with the pro-posed procedure is more than 98% with almost an 0% of false positive rate. However, for a small number of frames
the person-specific model may not return a face. We treat those cases as a tracking problem and assume a first order Markov dependency. That is, we deal with those frames by initializing their shapes with the shapes of the previous frames and applying the person-specific GN-DPM.
3. Experiments
In this section, we show that our proposed pipeline out-performs all state-of-the-art tracking methods by a substan-tial margin. We also explain how the proposed framework was employed as a semi-automatic tool to annotate the ma-jority of the videos of the 300-VW Challenge [35]2.
3.1. Dataset and Implementation
We found that the majority of the videos currently used for demonstrating face tracking results are both easy and ex-tremely short. Therefore, we carried out experiments on two video categories: (1) videos for which accurate face detec-tion can be achieved by most existing detectors, and (2) very challenging videos where even state-of-the-art detectors fail to detect the face in the majority of frames. The motivation behind this classification is that although existing methods can perform quite well for the sequences of category 1, in category 2 we find a noticeable deterioration in the number of true positives and thus the point-to-point error. The input videos were manually annotated with 68 facial landmark points, using the standard mark-up of Multi-PIE [19].
Category 1: Includes 14 videos (about 30,000 frames in total) that are selected from the testset of the 300-VW Challenge2(scenarios 1 and 2). These videos exhibit large variations in pose, resolution and illumination.
Category 2: Includes 2 very challenging videos with 3058 frames in total. The videos exhibit very challenging conditions, even for the state-of-the-art detectors and track-ers. Roughly 10% of the frames were annotated, which was deemed a sufficient number given the nature of the videos. We note that these videos were so difficult that even manual annotation was challenging.
Implementation details for our pipeline: For Step 1 (i.e., [23, 22]) and Step 3 (i.e., GN-DPM [38]), we used the publicly available implementations in Menpo4. The generic bases of GN-DPM are trained on the iBUG and HELEN datasets [32, 30], approximately 500 images in total. The person-specific update of GN-DPM bases is conducted us-ing 100 randomly selected frames of the video. The GN-DPM is optimized using a multi-scale Gaussian pyramid of 2 levels. The number of employed shape components (nS) is 3 and 12 for the low and high pyramidal levels respec-tively. The number of appearance components (nA) is 50 and 100 respectively and dense SIFT features were used. Finally, the DPM of Step 2 is trained in a strongly super-vised manner.
Figure 3: Indicative qualitative results. Left: Ours. Middle: Deformable detector ([23] + [22]). Right: SPOT [47]
3.2. Pipeline Experimental Analysis
Herein, we measure the impact of each step of our pipeline. We report the mean point-to-point error normal-ized with the face size (i.e., the standard metric that was proposed in [48]) for each category of videos. Figure 5 shows the output of each stage of the pipeline by applying a single loop of Step 4. For the first category of videos, the generic face detector and landmark localization method pro-duced favourable initial results. The person-specific DPM increased the true positive rate and the person-specific GN-DPM of Step 3 improves the landmark localization accu-racy. That is, for the very low error value of 0.03, the per-centage increased from 63% to 72%. Note that a single run of the proposed system is adequate since the update of Step 4 does not change the results. By inspecting the results of the second category videos, we see that the effect of the pipeline is much more evident. That is, the generic face de-tector failed to detect faces in more than 25% of frames, in contrast to the final iteration of our proposed pipeline which managed to reach 100% true positive rate and also improved the facial fitting accuracy. Nevertheless, the up-date of Step 4 does not contribute much, due to the difficulty of the videos.
3.3. Comparison with state-of-the-art
Figure 4 compares our pipeline with other state-of-the-art frameworks. Specifically, we compare with methods that use the standard approach of generic face detection plus generic facial landmark localization. We selected two such representative methods: (1) the method used to initialize our pipeline in Step 1 which consists of [23] followed by [22], and (2) the publicly available implementation of Chehra [9]. For both these methods, when the face detector fails to
re-turn a result we assume a 1st order Markov dependency and initialize from the most recent returned detection. More-over, we also compare with methods that employ state-of-the-art tracking plus facial landmark localization. For this category of techniques, we selected some of the best per-forming tracking-by-detection trackers that have appeared recently and that provide open-source implementations, i.e., SPOT [47], FCT [46] and Correlation tracker (Correl) [16]. For these trackers, the ground truth bounding box of the first frame was provided as the initial input. The generic land-mark localization method used in combination with these trackers is the state-of-the-art method of [22].
Figure 4 shows cumulative error distribution curves based on the 49 internal points of the face (excluding the points of the boundary), as this is the mark-up returned by Chehra. Our pipeline significantly outperforms the rest of the trackers in both categories. For the very challenging videos of category 2, the state-of-the-art trackers fail to de-tect several frames. However, our proposed pipeline man-ages to maintain a high true positive rate.
3.4. Semi-Automatic Annotation Tool
The proposed pipeline was used as a semi-automatic tool in order to annotate86 videos for the 300-VW Challenge2
. The annotation procedure involved the following five steps:
1. Download videos from the Internet (e.g. Youtube). 2. Apply the proposed pipeline as described in Sec. 3.1. 3. Manually correct the annotation for every eighth frame
of each video, if required.
4. Use the corrected frames from the previous step to train and fit a GN-DPM per video (Step 3 of pipeline). 5. Visually inspect the results and finally manually
cor-rect the annotations, if required.
The above procedure manages to return accurate anno-tations in the majority of cases. Therefore, the required amount of manual correction was limited given the large number of frames and the difficulty of the videos. In to-tal, the manual intervention of steps 3 and 5 required837 hours of human labour. The annotations were performed by trained annotators using the web-based landmarking tool that is provided by the Menpo Project5. In order to practi-cally measure the benefit of the proposed semi-annotation tool, we manually annotated6 out of the 86 videos, which required approximately 260 working hours. Thus, we es-timate that the manual annotations of the frames for all 86 videos would have taken around 3727 human working hours, i.e., approximately 4.5× more than the proposed procedure. These numbers highlight the effectiveness of our pipeline for the semi-assisted annotation of large-scale databases of videos. The pipeline and the annotation tool
0.00 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 Normalized Point-to-Point Error
0.0 0.2 0.4 0.6 0.8 1.0 Im ag es Pro po rti on
(a) Video category 1 (average difficulty)
0.00 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 Normalized Point-to-Point Error
0.0 0.2 0.4 0.6 0.8 1.0 Im ag es Pro po rti on
(b) Video category 2 (very challenging)
Figure 4: Comparison of our pipeline with state-of-the-art techniques, evaluated on 49 facial landmark points.
0.00 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 Normalized Point-to-Point Error
0.0 0.2 0.4 0.6 0.8 1.0 Im ag es Pro po rti on
(a) Video category 1 (average difficulty)
0.00 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 Normalized Point-to-Point Error
0.0 0.2 0.4 0.6 0.8 1.0 Im ag es Pro po rti on
(b) Video category 2 (very challenging)
Figure 5: Illustration of the contribution of each step of the proposed pipeline, evaluated on 49 facial landmark points.
were built within the Menpo Project [1] and will be pub-licly available soon. The annotation procedure was facili-tated using the cloud service of [24].
4. Conclusions
In this paper we presented the first, to the best of our knowledge, pipeline that can perform long-term deformable face tracking in challenging videos. The pipeline takes ad-vantage of generic face detection and landmark localiza-tion algorithms to iteratively train powerful and accurate person-specific face detectors and landmark localization techniques. The pipeline was used as a semi-automatic tool to annotate most of the videos of the 300-VW Challenge2.
Acknowledgements
Grigorios G. Chrysos is supported by the EPSRC project EP/L026813/1 Adaptive Facial Deformable Mod-els for Tracking (ADAManT). Epameinondas Antonakos is funded by the EPSRC project EP/J017787/1 (4D-FAB).
Stefanos Zafeiriou is partially supported by the 4D-FAB and ADAManT projects. Patrick Snape is funded by a DTA from Imperial College London and by a Qualcomm Inno-vation Fellowship.
References
[1] J. Alabort-i-Medina, E. Antonakos, J. Booth, P. Snape, and S. Zafeiriou. Menpo: A comprehensive platform for para-metric image alignment and visual deformable models. In
ACMMM. ACM, 2014.
[2] J. Alabort-i-Medina and S. Zafeiriou. Bayesian active ap-pearance models. In CVPR, 2014.
[3] J. Alabort-i-Medina and S. Zafeiriou. Unifying holistic and parts-based deformable model fitting. In CVPR, pages 3679– 3688, 2015.
[4] E. Antonakos, J. Alabort-i-Medina, G. Tzimiropoulos, and S. Zafeiriou. Hog active appearance models. In ICIP, Octo-ber 2014.
[5] E. Antonakos, J. Alabort-i-Medina, G. Tzimiropoulos, and S. Zafeiriou. Feature-based lucas-kanade and active appear-ance models. IEEE TIP, 24(9):2617–2632, 2015.
[6] E. Antonakos, J. Alabort-i-Medina, and S. Zafeiriou. Active pictorial structures. In CVPR, 2015.
[7] E. Antonakos and S. Zafeiriou. Automatic construction of deformable models in-the-wild. In CVPR, 2014.
[8] A. Asthana, S. Zafeiriou, S. Cheng, and M. Pantic. Robust discriminative response map fitting with constrained local models. In CVPR, 2013.
[9] A. Asthana, S. Zafeiriou, S. Cheng, and M. Pantic. Incre-mental face alignment in the wild. In CVPR, 2014.
[10] A. Asthana, S. Zafeiriou, G. Tzimiropoulos, S. Cheng, M. Pantic, et al. From pixels to response maps: discrimi-native image filtering for face alignment in the wild. IEEE
T-PAMI, 37(6):1312–1320, 2015.
[11] P. N. Belhumeur, D. W. Jacobs, D. Kriegman, and N. Kumar. Localizing parts of faces using a consensus of exemplars. In
CVPR, 2011.
[12] C. Cao, Q. Hou, and K. Zhou. Displaced dynamic expression regression for real-time facial tracking and animation. ACM
TOG, 33(4):43, 2014.
[13] D. Chen, S. Ren, Y. Wei, X. Cao, and J. Sun. Joint cascade face detection and alignment. In ECCV. Springer, 2014. [14] X. Cheng, S. Sridharan, J. Saragih, and S. Lucey. Rank
min-imization across appearance and shape for aam ensemble fit-ting. In ICCV, 2013.
[15] N. Dalal and B. Triggs. Histograms of oriented gradients for human detection. In CVPR, 2005.
[16] M. Danelljan, G. H¨ager, F. S. Khan, and M. Felsberg. Ac-curate scale estimation for robust visual tracking. In BMVC, 2014.
[17] P. F. Felzenszwalb, R. B. Girshick, D. McAllester, and D. Ra-manan. Object detection with discriminatively trained part-based models. IEEE T-PAMI, 32(9):1627–1645, 2010. [18] P. F. Felzenszwalb and D. P. Huttenlocher. Pictorial
struc-tures for object recognition. IJCV, 61(1):55–79, 2005. [19] R. Gross, I. Matthews, J. Cohn, T. Kanade, and S. Baker.
Multi-pie. IMAVIS, 28(5):807–813, 2010.
[20] V. Jain and E. G. Learned-Miller. Fddb: A benchmark for face detection in unconstrained settings. UMass Amherst
Technical Report, 2010.
[21] Z. Kalal, K. Mikolajczyk, and J. Matas. Tracking-learning-detection. IEEE T-PAMI, 34(7):1409–1422, 2012.
[22] V. Kazemi and J. Sullivan. One millisecond face alignment with an ensemble of regression trees. In CVPR, 2014. [23] D. E. King. Max-margin object detection. arXiv preprint
arXiv:1502.00046, 2015.
[24] V. Koukis, C. Venetsanopoulos, and N. Koziris. ˜ okeanos: Building a cloud, cluster by cluster. Internet Computing,
IEEE, 17(3):67–71, 2013.
[25] V. Le, J. Brandt, Z. Lin, L. Bourdev, and T. S. Huang. Inter-active facial feature localization. In ECCV. Springer, 2012. [26] J. Li and Y. Zhang. Learning surf cascade for fast and
accu-rate object detection. In CVPR, 2013.
[27] D. G. Lowe. Object recognition from local scale-invariant features. In ICCV, 1999.
[28] M. Mathias, R. Benenson, M. Pedersoli, and L. Van Gool. Face detection without bells and whistles. In ECCV. Springer, 2014.
[29] S. Ren, X. Cao, Y. Wei, and J. Sun. Face alignment at 3000 fps via regressing local binary features. In CVPR, 2014. [30] C. Sagonas, E. Antonakos, G. Tzimiropoulos, S. Zafeiriou,
and M. Pantic. 300 faces in-the-wild challenge: Database and results. IMAVIS, 2015. in press.
[31] C. Sagonas, Y. Panagakis, S. Zafeiriou, and M. Pantic. Raps: Robust and efficient automatic construction of person-specific deformable models. In CVPR, 2014.
[32] C. Sagonas, G. Tzimiropoulos, S. Zafeiriou, and M. Pantic. 300 faces in-the-wild challenge: The first facial landmark localization challenge. In ICCV’W, 2013.
[33] C. Sagonas, G. Tzimiropoulos, S. Zafeiriou, and M. Pantic. A semi-automatic methodology for facial landmark annota-tion. In CVPR’W, June 2013.
[34] E. Sariyanidi, H. Gunes, and A. Cavallaro. Automatic anal-ysis of facial affect: A survey of registration, representation and recognition. IEEE T-PAMI, 37(6):1113–1133, 2015. [35] J. Shen, S. Zafeiriou, G. Chrysos, J. Kossaifi, G.
Tzimiropou-los, and M. Pantic. The first facial landmark tracking in-the-wild challenge: Benchmark and results. In ICCV’W, 2015. [36] Y. Taigman, M. Yang, M. Ranzato, and L. Wolf. Deepface:
Closing the gap to human-level performance in face verifica-tion. In CVPR, 2014.
[37] G. Tzimiropoulos, J. Alabort-i-Medina, S. Zafeiriou, and M. Pantic. Generic active appearance models revisited. In
ACCV. Springer, 2013.
[38] G. Tzimiropoulos and M. Pantic. Gauss-newton deformable part models for face alignment in-the-wild. In CVPR, 2014. [39] P. Viola and M. J. Jones. Robust real-time face detection.
IJCV, 57(2):137–154, 2004.
[40] L. Wolf, T. Hassner, and I. Maoz. Face recognition in un-constrained videos with matched background similarity. In
CVPR, 2011.
[41] X. Xiong and F. De la Torre. Supervised descent method and its applications to face alignment. In CVPR, 2013.
[42] J. Yan, Z. Lei, L. Wen, and S. Z. Li. The fastest deformable part model for object detection. In CVPR, 2014.
[43] B. Yang, J. Yan, Z. Lei, and S. Z. Li. Aggregate channel features for multi-view face detection. In IJCB, 2014. [44] L. Zafeiriou, E. Antonakos, S. Zafeiriou, and M. Pantic.
Joint unsupervised face alignment and behaviour analysis. In ECCV. Springer, 2014.
[45] S. Zafeiriou, C. Zhang, and Z. Zhang. A survey on face detection in the wild: past, present and future. CVIU, 138:1– 24, September 2015.
[46] K. Zhang, L. Zhang, and M.-H. Yang. Fast compressive tracking. IEEE T-PAMI, 36(10):2002–2015, 2014.
[47] L. Zhang and L. van der Maaten. Preserving structure in model-free tracking. IEEE T-PAMI, 36(4):756–769, 2014. [48] X. Zhu and D. Ramanan. Face detection, pose estimation,
