Model Fitting via Soft Constraints

Marker-based motion capture (MoCap) is widely used to animate human characters in films and games. In this section, an application of fitting the Shell PGA model to motion capture data is presented by means of a soft penalty approach. In particular, this allows us to reconstruct a discrete shell from (potentially noisy) input data from a motion capture device. In this case, the input data is given as a vector of L sparse marker positions, i.e. x = (x`)`=1,...,L,

corresponding to vertex positions_X`(s) on the mesh s. Knowing these correspondences, we

measure the mismatch of some discrete shell s _{∈ M and the given landmarks by F}x[s] =

PL

`=1kX`(s)− x`k 2

R3. Following Chapter 4, we consider shell deformation energy W[s,P[s]]

as a prior for the identification of a reconstructed discrete shell. Hence, we seek a minimizer s of the model fitting energy given by

M

γ = 0

γ =_∞

x = (x

)

R

P

[s

]

P[s]

s

s

M

s

Figure 6.1: Right: Projection of an unseen shape s onto the model space_MJ: scale s to sloc,

project sloclocally toPloc[sloc]∈ CJ, and finally rescale to getP[s] ∈ MJ. Left: Model fitting

of s∗driven by sparse landmarks X _{∈ R}3Ldepending on fitting parameter γ > 0.

for some weight γ > 0, which controls the strength of markers fitting. In other words, a large value of γ will encourage exact matching of the markers and their correspondence of s. Please refer to Fig. 6.1 for the visual illustration of model fitting.

Algorithm 5 Motion capture using Shell PGA model

1: Input: MoCap markers xf =1,...,F = (x`)`=1,...,L∈ R3, average ¯s, principal variations p−J,...,J,

marker weight γ

2: Output: reconstructed dense shape sf =1,...,F

3: initialise_P[sf =1] with average shape ¯s

4: // optimise s for frames f = 2, . . . , F

5: for f := 2 to F do

6: minimising W[sf,_P[sf −1]] + γ_Fx[sf]

7: update_P[sf] by projecting sf on p−J,...,J

8: re-estimate sf using (Eq. 6.1)

9: end for

For the numerical solution of this problem, we make use of the following alternating scheme (based on the initial guess_{P[s] = ¯s or using temporal information): First, we minimise Eq. (6.1)} in s for fixed_{P[s]. If necessary, we re-compute P[s] (see Sec. 4.3.4) and go back to the first step.} In our experiments this scheme quickly converges and only very few iterations already give very satisfactory fitting results. In practice, we use two iterations for the results shown.

Figure 6.2: Qualitative results of fitting to motion capture data. Frames from original sequence (top) shown with corresponding reconstruction (bottom).

Figure 6.3: Comparison of reconstruction from motion capture data with the MoSh model [10]. Although MoSh (top) is trained on more than 5,000 scans and uses an additional skeleton model, our method with K = 4 (bottom) obtains similar results using 10 principal variations only.

Qualitative results of model fitting. In Figures 6.2 and 6.3 we show qualitative results of fitting to 41 markers in sequences from the CMU mocap dataset and 89 markers from MPI MoSh dataset [10]. Fig. 6.2 shows the result in which the learnt body model has quite different geometry to that of the performer. Note that the video frames are just shown for comparison - we use only the 3D markers positions as input. Our fitted model is still able to capture the dynamic poses of the performance.

In Fig. 6.3 we compare against the method [10]. It should be noted that their method uses a model of substantially higher complexity than ours. In particular, it is trained on 3,803 body scans in neutral pose and 1,832 body scans in dynamic poses and uses a 19 parameter skeleton model and retains up to 300 dimensions of the statistical deformation model (10 used in Fig. 6.3). Our result is obtained using a model trained on 20 scans of a single person (chosen to match the body shape of the performer), is entirely mesh-based (we have no articulation model) and we also use only 10 principal variations (J = 10). Nevertheless, our results are qualitatively very similar to [10].

γ = 0.1

γ = 1.0

γ = 5.0

Figure 6.4: Selected frames to demonstrate the effect of marker weight (Top: 0.1; middle: 1; bottom: 5). Red: model correspondence; Green: MoCap markers.

Effect of parameter γ. To analyse the effect of γ which controls the strength of markers fitting term in our application, a series of values for γ are tested in the model fitting process.

Several frames are selected and shown in Fig. 6.4 for illustrations. As γ gets bigger, we can see that MoCap markers (in red) and the model correspondence (in green) are getting closer, while the downside is that it can also distort the model. For example, in face and arm elbows areas we can observe some undesired artefacts. Therefore, it is vital to choose a suitable γ for nice results. In our model fitting experiments, we empirically choose γ = 0.5 to have a trade-off between marker closeness and overall model quality.

50021

50002

50009

Figure 6.5: Selected frames to show the results of fitting different characters to a “Dance” sequence. Characters from top to bottom correspond: 50021, 50002, and 50009, from Dyna [11] dataset.

Fitting with different characters. Given some motion capture data, the model fitting pro- cess is quite robust and not restricted to the underlying identities of the performer. Theoretically, we can fit the model learned from an arbitrary character to the marker data, provided that the correspondence between model mesh and markers are determined in a meaningful way. Indeed, we consider this to be a very nice property for animating motion capture data with a chosen character.

In Fig. 6.5, we show such a comparison of fitting three different characters to the same “dance” sequence. The reconstructed meshes in each case look physically valid and retain the