Animating a Human

The geometry that we obtain from any of the method described in Sect. 2.2.1, should be somehow animated to reconstruct the motion of the human actor in the video. In this thesis we make use of two techniques, using the kinematic skeleton or deformation.

We make use of the animation based on the kinematic skeleton in the relightable free-viewpoint video project, Chapters 5 and 6. In this project, first a kinematic skeleton is implanted into the geometry of the single skin template human model, Fig. 2.3c. Thereafter, the skeleton is attached to the surface by assigning the weights to each vertex of the geometry in accordance with its relative position to each bone. A bone would exert more influence on its nearby vertices. This influence is represented by the weights, which control the deformation of the mesh as the joints are rotated. Each vertex can be influenced by multiple bones and the weights from each bone are blended. The technique of assigning the weights in this way is commonly called linear blend skinning [Baran07]. Finally, the motion description in terms of joint parameters is automatically estimated using a silhouette based analysis-through-synthesis method (Sect. 6.3).

Another approach for animating the model would be to use mesh-deformation methods [Botsch07]. These methods are employed to great effect in performance capture of humans [de Aguiar07a] [de Aguiar08]. In our work of parametrization- free animation reconstruction using dense 3D correspondences, we make use of a mesh deformation approach to animate the reconstructed visual hull, Chapters 9 and 10. Our solution is independent of any specific deformation approach, therefore we refer the reader to a recent survey in the area of surface deforma- tion [Botsch07].

Chapter 3

Multi-view Video Studio

This chapter describes our recording studio. First, the studio room, the camera system and the lighting setup are described. Thereafter, the acquisition pipeline is presented, with all necessary steps to generate the input data for the projects described in this thesis.

All of the projects presented in this thesis require high quality multi-view video data as input. This data is recorded in our multi-view video studio, where we simultaneously capture video streams from eight synchronized video cameras. In this chapter we will present our multi-view video studio in detail. The stu- dio is an extension of [Theobalt03], which was a simpler multi-view acquisition setup. We present our new acquisition studio, which provides high quality data that are recorded not only using the calibrated cameras but also under completely calibrated illumination conditions. These data are the main requirement of our work on relightable free-viewpoint video (Chapters 5 and 6), and subsequently high quality reconstruction of time-varying geometry (Chapters 7 and 8). The ac- quisition setup of the studio is enhanced with the addition of hiqh frame rate and high resolution cameras along with the better lighting setup, which facilitate us greatly in the reconstruction of high quality surface models. High frame rate and high resolution data were also invaluable for our work on the parametrization-free animation reconstruction using dense 3D correspondences (Chapters 9 and 10). We will start this chapter with a review of related multi-view acquisition sys- tems. Thereafter, we will describe the recording studio, and discuss our camera and lighting system that is installed in the studio. Finally, we will present the

20 Chapter 3: Multi-view Video Studio

acquisition process, which is comprised of camera, color and lighting calibration, background subtraction and finally the actual recording of the human actor.

3.1

Related Multi-view Acquisition Facilities

Multi-view data is used in variety of research areas. Various setups for their ac- quisition exist, based on the specific needs of the research. The project presented in this thesis are versatile in the sense that they encompass many research areas that require these data. Therefore, our multi-view video studio is designed in such a way that the specific requirements for data are not compromised.

Image based reflectance estimation requires very high quality image data. For estimating the surface reflectance models of real-world object, a series of im- ages obtained from different viewing directions and taken under different inci- dent illumination conditions are required. For static scenes, acquisition setup using high quality photo cameras and a set of light sources have been pro- posed [Ward92, Goesele00]. [Debevec00] presented a light stage to capture the reflectance field of animatable face model. [Einarsson06] extended it further by using a large light stage, a tread-mill where the person walks, so that they can acquire simple motion and reflectance field of humans. Unfortunately, their setup can only process simple periodic motions, such as walking. In contrast our multi- view video studio allows the extension of the photo camera based reflectance es- timation method into video based dynamic reflectometry, without any restriction on the type of motion.

Multi-view video streams are readily used in the area of video-based motion cap- ture. In our work we focus on marker-less motion capture, because it allows recording of the human actor without any optical markers attached on the body. Video acquisition in a 3D room that allows recording with up to 48 cameras is presented by [Kanade98]. Systems for motion acquisition using reconstructed volumes are presented in [Cheung00, Borovikov00, Luck02, Brostow04]. Com- mercial solutions for marker-less motion capture are now also available [Motion]. For an extensive review of video-based motion acquisition systems, we would like to refer the interested reader to [Poppe07].

Another research area that makes use of multi-view video streams is 3D video. In addition to capturing the motion, multi-view video streams can be used to recon- struct the dynamic shape and appearance models of the human actor. This enables the user to change the viewpoint of the scene during the rendering. [Narayanan98] made use of 50 cameras and reconstructed 3D models of dynamic scenes using