Color Interpolation

Since the reconstructed texture maps are composites from multiple camera images, discontinuity artifacts usually become visible. The reason for those artifacts is that the true surface appearance varies by distance and incidence angle while our simple model assumes a Lambertian BRDF. For a consistent texturing we want to minimize the visibility of these discontinuity artifacts. We approach this problem by using a blending technique, which globally adjusts the color of all pixels.

Figure 5.12: Our poisson blending approach uses a guidance vector field V com-posed from color gradients and boundary constraints to guarantee smooth bound-aries between texture regions reconstructed from different photos Ω1. . . Ω₃
. The result is a blended composite with no visible discontinuities at region boundaries.

Our algorithm extends the ideas of [Pérez et al., 2003] to use a Poisson formulation for the multi-view blending problem. The procedure is as follows: for a texture with regions reconstructed from n camera images, we can treat the regions as separate functions: f_1:n. Now, let Ω_1:n be the definition space of f_1:n, ∂Ω_i,j be the boundary between Ωi and Ωj, and ∂Ωi the texture boundary of the i^th texture. Finally, we define V to be a guidance vector field defined over Ω1:n. See Figure 5.12 (left) for an illustration of this notation.

Our goal is to find a new set of functions f_1:n⁰ which have the same definition space as f1:nand no visible discontinuities at their boundaries. We cast this problem as a membrane interpolant that satisfies:

f_1:n⁰ = min

f_1:n

X

i

x

Ωi

|∇fi− V|² , (5.15)

with the Dirichlet boundary conditions fi |∂Ω_i,j= fj |∂Ω_i,j and fi |∂Ω_i= fi|∂Ω_i. We set the guidance vector field V to equal the derivatives of f1:n, which means we constrain the derivatives of f1:nto be the same as the derivatives of f1:n. The

118 5. PHOTO-REALISTICTEXTURERECONSTRUCTION

first boundary constraint guarantees a smooth boundary between texture regions, while the second constraint is necessary because the gradient operator is invariant through multiplicative factors. The solution of Equation 5.15 is the unique solution of the following Poisson equation:

∇ · ∇f1:n= 4f1:n= ∇V over Ω1:n, (5.16)

with the same boundary conditions as Equation 5.15. In the discrete texture do-main, this can be efficiently solved as a sparse linear system. For each pixel p ∈ T , let Np be the set of its 4-connected neighbors and let {p, q} be a pixel pair such that q ∈ N_p. The boundary between the two regions i and j is now defined as:

∂Ωi,j= {p ∈ Ωi: Np∩ Ωj6= ∅} (5.17)

and the texture boundary of region i is:

∂Ω_i= {p ∈ Ω_i: |N_p∩ T | < 4} . (5.18)

The finite difference discretization of Equation 5.15 yield the following quadratic optimization problem:

where vpqequals the projection of the guidance vector field V onto the oriented edge−→

The solution for this optimization problem satisfies the following simultaneous linear equations for all i ∈ 1 : n and p ∈ Ωi:

5.3. MULTI-VIEWTEXTURERECONSTRUCTION 119

(a) Stitched textures. (b) Naïve blending. (c) Poisson blending.

(d) Texture panorama without blending.

(e) Texture panorama blended with poisson blending.

Figure 5.13: The texture blending globally optimizes the texture color and re-moves discontinuities at boundaries between texture regions reconstructed from different camera images. Compared to a naïve blending method, such as aver-aging, the proposed algorithm results in consistent texture maps without visible discontinuities.

This sparse, symmetric, positive-definite system can be solved using a sparse linear solver. We directly solve the system through LU decomposition with partial

piv-120 5. PHOTO-REALISTICTEXTURERECONSTRUCTION

oting and forward/back substitution. For large T , the size of the resulting system can easily reach an order of 10⁶and LU decomposition becomes intractable. In this case, a well-known iterative solver (e.g., Gauss-Seidel or Conjugate Gradient) can be used to achieve an approximate solution within a few iterations. Results of this blending approach are presented in Figure 5.13.

5.4 Conclusion

In this chapter, we described algorithms for the reconstruction of the surface ap-pearance and, in particular, the reconstruction of a two-dimensional diffuse texture.

The texture stores a constant BRDF for each surface point and is mapped onto the reconstructed 3D surface for visualization. This technique results in a very realis-tic reconstruction and reproduction of the surface appearance and greatly enhances the visual impression by adding more realism.

Our reconstruction approach consists of the following steps: surface partition-ing (segmentation and slicpartition-ing), surface unfoldpartition-ing, color reconstruction, and color blending. The surface partitioning is a pre-processing step used to slice a com-plex surface mesh in pieces with less comcom-plexity. This is necessary to permit an unfolding of each piece onto a plane. We presented an efficient split-and-merge approach which first segments the mesh into almost regions based on surface nor-mal similarity followed by an iterative merging of mesh segments to avoid over segmentation. Empirically, we found that this procedure results in segments with low complexity. The unfolding step computes a mappping which maps the surface of each segment onto a corresponding texture. We presented an approach using a least-squares conformal mapping computed from the surface triangles. This map-ping minimizes geometric distortions, which is very important as to avoid artifacts in the texture reconstruction. We demonstrated that this approach globally mini-mizes distortion over all surface triangles and clearly outperforms naïve orthogo-nal mappings. The texture domain is then discretized and the color reconstruction step retrieves a color for each texture pixel. This is done by finding the best camera images that carry information about the surface area corresponding to the texture pixel. This procedure results in one texture map per mesh segment composed of multiple camera images. This compositing may lead to visible discontinuity arti-facts. Hence, we proposed a novel variational blending technique which globally adjusts the color of all pixels within one texture and eliminates compositing arti-facts. This blending method enforces smooth transitions between texture regions reconstructed from different camera images and employs a vector guidance field constructed from the texture gradients to propagate those constraints throughout the texture.

5.4. CONCLUSION 121 In the quest for more realistic imagery, we can conclude that reconstruction of the surface appearance is a key component of the 3D modeling system. Reconstruct-ing diffuse textures and mappReconstruct-ing those onto the reconstructed surface mesh is a relatively efficient mean to create a realistic surface appearance. We presented a pipeline for a way of automatically reconstructing such texture maps from multiple camera images. For a visualization system it is typically true that realism demands complexity, or at least the appearance of complexity. We demonstrated that with diffuse texture maps, the appearance of complex 3D models can be achieved with-out actually modeling and rendering every 3D detail of a surface.

6 Results and Applications

6.1 Photo-Realistic Reconstruction of Indoor Environments

The 3D reconstruction and visualization of architectural scenes is an increasingly important research problem, with large scale efforts underway to recover models of cities at a global scale (e.g., Street View, Google Earth, Virtual Earth). The process of creating a realistic virtual model of an environment begins with modeling the geometry and surface attributes of objects in an environment along with any lights.

An image of the environment is subsequently rendered from the vantage point of a virtual camera. Great effort has been expended to develop CAD systems that allow the specification of complex geometry and material attributes. Similarly, a great deal of work has been undertaken to produce systems that simulate the propaga-tion of light through virtual environments to create realistic images. Unfortunately, current methods of modeling existing architecture, in which a modeling program is used to manually position the elements of the scene, have several drawbacks.

First, the process is extremely labor-intensive, typically involving surveying the site, locating and digitizing architectural plans (if available), or converting exist-ing CAD data (again, if available). Second, it is difficult to verify whether the resulting model is accurate. Most disappointing, though, is that the renderings of the resulting models are noticeably computer-generated; even those that employ liberal texture-mapping generally fail to resemble real photographs.

The system presented in this thesis can be used to effectively create photo-realisic reconstructions of large indoor environments. A number of experiments have been conducted using the described approach. In particular, we have created a 3D model of Bosch’s office in Palo Alto. Snapshots of this model are depicted in Figure 6.1.

The largest fraction of the time required for the complete 3D reconstruction pro-cess was spent on the data acquisition; 6 hours were nepro-cessary to scan one office floor by taking 127 scans (approx. 3 min per scan). Registration, surface and tex-ture reconstruction took around 100 minutes for the Bosch dataset on a standard desktop computer (3.4 GHz, 4GB RAM). About 70% of this time was spend on IO operations on the 8GB of compressed raw data. The registration was performed on a sub-sampled dataset and took 20 minutes to converge. Projecting the regis-tration results onto the high-resolution data yielded good results. Our volumetric

124 6. RESULTS ANDAPPLICATIONS

(a) Reconstructed 3D Model of Bosch’s RTC.

(b) A conference room rendered from a virtual camera.

(c) Kitchen.

Figure 6.1: Reconstruction of photo-realistic models from real indoor environ-ments. The reconstructed office model presented here consists of 28,167,234 ver-tices and 54,745,336 triangles covering an area of 50 m by 140 m.

6.2. RAPIDINSPECTION 125