Local Depth Estimation

To estimate detailed depth, our model relies on image superpixels. Each superpixel is represented as a plane in 3D, which translates the depth estimation problem into finding the best plane parameters for each superpixel. In particular, here, we encode each plane with the depth of its centroid and its normal direction.

More specifically, letY={y₁,y2, ...,yNs}be the set of discrete variables represent-

ing Ns superpixels in an image, where each yi can take values from a discrete state

space S. We define this state space by quantizing the range of valid depth for the superpixel centroid into V values, with the range determined from the maximum and minimum depth of the training data. Furthermore, we make use of the Man- hattan world assumption, and restrict the superpixels normal direction to 3 possible dominant directions, defined by the vanishing point estimation method of [Rother,

§3.3 Methodology 25

Candidate  Neighbor  Images  &  Depths

Input  Query

Target  Depths

Figure 3.3: An example for global neighbor search: We show several neighbor im- ages to a query image retrieved using image global context features. These neighbors

depict either analogous layout, or objects similar to those in the input image.

2002]. This lets us define the first energy term in Eq. (3.1) as El(Y) =

∑

p

φp(yp) +

∑

φp,q(yp,yq), (3.2)

whereφp is a unary potential encoding the cost of assigning labelypto superpixelp,

andφp,q(yp,yq)is a pairwise potential encouraging coherence across the superpixels.

The unary potential is based on the regression term in [Liu et al., 2014]. Due to the fact that superpixels occupy very small areas, and their features are insufficient to find the neighbors similar in depth, we rely on global context. To this end, we first retrieveK candidate training images similar to the input image by nearest-neighbor search based on a combination of distances on mutiple features, i.e., L2 distances on GIST [Oliva and Torralba, 2001] and ObjectBank [Li et al., 2010] features, and

χ2-distance1 on PHOG [Bosch et al., 2007]. Among these features, GIST and PHOG

capture low-level scene representations, which summarize the gradient distribution of a scene image, while ObjectBank captures a high-level image representation, i.e., a response map to a set of pretrained object detectors. In practice, we select the can- didate images from the training data in a leave-one-image-out manner. An example is shown in Figure 3.3. With the neighbor training images at hand, we compute, for each superpixel of the input image, the plane parameters of the corresponding area in each candidate, by fitting a plane from ground truth depths using RANSAC. We then use Gaussian Process (GP) regressors [Bishop, 2006] to predict the plane param- eters of the superpixel of interest from these plane parameters (i.e., one regressor for each of the four plane parameters). The GP regressors rely on an RBF kernel.

1_The

(a) (b) (c)

Figure 3.4: An example of boundary occlusion map: (a) Image; (b) depth gradients, where color indicates gradient (red is large, blue is small).; and (c) ground truth

boundary occlusion map. The boundary occlusion is labelled in purple.

Given the regressed depth on superpixel center and the regressed surface nor- mal, we then can retrieve the depth of arbitrary points on the superpixel by the following procedure. Let [nx,ny,nz] be the normal vector, dc the depth of the cen-

troid of a superpixel. The corresponding 3D coordinates [Xc,Yc,Zc]then are given

by dcK−1[xc,yc, 1]T, where [xc,yc] are the 2D coordinates of the centroid, and K is

the camera intrinsic matrix. Let f = [nx,ny,nz,e] denote the paramters of a surface

plane defined on the superpixel. As we know, for any point on a plane, we have fT[X,Y,Z, 1] = 0. Following this rule, we can calculate the fourth parameter of the plane as e = −nxxc+nyyc+nzzc. Then, for an arbitrary point on the superpixel,

with 2D coordinates [x,y], its depth d can be calculated as the intersection of the plane with its visual ray, i.e.,d= _[ −e

nx,ny,nz]TK−1[x,y,1].

Letdi_r,pbe the depth of theith pixel of superpixelp, estimated from the regression results. We define our unary potential as

φp(yp) = 1 Np Np

∑

where Np is the number of pixels in superpixel p and dip is the depth of pixel i in

superpixel pfor a particular stateyp.

The pairwise term φp,q relies on an occlusion classifier trained on the features

of [Hoiem et al., 2007b]. In practice, we assign non-occlusion/occlusion labels ac- cording to the average depth gradient on the edges, as shown in the example in Fig- ure 3.4. The two-class classifier is trained based on boosted decision trees [Collins et al., 2002], with a non-occlusion/occlusion sampling ratio of 3.0. Given the pre- dicted occlusion label opq for the boundary between two neighboring superpixels p

§3.3 Methodology 27 Input& Image Re gio n&I Re gio n&2 … Re gio n& ! " Dataset GIST PHOG ObjectBank Neighbor&Images Neighbor&Images Neighbor&Images Feature51 Neighbor& Regions Distribution&of& normals on&regions Neighbor& Regions Feature5M … … Distribution&of& depths&at&centroids& of&regions&

Figure 3.5: Procedure for generating Mid-level unaries: This depicts the precedure for generating mid-level unaries on regions.

andq, this potential is expressed as

φp,q(yp,yq) =wl·          0 ifopq=1 gpqknp(yp)−nq(yq)k2+ 1 Npq∑ Npq j=1(d j p(yp)−dqj(yq))2 ifopq=0 (3.4)

where Npq is the number of pixels shared by superpixels p and q, np(yp) is the

normal corresponding to a particular state yp, and gpq is a weight based on the

image gradient on the boundary of the superpixels, i.e., gpq = exp(−µpq/σ) with µpq the mean gradient on the boundary. We assume that two neighbor superpixels

have more chance to belong to two hinged planes, while the image gradient on the boundary is large, and in this case we penalize less their surface normal difference.

While inspired by [Liu et al., 2014], the energy described above includes at most pairwise terms, and therefore allows us to perform inference more efficiently. Impor- tantly, however, this energy still reasons at a local level. In the following, we present our approach to incorporate higher-level scene structures via the additional terms in Eq. (3.1).