Inverse Graphics

Modern graphics engines (e.g., game engines like CryEngine [2] and Lumberyard [4]) leverage dedicated hardware setups and provide real-time renderings with stunning level of realism. Some sample renderings from modern game engines [2, 4] are shown in Fig. 2.1(a). Vision problems can be tackled with posterior inference (Eq. (2.1)) in such accurate computer graphics systems. This approach for solving vision problems can be understood as Inverse Graphics [22]. The target variables y correspond to the input to the graphics system and the observation variables are the output x= G(y, θ ). The de- terministic graphics system G can be converted into a probabilistic generative model by defining a prior over the target variables (input to the graphics system) P(y) and also defining an approximate likelihood function P(x|G(y,θ)) characterizing the model im- perfections. If the model imperfections are neglected, the likelihood can be given using the delta function P(x_{|y) = δ (x − G(y,θ)). An example ‘inverse graphics’ problem,} which we tackle in the next chapter, is depicted in Fig. 2.2, where the graphics engine renders a depth image given 3D body mesh and camera parameters, and a vision problem would be the inverse estimation of body shape given a depth image.

Modern graphic engines are based on physics principles and thus most of the rendering parameters θ are set to mimic the real world physics. Learning in graphics generative

2.2 Generative Vision Models

models mainly involves learning the prior P(y) over the target world properties which are input to the graphics system. Depending on the type of model, several learned priors are proposed in the literature. An example is the SCAPE model [11, 117] for modeling the prior over human body shape and pose for the example shown in Fig. 2.2.

Since modern rendering engines involve complex non-linear functions, it is usually not feasible to obtain a closed-form solution for the posterior distribution (Eq. (2.1)). Even several approximate inference techniques like variational optimization techniques [147] cannot be easily employed for posterior inference in complex graphics systems. ‘Monte Carlo’ sampling provides a generic inference technique for such complex models and can be used even when the internals of the graphics systems are not known. The aim of sampling methods is to characterize the posterior distribution with independent and identically distributedsamples. There exists many Monte Carlo sampling strategies such as uniform sampling, rejection sampling, importance sampling [107, 216] etc. Here, we limit our discussion to the widely used Markov Chain Monte Carlo (MCMC) sampling techniques.

MCMC Sampling

MCMC sampling [181] is a particular instance of sampling methods that generates a sequence of target variables (samples) by simulating a reversible Markov chain.

y1→ y2→ y3→ ··· → yn (2.2)

This Markov chain of samples approximates the target distribution (posterior distribu- tion in the case of ‘inverse graphics’). The Markov property states that at every sequence step t, given the present sample yt in the sequence, the next sample yt+1 is independent

of all the previous samples P(yt+1|yt, . . . , y1) = P(yt+1|yt).

Let us denote the target distribution with π(_{·). In the inverse graphics setting, the target} distribution is the posterior π(y) = P(y_{|x,θ). And, let us denote the transition probability} between the two states (samples) in the Markov chain be T(yt+1|yt) or T (yt → yt+1)∈

[0, 1]. One way to ensure that the Markov chain is reversible and converging to the target distribution is to check whether the following detailed balance condition holds for any two states yt and ¯y [147]:

π(yt)T (yt → ¯y) = π(¯y)T (¯y → yt) (2.3)

Note that the above detailed balance condition is satisfied when the transition proba- bility distribution is close to the target distribution. Since we do not know the target dis- tribution, designing transition probability distributions that satisfies the detailed balance condition is difficult. The Metropolis-Hastings (MH) algorithm [181], instead of devis- ing special transition probabilities, introduces the acceptance probability to each Markov chain transition A(yt → ¯y) ∈ [0,1]. The detailed balance condition then becomes:

Chapter 2 Models and Inference in Computer Vision

π(yt)T (yt→ ¯y)A(yt → ¯y) = π(¯y)T (¯y → yt)A(¯y→ yt) (2.4)

It can be verified [147] that the following acceptance probability satisfies the above detailed balance condition:

A(yt→ ¯y) = min



1, π(¯y)T (¯y→ yt) π(yt)T (yt→ ¯y)



(2.5) With the use of the above acceptance rule, instead of designing task-specific transition probability distributions, any transition probability distribution T with non-zero proba- bility over the range of all target variables can be used for MH sampling. T is also called ‘proposal distribution’ since it is used to propose the next sample in the Markov chain, which is then either accepted or rejected based on the acceptance probability. Below, we summarize the Metropolis-Hastings (MH) MCMC algorithm.

Metropolis-Hastings (MH) MCMC: Sampling from π(_{·) consists of repeating the} following two steps [174]:

1. Propose a transition using a proposal distribution T and the current state yt

¯y_{∼ T (·|y}t)

2. Accept or reject the transition based on Metropolis Hastings (MH) acceptance rule:

yt+1=

(

¯y, rand(0, 1) < min1,π(¯y)T (¯y→yt) π(yt)T (yt→¯y)

 , yt, otherwise.

Different MCMC techniques mainly differ in the type of the proposal distribution T. Note that we do not need to compute the target (posterior) probabilities, but only the ratio of posterior probabilities π(¯y)

π(yt). This makes MCMC sampling suitable for the

inverse graphics setting where it is not feasible to get a closed-form solution for the normalization constant in the posterior distribution (denominator in Eq. (2.1)).

The key aspect of the MH sampling is the number of steps it requires until it con- verges to the target distribution. If the proposal distribution is very different from the target distribution, the samples tend to be frequently rejected resulting in a long wait for convergence. In practice, it is difficult to measure the convergence of any sampler since we do not know the target distribution. In Chapter 3, we discuss several diagnostic mea- sures that indicate the convergence of MCMC sampling. In the case of ‘inverse graphics’, each forward rendering step takes a considerable amount of time and we would like the sampler to accept as many samples as possible. A key for improving the MH sampling efficiency is to design the proposal distributions that match the target posterior distri- bution. In Chapter 3, we devise such technique by leveraging discriminative learning

2.2 Generative Vision Models a f c,e b d i m j l k n

Figure 2.3: An example factor graph. Variable nodes (circles) and fac- tor nodes (squares) representing the factorized function P(a, b, c, d, e, f ) = ψi(a)ψj(a, b)ψk(b, d)ψl(b, c, e, f )ψm( f )ψn(c, e, f ). Also shown are sample variable-to-

factor messages (green arrows) and, factor-to-variable messages (red arrows).

techniques for learning the proposal distribution. Refer to [174, 147] for more details on MCMC sampling. In Chapter 3, we study the behavior of MCMC sampling and its vari- ants for inverting graphics engines and propose techniques for improving the sampling efficiency.