Custom precision in MCMC

As mentioned earlier in this chapter, the typical problem that MCMC aims to solve is the estimation of the integral (2.8). MCMC draws samplesθ(i), i_{∈ {1,...,N} from p(θ) (i.e. the target distribution)} and uses them to evaluate the output estimate (2.9), which is an unbiased estimator of (2.8).

Equations (2.8) and (2.9) are theoretical; they assume that all computations inside MCMC are per- formed in infinite precision. This means that MCMC samples are distributed according to the true distribution p(θ ) and the output estimate (2.9) converges to the true value (2.8) for N_{→ ∞. Nev-} ertheless, in practical MCMC implementations on digital devices, infinite precision is impossible to achieve. As a result, almost all work in MCMC literature uses either double or single precision float- ing point, since these are the two format that are inherently supported by CPUs. Both formats are considered adequate for most existing problems (although there is no guarantee that they are indeed adequate). In this thesis, double precision is used as equivalent to infinite precision in Chapters 3 and 5. In these chapters, precision optimizations take place and double precision was chosen as the

reference (infinite) precision because it is the highest precision supported by CPU and GPU architec- tures. On the other hand, Chapter 4 uses single precision for all implementations, since no precision optimization takes place in this case.

All MCMC algorithms consist of two main parts: 1) The evaluation of the probability density of each proposed sample p(θ∗). This part is specific to the targeted problem (i.e. the likelihood and prior). 2) The generic operations, which mainly include proposing samples and accepting/rejecting them according to their probability density value. This part is generic, i.e. it is the same regardless of the targeted problem, as long as the MCMC method does not change. Examples of what these two parts look like have been given earlier in this chapter when describing various MCMC methods and Bayesian models.

The probability evaluations (first part) take up the bulk of the computation time, especially when com- plex models and large-scale data are employed. In an FPGA implementation this means that most of the FPGA area will be devoted to probability evaluation modules. The generic MCMC operations are less computationally demanding. Therefore, when interested in achieving high sampling throughput for large-scale problems (which is the goal of this thesis), the crucial task is the minimization of the cost of implementing p(θ ). Part of this thesis (Chapters 3 and 5) focuses on how to achieve this goal by reducing floating point precision in FPGAs. As mentioned above, lower precision leads to area savings in the FPGA fabric, which in turn allows for more parallel modules to be instantiated.

Treating the two parts of MCMC as separate precision domains (all in floating point precision), it is possible to acquire four different precision combinations, shown in Table 2.1. Double precision is used in the place of infinite precision in this section (and in all parts of the thesis that deal with precision optimization). Combination D/D (double precision / double precision) is the mainstream approach followed in MCMC literature; it theoretically guarantees convergence to the “true” probability distri- bution but it is not efficient computationally, since it requires double precision arithmetic to be used in all parts of the implementation (meaning no area savings in the FPGA).

Combination D/C (double precision / custom precision) performs all generic operations in double precision, thus guaranteeing convergence to some probability distribution. Due to the use of custom precision for probability density evaluations, this stationary distribution is not the “true” distribution (p(θ )) but an approximation of it (denoted pc(θ ), where c = (mantissa bits, exponent bits) is the

Table 2.1: Combinations of precision configurations in the two precision domains of MCMC and the effects on: 1) convergence to the target distribution, 2) area savings/sampling throughput in an FPGA implementation. DP stands for double precision floating point, CP stands for custom precision floating point.

Probability density domain in DP

Probability density domain in CP

Generic domain in DP Combination D/D: “Correct”

kernel (all theoretical properties of MCMC maintained) - targets “true” target distribution p(θ ). No area savings, no increase in sampling throughput.

Combination D/C: “Correct”

kernel (all theoretical proper- ties of MCMC maintained) - targets approximate but known target distribution pc(θ ). Large

area savings, large increase in sampling throughput.

Generic domain in CP Combination C/D: “Perturbed” kernel (no guarantee of where/if it converges) applied to the cor- rect target distribution p(θ ). It actually samples from an un- known distribution. Small area savings - small increase in sam- pling throughput.

Combination C/C: “Perturbed”

kernel (no guarantee of where/if it converges) applied to the ap- proximate distribution pc(θ ). It

actually samples from an un- known distribution. Large area savings - large increase in sam- pling throughput.

to the use of low precision for probability density evaluations (which is the most computationally intensive part and typically takes most of the FPGA area). This approach is thus the most promising of the four approaches in the table.

Figure 2.10 shows the shape of pc(θ ) (in this case a standard Gaussian density) for various precisions

c. Only the number of mantissa bits changes. The effect of reducing the mantissa bits is that the approximation to the “perfect” density becomes coarser. The support of the distribution remains the same, since the number of exponent bits is not reduced.

Combinations C/D (custom precision / double precision) and C/C (custom precision / custom preci- sion) are less interesting. Neither of them guarantee convergence to any target distribution because custom precision is used for generic operations. It is possible that unexpected behaviour will occur when following one of these approaches (i.e. either the sampler does not converge or it converges to an unknown distribution). In addition, Combination C/D does not provide significant area savings (and thus speedup) in the FPGA due to the use of double precision for probability evaluations. Area savings from implementing the generic operations in reduced precision are limited.

−50 −4 −3 −2 −1 0 1 2 3 4 5 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5

θ

p

(

θ

)

Figure 2.10: Shape of pc(θ ) when changing the precision configuration c. DP stands for double

precision. Exponent bits are constant (=11).

approximated integral is no longer given by Equation (2.8). It is now the following:

Ic= Epc[ f (x)] = R

f(θ )pc(θ )dθ (2.24)

The value of the custom precision output estimate is:

˜ Ic= ˜Epc[ f (x)] = 1 N N ∑ i=1 f(θ(i)_{) ≈ I}c (2.25)

whereθ(i), i_{∈ {1,...,N} are samples drawn from p}c(θ ).

Therefore, combining (2.8) and (2.24), it is clear that the following bias is introduced in the output estimate due to the use of custom precision:

In Chapter 3, it will be shown that in the case of popMCMC, it is possible to use custom precision densities (like the ones in Figure 2.10) and at the same time correct or avoid the bias (2.26) in the output estimate. Chapter 5 will propose a way to optimize the precision configuration c given a user- defined threshold for the bias (2.26).