Floating Point Error Analysis

The set of real numbers is continuous and infinite while computer memory is finite. A real number is therefore represented in computer memory by a discrete approximation with a cer- tain amount of error. The IEEE has created a standard for representing real numbers on com- puters [7] which defines several floating-point formats which store real numbers as a fixed amount of digits with a known error [48]. Real numbers are stored in floating point format as a signed integer significand and exponent in the form ±d.ddd × βe. The number of digits in the significand, p, is known as the precision and this, along with the exponent, e, and base, β characterise each floating point format. The IEEE standard defines three binary floating point formats (with β = 2) that use 32, 64 and 128 bits of memory divided among the significand and exponent. Each has a fixed value for p and a fixed range for e. These correspond to the 32 bit single-precision float, 64 bit double-precision double and 128 bit extended precision long doubletypes in C. Any floating point implementation only has to implement one of these types to conform to the IEEE specification.

If the significand of a floating point number starts with 1.xxx then it is said to be nor- malised. The smallest difference between one normalised floating point number and the next is known as the Unit in Last Place or ULP. This refers to the value of the least significant bit of the significand when the exponent is zero and is used to measure the absolute error obtained when converting a real number in infinite precision into a floating point format with fixed precision. The maximum error when converting a real number to any IEEE floating point representation is 0.5 ULPs. An alternative measurement is the maximum relative error which is equal to r−f_r where r is the real number and f is the number in floating point format. The upper bound of this formula is known as the machine epsilon, ε, and is available in C using the FLT EPSILON, DBL EPSILONand LDBL EPSILON macros.

Floating point numbers must be denormalised when performing arithmetic in order to have their exponents match. The IEEE specification also defines five basic operations on floating point numbers (+, −, ×, ÷ and√x) that must be implemented to less than 0.5ε precision and must use a guard digit when denormalising arguments. Other operations such as sin, cos, tan, log, exp and xy_{may have greater error but most mathematical libraries compute these to within}

between 0.5ε and 1ε of the real answer [62, 11].

ditive. This has lead to the design of special algorithms for summing that attempt to contain the error introduced [55]. When designing linear algebra operations on vectors and matrices there are additional issues such as optimising the memory access pattern to improve performance. This can lead to floating point operations being reordered and different implementations pro- ducing different results. In order to judge the correctness of our implementations we compare the output to that of an algorithm coded using the minimum number of floating point operations, and therefore the minimum amount of additive error. We also take into account the number of operations needed to compute the result.

3.6

Summary

We have introduced the necessary general methodology and technical information for develop- ing novel blocked linear algebra functions, including discussions of the theoretical maximum bandwidths available on different GPU architectures. We proceed in the next chapter with first of our contributions by presenting novel approaches for accelerating Cholesky decompositions on hybrid architectures. After carefully detailing the underlying algorithms, we present simula- tion comparisons with the state of the art numerical linear algebra library, MAGMA.

Chapter 4

Hybrid Cholesky Decomposition

4.1

Introduction

Classical matrix decompositions from the linear algebra literature, such as the LU, QR and Cholesky decompositions [49], may all be used to solve systems of linear equations of the form Ax = b. All of these methods require A to be square but in the case where A is also symmetric and positive definite, then the Cholesky decomposition requires only around half the number of operations to compute. The Cholesky decomposition splits a matrix into the product of an upper (or lower) triangular matrix and its transpose, such that A = UTU or A = LLT, and it can only be applied to symmetric, square, positive-definite matrices. Such matrices occur frequently in a wide variety of scenarios, for example in statistics where it is used to generate multivariate Gaussian random vectors with a known covariance [23]. Many machine learning algorithms also rely on the Cholesky decomposition, for example Gaussian processes [129, 103, 77], which are used for linear regression and prediction [130]. In computational statistics, the use of the Cholesky is ubiquitous in MCMC, for example Adaptive MCMC algorithms tune the proposal distribution of the sampler by adapting the covariance matrix at each iteration [107], which requires the Cholesky decomposition to be regularly reevaluated to propose a new sample. The Cholesky decomposition is therefore used heavily in statistical simulations and, particularly for high-dimensional problems, it is often the main bottleneck of the entire algorithm [59].

Each element of the Cholesky decomposition of a matrix A is defined recursively, as shown in Equation 4.2 for a lower triangular decomposition A = LLT and Equation 4.1 for an upper triangular decomposition A = UTU . The subtraction of elements from Ai,jis common to both

cases i==j and i!=j in both upper and lower algorithms. In the case of the lower triangular algorithm, the calculation of each element Li,jdepends on the prior calculation of all elements

Li,0→j, Lj,0→j and Lj,j. Similarly in the upper triangular algorithm, the calculation of each

limits the parallelism inherent in the basic algorithm. Li,j =      q

Ai,i−Pi−1k=1L2i,k if i = j 1

Li,i



Ai,j−Pi−1_k=1Li,kLj,k

 if i < j