Thesis Summary

The work described in this thesis has been concerned with the parallelisation of a class of algorithms known as Markov Chain Monte Carlo. The statistical basis for this type of algorithm prevents the simple application of traditional parallelisa- tion methods, and most existing multiprocessor implementations originate from the statistics discipline with the intent to improve the rate of chain convergence. This thesis address the problem from the high performance systems discipline, and seeks to spread the computational burden across multiple processors and/or machines.

To concisely summarise the contributions of this thesis:

• Two new methods have been presented that allow Markov Chain Monte Carlo algorithms to take advantage of multi-core and multi-processor machines. Termed ‘speculative moves’ and ‘speculative chains’, they are described and evaluated in chapters 3 and 4 respectively. Being purely implementation-based changes, the end results are unaffected whilst the runtime of typical MCMC programs can be reduced by∼40% by using just two processes.

• A new modification of Markov Chain Monte Carlo termed ‘periodic paralleli- sation’ has been proposed, that permits partial parallel processing on a large scale with a limited (and statistically acceptable) impact on the results. This method is covered in section 5.1.

• A number of methods have been considered that can reduce the runtime of MCMC applications concerned with image processing problems by considering portions of the image (temporarily or permanently) as independent images in their own right. Whilst lacking the statical certainty accompanying the other parallelisation methods presented, the potential runtime improvements are substantially higher whilst giving results that will be reasonable for many applications. These methods are explored in sections 5.2 to 5.4.

• The new methods proposed in this thesis have been fully implemented on a number of different machine architectures, and the suitability of these archi- tectures for these new approaches demonstrated and compared. The practical results and comparisons can be found in the chapters concerning each of the methods.

• A means of predicting the runtime of MCMC programs using our speculative moves (section 3.3), speculative chains (section 4.3.1) and periodic partition- ing methods (section 5.1.1) have been constructed. The remaining methods presented in this thesis have had practical examples conducted demonstrat- ing the typical runtime improvements that can be expected (see section 5.2 and sections 5.3.1 and 5.4.1). This provides: (i) increased certainty in real- world MCMC applications, (ii) a means of comparing alternative supporting architectures in terms of value for money and/or performance.

• A programming framework that automates much of the construction of MCMC programs has been developed. The parallelisation methods of speculative moves, speculative chains and periodic parallelisation can automatically made available with no extra work necessary from the the implementer. The usage of this framework is described and demonstrated in the appendices.

After the introductory chapter described the layout of this thesis, its primary contributions, and introduced the terminology used throughout, chapter 2 presented the background research relevant to the contributions of this thesis. This included an explanation of the Markov Chain Monte Carlo method and a discussion of how and where it may be applied. A summary of the existing methods of improving MCMC using parallel processing was presented with examples, along with an explanation of the conventional means of parallelising the MCMC algorithm and how these methods differ from the novel methods presented in this thesis. The chapter went on to establish a specific context for the work presented in the rest of the thesis

by describing in detail two MCMC applications for the segmentation of circular formations in a bitmap image, and in doing so further explaining the details of the most general purpose form of the MCMC algorithm (the Metropolis-Hastings transition kernel). Some simple non-parallel optimisations were also covered here for the benefit of readers implementing their own MCMC application. The example applications shown here also served as the testbed for the parallelisation methods presented in later chapters.

Having provided background and context to MCMC applications in chap- ter 2, chapter 3 presented the first contribution of this thesis, the parallelisation method ‘speculative moves’. Once the rational for this method and the revised MCMC implementation were explained, a formula for calculating the predicted run- time whilst using speculative moves was constructed. The speculative moves method was then tested on the practical example programs presented in chapter 2 using a number of different hardware platforms, and these results are compared with those predicted from the mathematical formula.

The logical development of speculative moves, termed ‘speculative chains’, was covered in chapter 4. Since a mathematical formula describing the benefits of this method quickly becomes unmanageably complex when attempting to describe anything but the simplest situations, a simulator was constructed and used to predict the runtimes that can be obtained using speculative chains. As with speculative moves, speculative chains was tested on the practical examples from chapter 2 using a number of hardware platforms.

Periodic parallelisation and a variety of image-splitting methods were pre- sented in chapter 5. Unlike speculative moves and chains, the methods presented in chapter 5 modify the basic MCMC algorithm in ways that are not be appropriate for all applications. However, with suitable applications, careful implementation and thorough testing these parallelisation methods can produce a substantially larger reduction in runtime that either speculative moves or chains would be capable of.

The software developed for this thesis consists of a framework with which to construct MCMC applications quickly and efficiently, without the implementer needing to write repetitive boilerplate code. Applications constructed with this framework (termed pMCMC) can implement the three major and new parallelisa- tion methods presented by this thesis with minimal work from the application im- plementers. An overview of the pMCMC framework and its benefits to any MCMC implementers occupies appendix A. To demonstrate the ease by which fully-featured MCMC applications may be developed using this new framework constructed for this thesis, appendix B contains an example implementation using pMCMC on one of the circle-finding methods from section 2.5. Finally the usability of the applica- tions built with pMCMC is shown in appendix C, where an example of how end-users interact with a pMCMC program at runtime is provided.