Bayes Linear Methodology

Variational Methods and Monte Carlo methods depend on a fully specified prior distribu- tion for the latent variables. For large complex data, it is often difficult to fully specify a prior distribution, which agrees with experts’ opinions (Craig et al., 1998). It may only be possible to specify partial prior beliefs. Instead of working with a full prior probability dis- tribution, an alternative is to use Bayes Linear (BL) methodology. This methodology works with the expectations directly rather than probabilities. Beliefs about the latent random variables are expressed through expectations and covariances rather than full probability distributions (Goldstein and Wooff, 2007) and these values are updated given observations. No assumptions are made about the underlying probability distribution of the model. The method of using subjective expectation to quantify expectations and uncertainty is some- times known as prevision. A rigorous development of the methodology for working directly with expectation as the primitive quantity is given in De Finetti (1974).

The link between subjective probability and expectations is easily justified. The expecta- tions can be used to calculate if an event takes place using indicator functions (Whittle, 2012). If the indicator is one when the event takes place and is zero otherwise, the proba-

CHAPTER 2. INFERENCE IN GRAPHICAL MODELS 35

bility of the event is given by the expectation of the indicator function.

BL methodology allows beliefs to be specified at the level of detail required through a set of random quantities, C = {X1, X2, . . .} where for each Xi, Xj ∈ C, the prior beliefs are

specified though:

• The expectation: E(Xi).

• The variance Var(Xi) quantifying uncertainty in the judgement of E(Xi).

• The covariance Cov(Xi, Xj) expressing the judgement of the relationship between the

random variables, quantifying the extent to which Xi may linearly influence Xj.

For example, for two random variablesX andY, interest may lie in the set of random quan- tities C =

X, Y, X2, Y2, XY . For each of these random quantities a prior expectation, along with variances and covariance would need to be specified to use BL methodology. An alternative way to specify a full probability distribution would be to specify the expectation of every possible combination of the random variables.

Formally, the belief specification is formed by constructing a linear spacehCiwhich consists of all finite linear combinations of the elements ofC. Covariance defines an inner product and norm over the linear space hCi. Hence, the name Bayes Linear follows from this linearity underlying the inner product structure. Operations which can be done on linear subspaces can be done in BL methodology. Familiar properties, present when working with probabilities, are also present when working within BL methodology such as linearity, convexity, scaling and additivity.

Supposing a set of the random quantities are observed. The beliefs on the remaining expec- tations and variances are adjusted directly to incorporate the outcome. If the base of the analysis is C= {X,Y} and the n-dimensional set Y is observed, then judgement on X is adjusted givenY =y. The adjusted expectation and covariance are the best estimate for eachXi as a linear combination of the observed quantities. The best estimate is defined in

CHAPTER 2. INFERENCE IN GRAPHICAL MODELS 36

terms of minimising the mean squared error. LetY0 = 1, then the adjusted expectation is:

b E[Xk|Y] = n X i=0 hiYi, (2.3.25) which minimises: E   X_k− n X i=0 hiYi !2 , (2.3.26)

over all collections ofh= (h0, h1, . . . , hn) (Goldstein and Wooff, 2007).

This is a convex optimisation problem, and could be found using optimisation software. However, it is also possible to derive analytical equations for the best estimate. For any choiceh∗= (h1, . . . , hn), the value ofh0 is found by differentiating (2.3.26) with respect to

h0: ∂ ∂h0 Eh Xk−hT∗Y−h0 2i = ∂ ∂h0 E X_k2 −2hT_∗E[XkY]−2h0E[Xk] + 2hT∗E[Y]h0 +hT∗E YTYh∗+h20 , =−2E[Xk] + 2h∗TE[Y] + 2h0. (2.3.27)

Then setting equation (2.3.27) equal to zero:

h0 =E[Xk]−hT∗E[Y]. (2.3.28)

For this value ofh0, the value ofh∗ is found by differentiating (2.3.26) with respect to h∗:

∂ ∂h∗ Eh Xk−hT∗Y−E[Xk] +hT∗E[Y] 2i = ∂ ∂h∗

Var(Xk) +hT∗Var(Y)h∗−2hT∗Cov(Y, Xk)

,

= 2hT∗Var(Y)−2Cov(Xk,Y).

Assuming thatVar(Y) is of full rank, setting this equal to zero gives:

CHAPTER 2. INFERENCE IN GRAPHICAL MODELS 37

and substituting into equation (2.3.25), the BL adjusted expectation is:

b

E[Xk|Y] =E[Xk] + Cov(Xk,Y)Var(Y)−1(y−E[Y]).

The adjusted variance ofXk givenY is:

d Var(Xk|Y) =E h (Xk−Eb[X_k|Y])2 i .

Substituting inEb(X_k|Y) from (2.3.29), the adjusted variance is: d

Var(Xk|Y) = Var(Xk)−Cov(Xk,Y)Var(Y)−1Cov(Y, Xk). (2.3.29)

The adjusted expectations and variances can be viewed as a primitive which quantifies aspects of the beliefs given observations. Alternatively they can be seen as a simple tractable approximation to a fully Bayesian analysis for complicated problems. For linear Gaussian models, the adjusted values are equivalent to the posterior mean and covariance found using a fully Bayesian analysis.

A common application for BL methodology is for uncertainty analysis in high dimensional complex computer models for physical processes such as climate models (Williamson et al., 2013), the Thermohaline circulation in the Atlantic Ocean (Goldstein and Rougier, 2009) and galaxy formation (Vernon et al., 2010). Within these applications, the aim is to seek out areas of simulator inputs which closely match the historical observations (termed history matching in the oil industry). An emulator based on BL methodology can be used to make a decision where the simulator should be run in order to eliminate areas of uncertainity from the model. Many of the applications of BL aim to help with decision making within the application, such as deciding where to sink further wells in off-shore oil rigs (Craig et al., 1997) and using a decision theory based approach to experimental design (Farrow and Goldstein, 2006).

Chapter 3

Sequential Decision Problems

3.1

Introduction

In many real-world situations, decisions are made in order to help maximise some type of reward. The decisions are often difficult to make because of uncertainty in the system and about the outcome of the decision and any possible future outcomes. The decisions or the actions they generate do not only result in a reward but also result in new knowledge that can be used to help improve future decisions. This type of problem can be modelled as a sequential decision problem, which is a type of learning problem where the learning can be either offline or online (Powell and Ryzhov, 2012). In offline learning problems, no rewards are collected as the information is gathered. The decisions and feedback are sequential but a reward is only received at the end of the simulation. Ranking and selection is an example of an offline learning problem, where the concern is to find the best of two or more possible processes after a number of observations. See Kim and Nelson (2006) for an introduction to ranking and selection problems.

In online learning problems, the rewards are collected at the same time as information is gathered. Multi-armed bandit problems are one example of an online learning problem. They are one of the best known and most commonly used type of sequential decision problem

CHAPTER 3. SEQUENTIAL DECISION PROBLEMS 39

and arise in many different disciplines including statistics, operations research, machine learning, computing, control theory and economics. One of the original motivations for studying the multi-armed bandit problems coming from statistics was the sequential design of experiments (Robbins, 1952).

Sequential decision problems have been extensively studied because of the large number of applications. Different disciplines vary in their approach to these types of problems, driven by differing motivating applications. Machine learning has recently been very active in the area of research for multi-armed bandits. For example, in website optimisation (Scott, 2010), where the observations are easy to observe and the outcome is almost instantaneous, they combine the ideas of sequential learning motivation (Robbins, 1952) with ideas from reinforcement learning; the process of learning about uncertain environments and observing the outcomes inspired by behavioural psychology (Sutton and Barto, 1998). Many of the applications within statistics and operations research have less instantaneous rewards and a higher cost associated with them. For example, deciding where to drill to find oil, or deciding which drug to give a patient in a clinical trial.