Further work

Further improvements to the algorithm presented in Section 3.4 will be focussed on improving its ability to handle high-dimensional and highly variable risks. One of the least well-developed aspects of the algorithm at the moment is the minimum sampling procedure from Section 3.4.8; this is because characterising the behaviour of the minimum of a stochastic function is a difficult problem in general. One pos- sible solution to this would be to change our formulation of the sequential problem slightly, so that the risks which we compute retain their dependence on the design parameters from all stages; if we were to do this, we would obtain a risk emulator ¯

r₁(i)d[n]



at the first stage which was a function of the design parameters from all stages, representing the risk from the sequential decision procedure in which, at each stage, we have the option of carrying out an experiment at pre-specified design

parameter dj+1, or making an immediate terminal decision a∗j.

If we were to build the emulator in such a way, then the need for an approximate minimum sampling algorithm such as the one outlined in Section 3.4.8 would be eliminated for all stages except the first, since we would be able to sample all risks from known design settings exactly; this would, hopefully, improve the level of sys- tematic risk variability that we could explain using our emulators. An issue with fitting the emulator in this way would be the increase in the dimensionality of the input space for the emulators at stages (n − 1), . . . , 1, and so if carrying out the analysis in this way, we must assess whether the increased effort required to fit emulators with higher-dimensional input spaces translates into a suitably improved representation of the risk function. In any case, we would still need an algorithm along the lines of the one in Section 3.4.8 in order to assess the likely location of the minimum risk at stage j = 1.

Chapter 5

Design for developing models

5.1

Model development: Reification

The development process for a model for a system often involves a number of sim- plifications and compromises:

ˆ in the atmospheric modelling example of Hirst et al. [2013], it is extremely clear that the Gaussian plume model does not capture the true behaviour of the gas under given atmospheric conditions. The real atmosphere is considerably more turbulent than the Gaussian plume model allows, and frequently, changes in the wind over long distances cause systematic discrepancies between the predictions of the plume and the observed concentration (see, for example, Figure 1.1);

ˆ climate models are abstractions of extremely large and complex natural sys- tems, where even if all of the processes were fully understood, inclusion of all of these in the model would be computationally infeasible. Common simplifica- tions in such models include the solution of governing equations on extremely coarse grids which necessarily neglects processes on length scales smaller than the chosen mesh (as in the example discussed in Goldstein and Rougier [2004]), or the introduction of a highly idealised representation of the system (as in the compartmental representation of the Atlantic ocean used by Zickfeld et al. [2004] in their model).

It is generally the case that the scientists that develop these models acquire knowl- edge during the development process about how they might be improved so as to better describe the behaviour of processes in the true system. For example:

ˆ more accurate models of gas transport processes are well understood (as de- scribed in chapter 1), but are not implemented in the analysis by Hirst et. al. because of the extra computer power that would be required and the lack of availability of high-resolution atmospheric information;

ˆ in both of the climate examples described above, it is simple to imagine (though potentially complex to implement) ways in which the model might be im- proved; in the first instance, we could simply solve the equations on a finer grid, or introduce alternative representations of the sub-grid-scale processes, and for the Atlantic model, we might introduce additional compartments, or refine the model output by introducing differential equations which represent other processes within the existing compartments. Generally within climate models, it is always possible to introduce representations of additional physical processes which would bring the model closer to reality.

Where the modellers have such knowledge about how the model might better repre- sent the system, inferences and predictions about the system made without taking account of these will not be consistent with their current beliefs. One option is to continue development of the model; however, in doing this, it is highly likely that during this additional development, new ideas about further future improvements will be generated, rendering the new model inadequate as well (while incurring ad- ditional development costs). It would be better, therefore, to model the system using a framework which is capable of incorporating such beliefs without the need to actually build the future models that we postulate.

A consistent approach to handling such expert judgement has been developed by Goldstein and Rougier [2009]: in the situation where an emulator is used as a statistical representation of a simulator, we can handle beliefs about likely future improvements to the simulator or to the underlying model by using a multi-level framework, introducing additional components which represent the effects of future

5.1. Model development: Reification 163 developments and specifying relationships between existing and new processes across levels.