5.5 Example C:Chl ratio in HadOCC
5.5.3 Working with reduced s 1 (·) data
An advantage of hierarchical emulation that was mentioned earlier is the ability to use training data containing many runs of s0(x) and comparatively few of s1(x, v, w).
This is particularly helpful when s1(x, v, w) is much more costly to run than s0(x).
So far, our example has concentrated on emulators built from a training data design containing equal numbers of points from the two simulators. In this section, we build an emulator using 1,000 input points for the simpler version of HadOCC, and only 100 for the more complex version. These hundred runs from s1 are all
matched in their x and m1 values by a point in the s0 data, and therefore the design
satisfies the criteria in Section 5.2.2. They were taken from a 2,000 point design, so that a hierarchical emulator could also be built with 1,000 points each in the s0 and
s1 input spaces.
The emulator was constructed using the same choices as the previous ones in this example, namely a first order regression surface involving all terms and a single estimated correlation length for each level of the hierarchy. The transformation function g (R) = R0.5278 was used. Three other emulators were also built, to compare
with this reduced s1 emulator. A standard emulator was built using the 100 s1 and
1,000 s0 points (i.e. the same reduced s1 data as the hierarchical emulator), and
another using just the 1,000 s1 points3. A second hierarchical emulator was built
from the full 2,000 point design.
It was suspected that when a comparatively small number of s1 data were used,
the emulator would perform considerably better ‘closer to s0’, i.e. for smaller values
of R. This feature has not manifested itself particularly in the emulators with equal numbers of points for both simulators. The errors (emulator prediction minus simu-
3From the previous examples of standard emulators, this appears to be a better strategy than
5.5. Example - C:Chl ratio in HadOCC 100
lator output) for each emulator are plotted against R in Figures 5.6 (for emulators of log (s1)) and 5.7 (for emulators of log (s1) − log (s0)). The standard emulators sum-
marised in Figures 5.6a and 5.7a use the same data as the hierarchical emulators in Figures 5.6c and 5.7c. 0 5 10 15 −1.5 −1.0 −0.5 0.0 0.5 1.0 1.5 g(R) Error 0 20 40 60 80 100 140 180
(a) Standard emulator from 1,100 point design. 0 5 10 15 −1.5 −1.0 −0.5 0.0 0.5 1.0 1.5 g(R) Error 0 20 40 60 80 100 140 180
(b) Standard emulator from full 1,000
point design over s1.
0 5 10 15 −1.5 −1.0 −0.5 0.0 0.5 1.0 1.5 g(R) Error 0 20 40 60 80 100 140 180
(c) Hierarchical emulator from 1,100 point design. 0 5 10 15 −1.5 −1.0 −0.5 0.0 0.5 1.0 1.5 g(R) Error 0 20 40 60 80 100 140 180
(d) Hierarchical emulator from full 2,000 point design.
Figure 5.6: Errors for four emulators of log (s1), plotted against g(R) (bottom axis) and
R (top axis).
5.5. Example - C:Chl ratio in HadOCC 101
is used, particularly when R is small. Improvement is greatest in the hierarchical emulators of the difference, where the hierarchical structure enforces the relationship between the simulators, so that the difference when R = 0 is always zero. The error gradually increases with R, (see Figure 5.7c) until its accuracy appears roughly the same as the standard alternative.
0 5 10 15 −1.5 −1.0 −0.5 0.0 0.5 1.0 1.5 g(R) Error 0 20 40 60 80 100 140 180
(a) Standard emulator from 1,100 point design. 0 5 10 15 −1.5 −1.0 −0.5 0.0 0.5 1.0 1.5 g(R) Error 0 20 40 60 80 100 140 180
(b) Standard emulator from full 2,000 point design. 0 5 10 15 −1.5 −1.0 −0.5 0.0 0.5 1.0 1.5 g(R) Error 0 20 40 60 80 100 140 180
(c) Hierarchical emulator from 1,100 point design. 0 5 10 15 −1.5 −1.0 −0.5 0.0 0.5 1.0 1.5 g(R) Error 0 20 40 60 80 100 140 180
(d) Hierarchical emulator from full 2,000 point design.
Figure 5.7: Errors for four emulators of log (s1) − log (s0), plotted against g(R) (bottom
5.5. Example - C:Chl ratio in HadOCC 102
Figure 5.8 shows how the RMSE changes as the range of R in the data changes. Because the number of points used to find the RMSE changes along the x axis, we should be careful when comparing these values. In particular, the values with small maximum R contain far fewer points, and are therefore much more changeable.
Figure 5.8a compares four emulators being used to predict log (s1). The hierar-
chical emulator built with the reduced s1 data (the solid line) shows a clear increase
in RMSE as R increases, performing very well when R is small. For R less than roughly 100, this emulator outperforms the standard emulator built with all 1,000 s1
input points. The hierarchical emulator built with all 2,000 data points also shows increasing RMSE as R increases, but not as noticeably.
Figure 5.8b shows emulators of the difference, log (s1) − log (s0). The standard
emulators are emulators of the difference calculated from the training data, which showed to be the best standard method in Section 5.5. The reduced s1 standard
emulator is therefore built from the difference of log-output at each of the 100 points included with R 6= 0. The hierarchical emulator of the difference in this case, where there is only one hierarchical variable, is really built from the points for which a difference (or ψ data) is available, and therefore the hierarchical emulator built from reduced s1 data is also built only from these 100 points.
The advantage in the hierarchical emulation structure is much more marked with reduced s1 data. The only extra information contained in the hierarchical
emulator, compared to the standard emulator with reduced s1 data (dot-dashed), is
the inclusion of the relationship
s1(x, R, m2) = s0(x) + g (R) ψ (x, R, m2) ,
and it appears from Figure 5.8 that this is a valuable addition to the emulator. This supports the idea that when the more complex simulator is costly to run compared to the simpler one, hierarchical emulation is a very effective strategy.
5.5. Example - C:Chl ratio in HadOCC 103 0 50 100 150 0.0 0.1 0.2 0.3 0.4 Maximum R
RMSE (log of output)
Standard, full Standard, reduced Hierarchical, full Hierarchical, reduced
(a) Emulators of log(s1)
0 50 100 150 0.0 0.1 0.2 0.3 0.4 Maximum R RMSE (log of r atio) Standard, full Standard, reduced Hierarchical, full Hierarchical, reduced
(b) Emulators of log (s1) − log (s0) = log (s1(·) / s0(·)).
Figure 5.8: RMSE for four emulators, changing as the subset of data used changes. The prediction data was restricted to R less than ‘Maximum R’ for values from 1 to 180 (the maximum R takes in our experiment). The number of input points considered ranges from 5 (when R ≤ 1) to 1,000 (when R ≤ 180).
5.6. Summary 104
5.6
Summary
In this chapter we have introduced hierarchical emulation, a method for emulation for two simulators where one is an extension of the other. These must have the property that when a small subset of the inputs, the hierarchical inputs, are set to particular values, the two simulators behave identically. Hierarchical emulation makes use of this relationship to emulate the more complicated simulator using a combination of other emulators, one of which is an emulator of the simpler simulator. We have established a prior structure for the emulator that ensures separabil- ity between terms. The implications of this method on the design of experiments have been explored, and some criteria established for the structure of the training data. A hierarchical emulator requires some transformation functions g (·) for the hierarchical variables, and desirable properties for these have been explored.
In order to assess the performance of hierarchical emulation, a validation study was conducted using two versions of HadOCC. For this, a 1 million point staggered LHD was created (see Section 3.5.1) so that hierarchical and standard emulators could be compared with respect to various tasks. Overall, this showed hierarchical emulation to outperform the standard method, both in its predictive accuracy and its coherence with the emulation model. A further experiment to assess the perfor- mance of a hierarchical emulator built with a reduced amount of data from the more complicated simulator showed very promising results compared to the standard em- ulator, and reinforced our beliefs that including the relationship between the two simulators is beneficial. In Chapter 7, we present an object-oriented method for programming hierarchical emulation.
Although this makes some progress towards emulating multiple simulators, the set of situations in which it applies is still fairly restrictive. In Chapter 6, we introduce intermediate variable emulation, whose scope is much wider.
Chapter 6
Intermediate Variable Emulation
The methods introduced so far jointly emulate two simulators only when their input spaces are almost or entirely the same. However, this is not often the case, and so for a method to be useful for comparing simulators, it should not make this demand. In Section 4.2.3, we noted that even when they do not share the same input space, two simulators of a particular system will often represent many of the same processes. The descriptions of HadOCC and OG99NPZD in Chapter 2 and in the examples of simulator differences in Section 4.2 reveal this to be true in their case.
In this chapter we introduce intermediate variable emulation, which takes ad- vantage of this feature in order to emulate multiple simulators with different input spaces in such a way that they can be compared. The idea is related to a technique developed by Strong et al. (2012), where intermediate variables created in a similar way are used to help understand structural uncertainty in computer models.
6.1
Synopsis
The basic premise of intermediate variable emulation is that a simulator can be looked at with regard to three stages, each represented by sets of variables:
1. Input variables
2. Intermediate variables
3. Output variables.
6.1. Synopsis 106
The input variables are the numbers input by the user to run the simulator, such as those in Tables 2.1 and 2.3. The output variables are those that are returned by the simulator, which an emulator aims to predict. These will usually be some summary of the raw simulator output, for example the mean as in the example in Section 3.6, or a multivariate summary of a time series.
In standard emulation, as described in Chapter 3, an emulator takes in the input variables and produces a probability distribution for the values of the output variables. However, when dealing with several simulators of the same system, each with a different input space, this method makes comparison very difficult, as we discovered when emulating OG99NPZD and HadOCC in Section 3.6. Seldom can direct links between particular input variables be made, and the emulators cannot easily be used to make inferences about the different modelling choices made in each simulator.
Intermediate variables are a new construct, representing the states of some set of sub-processes in a simulator1. Simulator code often contains sub-modules which,
as they run, produce variables other than the simulator output. These are usually invested with some system meaning, for example the death, growth or concentration of a particular species, or a transfer or flux that takes place. Two simulators of the same system may well contain some corresponding sub-modules.
Suppose two computer simulators of an ecosystem each have a process represent- ing the death of a particular species. Although their input variables are different, making it impossible to identify points in each input space with one another, both are able to produce a time series of this species’ mortality. These have a mean- ing that can be treated as the same, and so features of these time series can be compared. Producing a very similar time series from each model may enable us to identify links between portions of the input spaces, or to see how different the parameterisations of the process are in practice. The input variables can be used to emulate intermediate variables to better enable this task.
1For a particular simulator there may be more than one possible set of intermediate variables;
6.1. Synopsis 107
Further than this, we can list all these intermediate variables for each simulator in such a way that, at least in theory, if we know the values for all the intermediate variables we no longer need to know the values of the input variables to calculate the simulator output. Having done this, in principle we can use the values of the intermediate variables to build emulators of the simulators’ outputs, since the rela- tionship is deterministic.
Simulators of a particular system will usually have outputs that are linked in meaning. Often they track the population of a particular species, or the concentra- tion of a chemical. In the case of HadOCC and OG99NPZD, MarMOT, which was introduced in Section 2.4, enables outputs to be produced that are assigned exactly the same system meaning.
In the intermediate variable emulation framework, the emulators built from the intermediate variables will have input (the intermediate variables) and output (the output variables) with the same physical meanings for each simulator. This means that the treatment of the sub-processes by the different simulators can be compared. As well as making inferences about the relationships between the input spaces and the intermediate variables, we can compare the relationships between the same in- termediate and output variables for the different simulators. By emulating the intermediate variables from each simulator’s input space, we can compare the effect of the input variables on a range of variables that covers each simulator’s behaviour, since the intermediate variables separate the input from the output.
Emulating the intermediate variables from the inputs of each simulator is mainly informative for the relationships between the two input spaces. Emulating the out- put from the intermediate variables helps compare the behaviour of each simulator in terms of how the processes they model contribute to the overall representation of the system. Figure 6.1 shows the relationship between the variables.
In the rest of this chapter, we will investigate how these emulators, and the intermediate variables themselves, can be used to explore simulator differences. Each step in the process will be explained and summarised, and then illustrated using OG99NPZD and HadOCC, the simulators introduced in Chapter 2.
6.1. Synopsis 108
Simulator 1 inputs Simulator 2 inputs
Intermediate variables
Output variables
Figure 6.1: A framework for intermediate variable emulation of two simulators. The quantities in the boxes represent variables, not exact values. Although the intermediate and output values for the different simulators will have different numerical values, in terms of system meaning they are the same.
Intermediate variable emulation for a single simulator
As well enabling the comparison of different simulators, intermediate variable em- ulation is also a useful tool for understanding a single simulator. The intermediate variables allow one to probe into how the simulator is modelling the system, and learn more about how the inputs contribute to various aspects of the model. In Section 6.5 the intermediate variables will be used to refine the input region in a way that cannot necessarily be done using output variables. This is done using ei- ther observations of the intermediate variables, if available, or by ruling out inputs leading to unrealistic behaviour in some intermediate variables. This could prove especially useful for refining the input space in inputs that are not very active in the output, but are used in some intermediate variables.
Although the focus in this chapter is on the use of intermediate variable emula- tion for comparing two simulators, many parts of the method and analysis could be implemented for one simulator. Implementing the theory in the examples through- out this chapter will lead to a greater understanding of each simulator on its own, as well as to the relationship between them.