6.6 Emulating output from intermediate variables
6.6.2 Methods and analysis
This analysis splits into two phases:
Studying the effects of the intermediate variables within each simulator Investigating the differences between the two simulators in terms of their use
of intermediate variables to produce output
An idea of the behaviour of each simulator can be gained by studying the correla- tions between the intermediate variables and the output. This can be done for each intermediate variable in turn to investigate the main effects, or using products of in- termediate variables to show the effects of interactions on the output variables. The
6.6. Emulating output from intermediate variables 162
first method is simple to display, for example by plotting the correlations between each principal variable and the output variable, grouped by intermediate variable. The second can be displayed by creating a matrix whose (i, j) − th entry is the cor- relation between the element-wise product of the ith and jth intermediate variables
and the output, and plotting this matrix using a heatmap. Examples of both sorts of plot are given later in this section and in Appendix C.2.
This stage gives useful insight into each simulator’s representation of the system, and so may even be used in a single simulator context to learn more about how the processes within a simulator are used to reach the output. Already these plots may reveal different features in the two simulators, particularly if an intermediate variable is very active in one but not in the other. However, the emulators can be used for simulator comparison in a much more informative way.
Because the emulators from intermediate to output variables now have the same input space for each simulator, an emulator of one simulator can be used over data from the other. If an emulator predicts another simulator’s behaviour fairly well, that indicates that the two simulators’ treatments of the intermediate processes are not so different. If performance is poor, this indicates a contrast.
When analysing the differences between these emulators, we must keep in mind the properties of each intermediate variable space, learned through analysing the intermediate variable data in Section 6.4. In particular, our lack of control over the intermediate variable space makes us unable to define the ranges of the intermediate variables in the training or prediction data, and the two simulators may produce values with very different ranges. Inferences made using the emulator of one sim- ulator to predict the behaviour of another will be highly unreliable if the emulator is operating outside the range of its training data unless the emulator’s variance, which will be high at these points, is taken into account.
For this reason, the standardised prediction error (SPE) is more a reliable quan- tity for analysis than those that do not take account of the emulator variance. Suppose there is a cluster of high error (emulator minus simulator, unstandardised) values in a particular region. If the SPE values are also high, this indicates that even when standardised by the emulator variance (which in this case is clearly relatively
6.6. Emulating output from intermediate variables 163
small) this trend exists. If there is no prevalence of high SPE values in this region, the emulator variance there is high, and therefore the difference highlighted by the error values should not be given much weight. This will especially be the case when a particular region of intermediate variable space is more sparsely populated by one simulator than by the other.
The behaviour of the SPEs from predictions using the emulator of one simulator over data from the other can be monitored across intermediate variable space. Ex- tending the validation method of emulating SPE values mentioned in Section 3.5, these SPE values can be emulated with the intermediate variables as inputs. If the emulator captures a large proportion of the variation in the SPEs, using the intermediate variables as inputs, this indicates that there is systematic behaviour in the SPE values. This in turn indicates that the emulator is not capturing some of the behaviour of the data, and therefore gives reason to believe that there are systematic differences between the two simulators.
For comparison, the SPE from predictions using an emulator of the same simu- lator should also be emulated, as well as some random vectors. The proportion of variation captured by the regression surface when each of these is emulated forms a good basis for comparison, particularly when the number of points is small relative to the dimension of the intermediate variable space, which could lead to over-fitting. If a relatively high proportion of variance in the errors for the emulator of another simulator is explained by the regression surface, this indicates that there are sys- tematic differences between the two simulators. These may not be particularly easy to discern or describe, but methods shown in the rest of this section should reveal the most important differences.
Plots similar to those used to understand the effects of intermediate variables on the output can show the effects of the intermediate variables on the SPE. Instead of plotting correlations between intermediate variables (or products of intermediate variables) and output, we plot their correlation with the SPE. Examples of these plots are given in Figures 6.19, 6.20, 6.21 and 6.22. Such plots are also used in Appendix C.2 to validate the emulators on data from the same simulator as their training data. This is imperative if conclusions are to be drawn from the emulators’
6.6. Emulating output from intermediate variables 164
performance over data from other simulators.
If the SPEs show a pronounced trend against a particular intermediate variable, this indicates that there is an effect in the prediction data which is not captured by the emulator of the other simulator. This intermediate variable may be inactive in the other simulator, or it may contribute differently. The plots showing correlation with simulator output will help discern which is the case, and will also enable one to differentiate between intermediates which have a similar and strong effect on both simulators, and those which are much less active.
Finally, understanding gained from this stage can be combined with that from the input to intermediate stage to show further which inputs are linked, and which are driving differences in the simulators.
Method summary
To summarise, when analysing the relationship between intermediate and output variables, the following steps are useful
12. Study the behaviour of each simulator in terms of its intermediate variables, using correlations between intermediate variables (or products of pairs of in- termediate variables) and output.
13. Use the emulator of one simulator to predict the behaviour of the other. Com- pare the RMSE to that using an emulator of the same simulator. If there is little difference, the simulators appear to use their intermediate variables in similar ways.
14. If the RMSE from the emulator of the other simulator is much larger, study the SPE values to unearth the roots of the difference. If a high proportion of variance can be explained by a regression surface, this indicates some system- atic trends. Study the correlations between intermediate variables and SPEs to try to reveal these trends.
These steps should show where the main differences lie between each simulator’s handling of the intermediate processes. Having studied the effect of each input
6.6. Emulating output from intermediate variables 165
space on the intermediate variables in Section 6.5, our findings can be combined to show which of the input to intermediate relationships are the most crucial to understand.
For intermediate variables that have the same effect on both simulators, differ- ences could still arise from the way they are created from the input variables. If an intermediate variable is very active therefore, working to understand the different ways the simulators calculate this intermediate variable could be very useful.
Input variables that are active only in intermediate variables that are largely inactive do not necessarily need to be as well understood. However, the intermediates for which they are active may be more important for different output variables.
This stage of intermediate variable emulation not only provides an insightful method for understanding a single simulator, but enables a direct comparison to be made using emulators, so that the differences between two simulators as functions can be seen and studied.