Parameter space sampling issues, and choice of methodology and its conceptual basis

3.2.1 Parameter spaces, simulations and experiment run sizes

Any method designed to quantify the degree of model equifinality in essence involves undertaking multiple model simulations using different parameter value combinations, and thence establishing how many simulation results are equifinal to the output measures of interest. In the simplest terms, it is a matter of proportions, and can be expressed by:

( )

where Eq(y) is the proportion of equifinal solutions for the output measure y, Neq. is the

number of simulations yielding equifinal solutions, and Nt is the total number of simulations

run in the sample of the parameter space. It will be appreciated from equation 3.01 that the accuracy and the precision of the estimate of equifinality, Eq(y), cannot be absolute, but will depend upon the rigour and density of the parameter space sampling.

The parameter space typical of any LEM, as reviewed in Chapter 2, therefore poses immediate practical problems to any researcher wishing to explore that parameter space. Firstly, each individual LEM simulation takes some time to complete, even on fast, modern PC’s and computing clusters (typically from hours to days, or even weeks, for each

simulation; Greg Tucker, 2002, and Joe Wheaton, 2006, personal communications). Secondly, to sample the parameter space rigorously, the number of parameters, and the number of values of interest for each, together generate a large number of possible combinations required in the simulations. Table 3.1 shows examples of the number of simulations needed to sample a parameter space of k dimensions at N value points per parameter. The total run size required by each combination is Nk simulations, and although the smaller run sizes may be completed in a reasonable time (compared with the usual time span of a PhD or similar project), if many computers are allocated to the task, it is clear that the larger run sizes require an infeasible number of LEM simulations given usual computing resources.

Table 3.1: Total simulation run sizes required to sample all combinations of k parameters sampled at N values for each. The shaded area denotes the run sizes required of a typical LEM on this basis, according to the parameter requirements listed in Table 2.2 (q.v.).

Number of parameters, k:

5 10 15 20

Number of values used for each parameter, N:

Total run size

2 32 1,024 32,768 1,048,576

3 243 59,049 14,348,907 c. 3.49 x 109

5 3,125 9,765,625 c. 3.05 x 1010 c. 9.54 x 1013

8 32,768 c. 1.07 x 109 c. 3.52 x 1013 c. 1.15 x 1018

Faced with these numbers, a researcher must employ methodologies that minimise the number of simulations required, while still retaining parameter space sampling that is dense enough to produce representative results for the whole parameter space. After an initial appraisal of possibilities, the author considered in detail two possible methods, ‘GLUE’ (Beven and Binley, 1992), and the use of a mediating function or ‘metamodel’ (e.g. Kleijnen, 1998; Law and Kelton, 1991; ).

Regarding ‘GLUE’, this is the acronym for ‘Generalised Likelihood Uncertainty

Estimation’, which has been developed as a method for calibrating numerical models that include many parameters (e.g. Beven and Binley, 1992; Beven, 1996; Beven and Freer, 2001; Beven, 2006). The method takes into account both parameter value and observation data uncertainties, and after many simulations provides an estimation of the likelihood that any particular set of parameter values provides a ‘behavioural’ calibration for the output measures of interest (ibid.). It has been applied mainly to problems in hydrology, where it has been observed that many different parameter value combinations can generate very similar output, thus demonstrating a high degree of model equifinality (ibid.). The random sampling of the parameter space used in GLUE, and the identification of parameter values more likely to generate certain solutions than others, suggested that an adaptation of a GLUE method for this research might be appropriate. However, it still requires a very large number of simulations with the LEM, probably 105 or more for a modest LEM parameter space of between 7 to 10 parameters (e.g. Beven and Freer, 2001). The problem of large run sizes would therefore not be solved by using GLUE, and accordingly, the method was not adopted for the research herein.

The use of a mediating function, also called a ‘metamodel’ (further explained in subsection 3.2.2 and section 3.3), which could act as a surrogate model for the full LEM simulations, is a well established method in engineering and operations research (e.g. Kleijnen, 1998;

Sanchez, 2006). Through the use of a suitable experiment design, the parameter space is first sampled sparsely and output measures from the simulations are obtained. By

regression analysis, these measures may be then expressed as mathematical functions of the parameter values (ibid.). The method therefore provides an equation that can be solved as a single calculation, using a spreadsheet or other software, for any point in the parameter space within the designated parameter value limits. The equation may therefore be used to sample the parameter space as densely as required, generating a prediction of the LEM’s output at each sample point. Provided the equation is sufficiently accurate (as indicated by

R2

and other regression statistics), the proportion of results equifinal for the measure of interest can then be calculated (equation 3.01, q.v.), thus providing a quantified estimate of the LEM’s equifinality for that measure.

Although this method still requires a large number of simulations to be run in order to provide the data from which the equations are derived, the total is still two or more orders of magnitude fewer than would be required if using a GLUE or similar procedure (e.g. Beven and Freer, 2001). A potential disadvantage is that the LEM output may be noisy or in some way erratic, so that it is not possible to derive equations which are accurate enough to be useful. Also, for different measures of output, and different output reference times, other equations have to be derived through more regression analyses, which entails further work. Despite these disadvantages, however, the success of the method in other fields (e.g.

Kleijnen et al., 2005) suggested its adoption was appropriate for conducting the research herein. A detailed explanation of the concept underlying the ‘metamodel’ methodology now follows.

3.2.2 The ‘metamodel’ concept

A simulation model may be likened to a complex function, which transforms the input data into a set of outputs (e.g. Law and Kelton, 1991; Kleijnen, 1998). However, in the case of a complex, discretised, numerical model, such as a LEM, the exact mathematical

representation of this function is not known (ibid.). Despite this, LEM outputs can be related to the inputs, including the parameter values, by which they have been generated. Specifically, at any particular reference time, the outputs can be summarised using a range of measures, here termed ‘metrics’, which may serve as descriptors of the simulated landscape. If the relationships between the parameter values and the (output) metrics are

Moreover, if these regression relationships are sufficiently accurate, then they may be used to predict the LEM’s output, as expressed by the metrics, in terms of any parameter value combination within the parameter space of interest. A regression equation of this type is called a metamodel, as it can be used as a substitute or proxy for the full simulation with the numerical model (e.g. Kleijnen, 1998). Such tools have been widely applied in some branches of engineering and science (e.g. Kleijnen, 1998; Chen et al., 2006; Sanchez, 2006) but there are very few examples of their use in the earth sciences1, and none in

geomorphology. Their key advantage is that they allow many possible model scenarios to be explored very quickly. In this research, therefore, solving the metamodel for any parameter value combination involves a single, rapid calculation, thus enabling the LEM’s parameter space to be sampled very densely (c. 106 times), and certainly much more densely than would be practical if relying on full simulations. In summary, the overall concept of the research methodology is presented in Figure 3.1. It should be noted here that for clarity, and to distinguish it from other regression equations or models referred to in the thesis, the term ‘metamodel’ is used from hereon to mean a regression equation of the type described above i.e. an equation derived from numerical model output, with the aim that it should be used as a proxy for simulations with the full model.

1

It is very difficult to be sure on this point, as the description of the types of model used in the literature makes metamodel approaches in the earth sciences very difficult to distinguish from the use of any other type of model, whether it be statistical, numerical, mathematical, a regression model fitted to field data, and so on. There does, however, appear to have been some interest in the use of metamodels in climatology and

oceanography (e.g. Chapman et al., 1994; Lynch et al., 2001; Beringer et al., 2002; Sexton et al., 2003), and quite possibly the technique will be increasingly used in other disciplines as it becomes more widely known.

Figure 3.1: The metamodelling methodology, summarised as steps within a two stage process, and also showing the order of their discussion within the thesis. Step 5 generates

6

1. Choose a sparse but efficient initial sampling scheme, to provide suitable and representative parameter value combinations from the

parameter space.

4. Derive metamodels to predict the metrics as functions of the parameter values used in the simulations, and test, running further simulations if necessary to obtain better metamodel fits. 3. Summarise model output from each simulation using a range of suitable metrics.

5. Sample the parameter space densely, using the metamodels to predict model output at each sample point.

6. For each metric, compare metamodel predictions with a reference result, and hence determine the proportion of equifinal solutions.

2. Sample the LEM’s parameter space, running simulations with parameter values chosen according to the sampling scheme.

STAGE 1: initial sampling and metamodel derivation

STAGE 2: dense parameter space sampling, metamodel predictions and calculation of equifinal proportions Discussed in Chapter 3 (s. 3.3-3.5) Discussed in Chapter 4 Discussed in Chapters 5 and 6

Figure 3.1 shows that metamodel methodology is essentially a two stage sampling

procedure. The first stage employs a sparse sampling of the parameter space and a limited number of simulations, from which the output can be used to derive the metamodels. Once these have been developed, they may be tested, by running additional test simulations, and improved if necessary, running further simulations as required. The metamodels may then be used to emulate the LEM, thus providing the basis for the second stage of the parameter space sampling. As explained above, predicted LEM output from very large sample sizes (c. 106 points) may then be obtained quickly from across the whole parameter space. By this methodology, therefore, the problem of quantifying model equifinality may be treated in a different way, namely by focusing on how many simulations are required to derive accurate metamodels of the LEM’s output. In particular, the initial sampling scheme must be efficient, so that it still adequately represents the parameter space even though there are only a limited number of initial sample points. In addition, the output metrics must be selected with care, and be appropriate for the landscape features and properties of interest. Finally, the derivation of the metamodels must be robust, and the method of their

application in the second stage of sampling statistically rigorous. The first two steps,

covering sampling and experiment design issues and setting up the LEM for the simulations, are considered in detail in this chapter, beginning with some comments on basic concepts and terminology. These are dealt with in some detail, as particular aspects turn out to be of great importance in the research more generally.