Choice of sampling method

In making a final choice of which sampling method to use, Monte Carlo and full factorial designs were quickly excluded. Both methods were considered too computationally expensive given the large number of parameters potentially of interest in this study. This left Latin Hypercube, fractional factorial and central composite designs as possible sampling methods. The discussion here therefore concentrates on these methods, and their

advantages and disadvantages are summarised in Table 3.4.

5

It is interesting to note here that no examples of the use of central composite designs were found in the geomorphological literature, nor were any found in a wider search, encompassing some 35 journals from geology, geophysics, hydrology and the oceanographic and atmospheric sciences. By contrast, a search of the scientific literature generally yielded over 900 references, mainly in journals from chemistry, biology,

medicine, electronics, engineering and related disciplines. Similarly, the author found no references to the use of factorial or fractional factorial designs in geomorphology, and very few examples from the geosciences more generally, compared with c. five thousand examples from the wider science literature, again many from engineering, biology, medicine, operational research and related disciplines.

Table 3.4: Summary of Latin hypercube, fractional factorial and central composite design sampling methods, with an assessment of their suitability for this study. k refers to the total number of parameters and p to the number included in any fraction (subsection 3.3.3 q.v.)

Sampling method Number of factor levels Number of simulations

Advantages Disadvantages Assessment for the purposes of this study

Latin hypercube

2 or more, randomised

Usually > 10k - Run size sometimes more

economical than when using fractional factorial or central composite designs - Randomised, so may show

unforeseen results and discontinuities - Larger samples may allow estimation of curvature

- Complicated to execute (needs checks for sample point correlation) - Must be used incrementally, to check thoroughness of sampling

- Additional sampling may be necessary, defeating initial run size economy

- Need for additional sampling also implies some redundancy in the initial sample

- Initial detection of discontinuities is unnecessary, as metamodels will be tested anyway

- Interim analysis, and possible additional sampling, demand both resources and time

2k-p fractional factorial design

2, fixed 2k-p - Economy of run size

- Ease of calculating the sample points - Resolution IV designs allow clear estimation of all linear main effects - High fractions useful in screening designs

- Cannot show curvature or discontinuities in main effects and interactions

- Some redundancy inevitable

- A priori reasoning of GOLEM’s response suggests that curvature in main effects should be expected

- No interim analysis, so less resource and time demands than Latin hypercube

3k-p fractional factorial design

3, fixed 3k-p - Ease of calculating the sample points - Suitable resolution allows estimation of curvature in both main effects and interactions

- Large run size compared with 2k-p designs

- Cannot show discontinuities - Some redundancy inevitable - Care needed to deal with complex aliasing

- A priori reasoning suggests curvature detection in main effects is useful - No interim analysis or sampling, so less time demanding than Latin hypercube - Run size likely to include more

redundancy than all of the other methods considered here Central Composite design 2, fixed, for cube points 3, fixed, for centre and axial points 1 central point, 2k-p factorial points, and 2k star points

- Economy of run size

- Ease of calculating the sample points - Allows estimation of curvature in the main effects

- Resolution V designs allow clear estimation of all linear, two-factor interactions

- Needs more runs that a 2k-p design - Cannot show discontinuities, or curvature in the interactions - Some redundancy inevitable

- A priori reasoning suggests curvature detection in main effects will be needed - No interim analysis or sampling, so less time demanding than Latin hypercube - Run size smaller than 3k-p, and probably about the same as or smaller than Latin hypercube

Since curvature in the main effects was considered possible, the use of a 2k-p fractional factorial was also quickly precluded. The possibility of curvature in the main effects was discerned partly through exploratory simulations with the chosen LEM (sections 3.4 and 3.5) during early stages of the research, and also through simple reasoning of the processes of interest in the landscape. The possibility of curvature in the interactions also arose, which again precluded use of a 2k-p design, but suggested that a 3k-p fractional factorial could be more appropriate. However, a 3k-p design was also excluded, mainly because of the much larger run sizes needed by this sort of design compared with Latin Hypercube sampling or central composite designs.

This left a choice between these two types of experiment, namely Latin Hypercube sampling and use of a central composite design. To clarify matters, the author drew on examples from the literature (e.g. Iman and Helton, 1988; Bowman et al., 1993; Chapman et

al., 1994; Sacks et al., 1989) and also sought further advice from Professor R. Cheng, of the School of Mathematics, University of Southampton. Iman and Helton (1988) in particular recommend that Latin Hypercube sampling should be used in many situations; by contrast, the findings of Bowman et al. (1993) and Chapman et al. (1994) were much less conclusive, both groups needing interim analysis and additional samples to explore further areas of the parameter space which appeared interesting or problematical. On balance, it was decided that no design is ideal, but that a central composite design offered the best compromise. In particular, some additional sampling was thought almost certainly to be needed whatever the initial sampling method, in order to test each metamodel’s accuracy. In this respect, the central composite design was seen to have the advantage that main effects could be

identified separately if needed using just the centre and star points, and this would allow more robust estimation of any important two-factor interaction terms. By contrast, the LH sample would make it more difficult to derive a metamodel without using both main effect and interaction terms together throughout the regression analysis. In addition, the central composite design would explore the factorial space to its limits, whereas LH sampling would only do so if sample points were deliberately placed at or near those limits.

Having made this choice, the key issues became which LEM to use in the simulations, the choice of the landscape setting and parameters of most interest, and the parameterisation of that model for the simulation experiment. The choice of LEM was made first and is now discussed, with a description of the model as used in this research.