The terms “model calibration”, “parameter estimation”, “sensitivity analysis” and “uncertainty analysis/estimation” are all used to describe very similar concepts in hydrology modelling. Model calibration can be defined as “the process of adjusting parameter values of a model to obtain a better fit between observed and predicted variables. [It] may be done manually or using an automatic calibration algorithm” (Beven, 2009). Both sensitivity analysis and uncertainty estimation contain this process, but they go further to understand the variation of outputs that different parameter values achieve. The simple, traditional approach to model calibration, whereby trial and error is used to adjust parameter values until the model output best meets observed data has some limitations, for example: calibration assumes that there is an optimum set of model parameter values; calibrated model parameter values may only be applicable to that particular model; the choice of method of comparison to the observed data will affect which parameter values are determined to perform best, and may be biased towards the calibrator’s specified use of the model (e.g. flood estimation); and adjustments to some parameters may impact the model output more than others, (Beven, 2012).
Sensitivity analysis and uncertainty estimation are both methods of assessing models’ responses to parameter values and structural changes, however they vary in their ultimate purpose. Sensitivity analysis can be defined as: “the study of how uncertainty in the output of a model (numerical or otherwise) can be apportioned to different sources of uncertainty in the model input” (Saltelli et al., 2008).
Uncertainty analysis, on the other hand, focuses upon quantifying the uncertainty in model output. Tao (2008) states that sensitivity and uncertainty analyses are not explicitly related to model calibration, as some models may not require a formal calibration to estimate
parameters. In this case, uncertainty estimates may come from prior knowledge or past experience of the system; however, when calibration is required, this can be used as a posterior for uncertainty estimation. Saltelli et al. (2000) give six aims of sensitivity analysis, to determine:
1. if a model resembles the system or process under study;
2. the factors that mostly contribute to the output variability and that require additional research to strengthen the knowledge base; 3. the model parameters (or parts of the model itself) that are
insignificant, and that can be eliminated from the final model; 4. if there is some region of the space of input factors for which the
model variation is maximum;
5. the optimal region within the space of the factors for use in a subsequent calibration study;
6. if and which (group of) factors interact with each other.
Sensitivity analysis can be either local or global. Local sensitivity
analysis (LSA) explores a local area of the parameter space, centred on nominal values; whereas global sensitivity analysis (GSA) extensively explores wide ranges of parameter space (Tao, 2008). GSA therefore comes with a much greater computational cost than LSA. However, derivative-based LSA requires more of the analyst’s time to set up and carry out, which is difficult if the model parameters are uncertain or of unknown linearity (Saltelli et al., 2008, Wainwright et al., 2014). Common methods of sensitivity analysis include: one at a time (OAT) (Daniel, 1973, Daniel, 1958), the Morris method (Morris, 1991), principal component analysis (PCA) (Vajda et al., 1985), Monte Carlo (MC) analysis, Sobol’ sensitivity indices (Sobol', 1993), and the Fourier Amplitude Sensitivity Test (FAST) (Cukier et al., 1973, Cukier et al., 1978). These methods are briefly described in turn here:
One-at-a-time is a screening method that evaluates the effect of
of the perturbed parameter model is compared to a ‘standard’ value, usually in the middle of a set of parameter perturbation values.
The Morris method is a global method variation of OAT that moves
around the parameter space one parameter at a time, but does not return the previous parameter change back to its standard value. It is an economic method in that the number of experimental runs is
proportional to the number of input parameters (Saltelli et al., 2000).
Principal Component Analysis is a sophisticated method that uses
linear sensitivity coefficients to extract meaningful kinetic information for several species of reactions at several time points (Saltelli et al., 2000). PCA uses eigenvectors and eigenvalues to reveal parts of the model that strongly interact, and their associated model response.
Monte Carlo analysis uses randomly selected points in the
parameter space to run the model, and then uses the results to determine uncertainty in model prediction, and the contribution of
parameter inputs to this uncertainty. Monte Carlo is a sampling strategy that may be used in other forms of sensitivity or uncertainty analyses.
Sobol’ analysis produces sensitivity indices and identifies the
influence of each parameter, interaction of parameters and their
combination effects on the model outputs (Sobol', 1993). It is a popular method in hydrological model sensitivity analysis as it considers the interaction of model parameters (Qi et al., 2013).
FAST is an alternative method to compute the same indices as the
Sobol’ method, however calculations are often limited to the first-order, or main effect.
Generally, sensitivity analysis is distinct from uncertainty analysis, though many studies have used a combined approach (e.g. Ratto et al., 2001, Kiczko et al., 2007). Uncertainty analysis aims to define the entire set of possible outcomes, along with their associated probabilities of occurrence. Sensitivity analysis however, as outlined above, aims to define the change in model output values that result from small changes in input values, and thus measures change in a localised region of the
parameter space (Loucks et al., 2005). Loucks et al. (2005) give five achievable outcomes of an uncertainty analysis:
1. a description of the range of potential outputs of the system at some probability level (e.g. the mean and standard deviation of the
outputs).
2. an estimation of the probability that the output will exceed a specific threshold of performance measure target value.
3. the assignment of a reliability level to a function of the outputs, e.g. the range of function values that is likely to occur with some
probability.
4. a description of the likelihood of different potential outputs of the system.
5. an estimate of the relative impacts of input variable uncertainties.
Methods of uncertainty analysis are discussed in more detail in section 3.4. Figure 3.1 shows the impact of both input data sensitivity and input data uncertainty on model output sensitivity. This figure demonstrates that input parameter uncertainty and model sensitivity combined can lead to high levels of output uncertainty.
3.3 A One-at-a-Time Sensitivity Analysis of the Mac-PDM.09