1. INTRODUCTION
2.4. Dynamic multi-objective model calibration
2.4.3. Impact of the objective functions on temporal variability in the Pareto-optimal parameters
referred to as calibration strategies). The calibration strategies considered are listed in Table 15, while the definitions of particular objective functions are given in the chapter 1.3.3. Note that the number of strategies is limited, because the multi-objective calibration requires mutually “conflicting” objective functions (chapter 1.3.5) and not highly correlated ones.
The model is calibrated over all overlapping 5-year long periods, with one water year of model warm-up prior to every simulation. Prior parameter ranges and the AMALGAM parameters are kept constant, regardless of the calibration strategy employed.
The parameter identifiability and model performance for given different calibration strategies are evaluated along with the consistency in parameter estimates.
Table 15. Calibration strategies considered in this research Calibration
strategy Number of
obj. funct. Objective functions used for the model calibration
1 2 Nash-Sutcliffe for flows and volume error
2 2 Nash-Sutcliffe for flows and log-transformed flows
3 2 Kling-Gupta efficiency and volume error
4 2 Coefficient of determination and volume error
5 2 Root mean square error based on high and low flows (Fenicia et al. 2007) 6 2 Heteroskedastic maximum likelihood estimator and root mean square error 7 3 Nash-Sutcliffe for flows and log-transformed flows, and volume error
2.4.4. Impact of the model structure on temporal variability of the Pareto-optimal parameters
To enable analysis of the model structural complexity impact on the consistency in parameter estimates, four versions of the 3DNet-Catch model are developed and presented in chapter 2.1.2.
The models are calibrated over 5-year long overlapping periods, with one water year of model warm-up preceding every calibration period. The model is calibrated using NSE and VE as objective functions and using the same AMALGAM parameters for all model structure versions. Calibration of the semi-lumped model versions is described in chapter 2.3.1, while the regularisation method applied for calibration of the fully-distributed model version is elaborated in chapter 2.3.2. Prior parameter ranges in this analysis vary
with the model structure, and they are specified in Appendix A. As for the distributed model version, the spatial parameter fields are represented by super-parameters, which are optimised. Consequently, the consistency analysis of this model version is based on the optimised super-parameters.
In addition to the parameter variability with the calibration period, parameter identifiability and model performance are analysed as well.
2.5. Assessment of temporal consistency in parameter estimates and in the model performance
Dynamic multi-objective model calibration results in an ensemble of the Pareto-optimal parameter sets for each calibration period. As a result of the adopted convergence criterion for the AMALGAM algorithm, number of the Pareto sets in the ensemble varies with the calibration period.
Distribution of the Pareto-optimal parameter values describes the parameter variability for each parameter and each calibration period (or uncertainty due to calibration period)17. Central tendency and dispersion measures of this distribution are analysed. The median is preferred over the arithmetic mean as the central tendency measure due to its resistance to presence of outliers (Kottegoda & Rosso, 2008), while the parameter dispersion in the calibration period is quantified in terms of the information content (IC) value. The latter is estimated following the approach presented by Wagener et al. (2003):
, 97.5 , 2.5
ˆ ˆ
1 norm norm
IC (2.5.1)
where ˆnorm, 97.5 and ˆnorm, 2.5 denote 2.5th and 97.5th percentiles of the distribution of the normalised Pareto-optimal parameters ˆ
norm, respectively. This statistic also enables quantifying the parameter identifiability: the narrower the optimised parameter range, the larger is the IC value and the parameter identification is better.
The parameter values may differ for several orders of magnitude (e.g. sub-surface soil layer thickness in millimetres and effective porosity in fraction between 0 and 1). The normalisation enables comparison among different parameters because the normalised parameters take values from 0 to 1 regardless of their prior ranges (Vrugt et al., 2006;
Luo et al., 2012). The optimised parameters θPARETO are therefore normalised with respect to the lower and upper bounds θMIN and θMAX of the prior parameter range:
PARETO MIN
During calibration, the parameter values are sampled from the uniform probability distribution with bounds θMIN and θMAX.
To illustrate the overall sensitivity of the Pareto-optimal parameters to calibration period and the changes in parameter identifiability, medians and IC statistic of the normalised parameters are presented in multi-temporal graphs (e.g. Hannaford et al. 2013).
Temporal parameter variability is quantified in terms of standard deviation St, i of the ensemble medians Me j (θi), where j denotes calibration period (j = 1, 2, ..., Ncal) from Ncal calibration periods, and i refers to the ith model parameter. On the other hand, standard deviation Su_prior, i of all initially sampled values of the ith parameter from the prior uniform distribution is (e.g. Kottegoda and Rosso 2008):
MAX, MIN,
This standard deviation describes initial variability of a parameter. If the optimised parameters significantly vary with the calibration period, standard deviation St of the temporal parameter variability is expected to exceed the initial variability and vice versa.
Therefore, parameter temporal consistency is estimated in terms of ratio of these two uncertainty (Vrugt et al., 2008). Smaller ratio indicates more consistent parameter estimate. Values greater than one suggest that the uncertainty due to calibration period
exceeds initial uncertainty, i.e. that the parameter is rather sensitive to calibration period.
The ratios estimated for the calibration periods of increasing lengths are used to inspect whether an increase in the calibration period length leads to more consistent parameter estimates.
Additionally, parameter variability with calibration period is quantified in terms of standard deviation of the median values of the normalised Pareto-optimal parameters, St, norm, obtained from all calibration periods of given length. The values of St, norm, calculated for periods of increasing length indicate whether the parameter sensitivity decreases with an increase of the calibration period length.
Along with the parameter estimates and IC statistic variability in model performance is analysed. Model performance is quantified in terms of medians of the objective functions and evaluation criteria (chapter 2.4) obtained from the Pareto-optimal ensemble. In addition, performance of the Pareto-optimal ensembles is quantified in terms of p-factor and r-factor. The former represents per centage of observations within the 95% prediction band (95PPU), while the later quantifies relative width of the 95PPU (Schuol and Abbaspour, 2006; Yang et al., 2008; Zhang et al., 2011):
At any point in time 95PPU is calculated as a difference between the predicted variables (simulated flows) corresponding to 2.5th and 97.5th, respectively. Target value of p-factor is one, whereas r-factor should approach zero (Zhang et al., 2011). Bastola et al. (2011) referred to the p-factor as the “count efficiency”, .
Correlation between hydro-meteorological indices (Table 14) and median values and IC statistic of the Pareto-optimal parameters is quantified in terms of the Pearson and Spearman correlation coefficients (chapter 2.4) which are calculated according to all
long periods using different combinations of objective functions (chapter 2.3.1).
Parameter variability is quantified in terms of St, i / Su_prior for each calibration strategy.
Along with this ratio, mean values of the IC statistic and model performance measures over all 5-year long calibration periods are calculated. Model performance is quantified in terms of mean number of the Pareto sets, median values of NSE, VE and NSElogQ and p- and r-factors.
Impact of the model structural complexity on consistency in parameter estimates, values of IC statistic and model performance is assessed analogously with the impact of the objective functions.