Model performance and sensitivity analysis

Modelling often requires calibration in order to obtain acceptable performance. Calibration is the process of adjusting certain model parameters in order to improve the match between model outputs and observed measurements. Not all model parameters may require this adjustment as some parameters may not significantly alter model performance after being adjusted. As a result sensitivity analysis is often first carried out to establish which of the model parameters require calibration. Sensitivity analysis involves determining the impact of change in model parameter values to modelling performance. It differs from model calibration as it seeks to establish the parameters that impact on performance significantly while calibration is applied to establish parameter values that maximize performance.

In this study model results were compared to observed soil moisture to assess the performance of the model. The model performance was assessed using three model evaluation measures which are the Nash-Sutcliffe efficiency (NSE), the Percent bias (PBIAS) and coefficient of determination (R²) details of which can be found in Moriasi et al. (2007). The NSE is a normalized statistic that determines the relative magnitude of the residual variance of the model simulation compared to the measured data variance. It is computed using Equation 5-25. The PBIAS measures the average tendency of the

Equation 5-24

5-38

simulation results being larger or smaller than the observed data and is computed using Equation 5-26. The R² is expressed as the squared ratio between the covariance and the multiplied standard deviations of the observed and predicted values as shown in Equation 5-27 (Krause et al., 2005). simulated mean values for all the data points.

Model performance evaluation measures are used for both sensitivity analysis and calibration. This is because in sensitivity analysis model parameters are adjusted just like in calibration then assessed to establish which parameter adjustments cause a significant change in model performance. On the other hand calibration requires adjustment to establish the value that maximizes model performance. In general the approach used to select and/or adjust model parmeters depends on computation costs and the parameter space in which the parameter adjustment has an impact on model output (van Griensven et al., 2006; Ndiritu, 2009). Parameters have both local and global impacts on model outputs. Local impacts refer to changes in model output due to variations in certain parameter values e.g. mean or maximum. Global impacts refer to changes in model outputs due to variations in parameter values from the whole possible parameter range.

Equation 5-25

Equation 5-26

Equation 5-27

5-39

A hybrid method of selecting the parameter space is often used in which local sampling methods are integrated into global sampling methods to reduce costs but at same time sample parmeters representative of the possible parameter range. Both sensitivity analysis and calibration can be carried out automatically or manually.

Automatic approach to sensitivity analysis or calibration is based on sampling of parameter values from an entire range of possible parameter values and possible parameter sets (van Griensven et al., 2006). In sensitivity analysis the approach is normally used to solve the problem of over parameterisation by identifying parameters that do not have a significant influence on modelling performance. The automatic approach involves sampling of a set of initial model parameters which are then varied using an automatic calibration method such as the shuffled complex evolution method until they convert to global values that optimize the set objective function. The process is repeated several times and each time the final set of parameters is recorded. Parameters that have low sensitivity will have high variation in their values as the process is repeated while those that are highly sensitive will have little variation. An application of automatic calibration approach in sensitivity analysis can be found in Ndiritu (2009). The automatic calibration approach is fast and therefore most suited to models that have a practical application as modellers or researchers will spend less time assessing the parameters that the model is sensitive to for a given application. However automatic model calibration may select parameter sets that do not make hydrological sense. This can be overcome by setting the range within which parameter search is carried out within the limits that are hydrologically meaningful. Another disadvantage of the automatic calibration is the need for an automatic calibrator such as the shuffled complex evolution method to be either incorporated in the model software or in a software that is compatible with the model software. Global sensitivity analyses such as those used in automatic modelling are not yet common in vadose zone modeling because they are difficult to implement (Skaggs et al., 2014).

The manual approach to sensitivity analysis modify selected input parameters while holding all other parameters constant (Hoyos and Cavalcante, 2015; Kumar et al., 2014).

The values of modified input parameters are varied above and below those of the model determining the performance of the model in each case. If the change in model

5-40

performance corresponding to changes in a specific input parameter is large then the model is sensitive to the value of that input parameter and therefore that input parameter should be accurately estimated in order to reduce uncertainty in the modelling. On the other hand if the change in model performance is low when the value of the input parameter is changed then the model has low sensitivity to that input parameter. Manual model calibration is widely used for sensitivity analysis because of its simplicity.

Model sensitivity was carried out to establish input model parameters that require accurate estimation (calibration) in order to reduce uncertainty in the model output. The PBIAS was used as a model performance measure to assess the sensitivity of the model to variations in different parameters. For this study manual calibration approaches were therefore selected for their simplist. The model parameters were varied by increasing and decreasing them by 10%, 20% and 30% above and below that of the model parameters.

Each time the model was run and the model performance in terms of the PBIAS was computed and was used to assess the sensitivity of the model. The sensitivity of the different parameters to model predictions was compared using the condition number which expresses the rate of change of the dependent variable with respect to the rate of change in the independent variables (Hoyos and Cavalcante, 2015). The condition number was calculated using Equation 5-28.

𝐶𝑁_𝑝 = ^𝑝̅

𝐷

∆𝐷

∆𝑝

Where:

𝐶𝑁_𝑝is the condition number for parameter p; 𝑝̅ is the mean of the parameter p considered; ∆𝐷 is the change in the predictant 𝐷.

The model performance (PBIAS) each time the parameter value was changed was used to represent the predictant for calculation of the corresponding CN. To compare the sensitivity of the different parameters the difference between the highest and smallest CN was used as measure of the sensitivity with the parameter having a higher difference considered as being more sensitive.

Equation 5-28

6-1

6 FIELD ASSESSMENT OF WATER CONSERVATION