Model Order Identification

Having estimated the parameters of a variety of different models, an optimal model

structure must be selected. Model order identification, namely the process of

identifying the most appropriate values of n, m and 5 in (2.1), is chiefly undertaken,

although not exclusively, with the assistance of carefully selected objective statistical criteria. In combination, these objective methods should provide both a measure of how well the model output explains the data and indicate the presence of model over- parameterisation. In this study, model identification is based upon the coefficient of determination, R j , and Young’s Identification Criterion, YIC, (see e.g. Young, 1989).

The coefficient of determination provides a statistical measure of how close the model output fits the observed system output and is defined as,

where a 2 is the sample variance of the model residuals e(k) (2.3), and a 2 is the

sample variance of the measured output y ( k) about its mean value y . Clearly, a

good model fit is obtained where the variance of the model residuals a 2 is low in comparison to the variance of the data a 2 and the R 2 approaches unity. Conversely,

a poor model has a residual variance a 2 that is close to the magnitude of the sample variance a 2 and the R 2 tends towards zero. It is important to differentiate R 2 from

the more conventionally adopted coefficient of determination R2. The latter is based

upon the variance of the one-step ahead prediction errors, rather than the model response errors and, whilst this is a popular criterion for assessing the performance of forecasting models, it is less suited for TF model order identification. In particular, model one-step ahead prediction errors are relatively straightforward to minimise, as

the predictions are based upon past values of the system output itself; in contrast, the

model response errors are more difficult to minimise as the model output is

formulated based on the system input only. In a hydrological context, R 2 is also

commonly called the Nash and Sutcliffe efficiency criterion with unity power.

Although an excellent measure of model fit, the R 2 criterion should not generally be

utilised independently to assess the merits of a model, since it does not consider the models relative complexity or the level of uncertainty associated with the parameter estimates.

For this reason, an additional statistical measure, which incorporates both of these features, is utilised within the process of model identification. The heuristic Young’s Identification Criterion Y IC, is defined as

rr2 1 i=np fT 2 f)

YIC = log, — + log, {ATEWV}; NEVN = — Y (2.19)

Gy np

,=1

where the variables of the leading term are defined as in equation (2.18);

np = n + m + 1 is the number of parameters in the TF model (2.1) denominator (n) and

numerator (m) polynomials; at is the ith element in the parameter vector a ( N ); p u is

the ith diagonal element of the covariance matrix P (A ) (where N is the total number

of samples); such that a p u is an estimate of the error variance associated with the ith

element of the parameter vector a ( N) after N samples.

The first term of equation (2.19) is a normalised measure of how well the model fits

the data; as the model fit improves, the ratio of decreases and the term

becomes more negative. The second term provides an indication of the relative uncertainty of each ith parameter estimate, normalised with respect to all the parameters in the npth order model; as the parameter error variance decreases, so the

parameter is better defined and the second term becomes smaller. A model that fits the data well, but has a high order, with ill defined parameters, will have a

correspondingly high YIC value because of the large magnitude of the second term.

The YIC criterion, therefore, provides a compromise between model fit and over

parameterisation. An optimal model, with a low YIC, should give a good fit to the

In practice, the minimisation of the YIC will not necessarily identify the best overall

model and should, therefore, be used in conjunction with the R j criterion. This will

ensure that the model selected will fit the data well, without compromising parametric efficiency and uncertainty. Additionally, it is important not to carry out the process of model order identification without due regard to the physical nature of the system under consideration. For example, if the system is known to have parallel or feedback processes operating within it, model structures that can describe these behaviours

should be evaluated. Furthermore, the philosophy underpinning DBM modelling

should not be disregarded during the identification stage; a model structure that has a clear physical interpretation may be favoured for selection over an alternative structured model with slightly superior identification statistics. An additional, tertiary model order identification criterion utilised in the research reported in this thesis was the AIC criterion (see Box and Jenkins, 1970).

2 .4 D B M IDENTIFICATION AND ESTIMATION OF NONLINEAR

SYSTEMS

Many environmental systems are non-stationary and nonlinear in nature and as a consequence, alternative approaches to model identification and estimation need to be adopted in order to develop models that successfully characterise their behaviour, e.g. the EKF (Whitehead and Homberger, 1982; Chen and Beck, 1993). In this section a novel DBM approach to modelling nonlinear systems is presented which has been successfully applied in the areas of economics, ecology, biology, engineering and environmental science (Young, 1993; Young and Beven, 1994; Young and Pedregal,

1997; Young, 1998a). The DBM approach has the overall objective of identifying a nonlinear TF with time invariant parameters through a process of objective statistical inference applied to the time series data.

The preliminary stage of the DBM methodology is to determine that the system in consideration is in fact nonlinear. This can be ascertained in the first instance, by analysing the residuals of the best SRIV identified constant parameter linear TF model, using standard statistical tests (e.g. Billings and Voon, 1986). In the second instance, this may be achieved by allowing the parameters of a linear TF to vary over time. The powerful Fixed Interval Smoothing (FIS) method of recursive estimation (Young, 1993) can be used to obtain a non-parametric estimate of the model time varying parameters (TVP). Any parameter that is found to be significantly time variant over the observation interval, may reflect the non-stationary or nonlinear behaviour of the system. Analysis then proceeds to investigate whether the identified temporal parameter variations are state dependent and can be efficiently parameterised. In accordance with DBM philosophy, in addition to enhancing the model fit to the data, it is essential that the identified state has a clear mechanistic interpretation in relation to the system under consideration. Having identified the structure of the nonlinear dynamic model, the final stage of the DBM methodology is to re-estimate all the model parameters against the time series through nonlinear optimisation.