Application of Bayes Theorem for model evaluation

The five types of uncertainty considered in the current analysis are shown in Figure 63, where (𝑒𝑖) is the expert conceptualisation of ecological response; (𝐿𝑖𝑗, 𝑈𝑖𝑗) is the degree of confidence in modelling ecological response as stated by experts; (∆𝑘) is the uncertainty in expert defined upper and lower ecological bounds relative to observations; (𝑞𝑗) is the set of hydrological model assumptions; and (𝑏𝑛) is the bias in observed data. The uncertainties 𝑒𝑖, ∆𝑘, 𝑞𝑗, and 𝑏𝑛 can be described as a set of discrete values which represent plausible assumptions, as shown in Equations 21 to 24. (𝐿𝑖𝑗, 𝑈𝑖𝑗) in Equation 25 represents the discrete set of model predictions as a set of lower and upper possible ecological condition scores.

Figure 63. Five sources of uncertainty considered: conceptualisation of

ecological response (ei); upper and lower ecological bounds (Uij, Lij);

uncertainty in these bounds (Δ); uncertainty in hydrology (qi);

uncertainty in observations (b).

,

1,...,



i

e



i

i

nE

(21)



, j 1,...,



j

q



j



nQ

(22)



, k 1,...,



k

k

nD

 



(23)



, n 1,...,



n

b



n



nB

(24)



,



m ij ij

x



L U

;

x

_m

x

_mt

, t1,...,nT

(25) where:

124 𝑞𝑗 = hydrological assumption set j, based on assumptions for Great Cumbung

Swamp inundation, rainfall and groundwater, with a total of 𝑛𝑄 sets of assumptions

∆𝑘 = the uncertainty k in the modelled upper and lower condition scores relative to the observed scores, with 𝑛𝐷 different possible values for ∆. This captures uncertainty in both the modelled and observed ecological condition.

𝑏𝑛 = bias of value n to adjust the observed data set, with 𝑛𝐵 possible values.

𝑥_𝑚 = uncertainty in estimated ecological response explicitly defined by experts, represented through lower (𝐿𝑖𝑗) and upper (𝑈𝑖𝑗) estimates of ecological response using expert model 𝑒𝑖 and hydrological assumptions 𝑞𝑗, with a total of

nT modelled scores.

The uncertainties 𝑒𝑖, 𝑥𝑚, and 𝑞𝑗 relate specifically to the expert based ERM, consisting of combined ecological and hydrological components. The first of these involves the comparison of different expert models, recognising the uncertainty in representing ecological response to water availability, and that different sources of knowledge can lead to different insights and understanding about ecological response. Secondly, experts identified uncertainty in their estimates of ecological response, hence provided ranges of possible condition scores rather than estimating a precise outcome. As described in earlier chapters, the ERM is unique in that it does not calculate an exact value for ecological condition, but instead assumes that there is sufficient uncertainty such that only an upper and lower bound of possible condition scores is feasible to model. The model is therefore indifferent to the actual ecological condition as long as it falls within the uncertainty bounds. The degree of stated uncertainty varies between experts, influencing the precision of the modelled condition scores. The third component considers the uncertainty in the hydrological model through exploring different hydrological assumptions. These three components have all been introduced in preceding chapters.

The fourth (∆) and fifth (𝑏) components consider uncertainty in both the model predictions and observations, and are introduced in this chapter for the specific purpose of evaluating model performance relative to observed data using likelihood functions. Whilst the experts define upper and lower bounds for estimating condition scores, it is assumed that there is also uncertainty associated with these bounds. In addition, there is uncertainty associated with the observed data, hence a discrepancy between observed and modelled condition scores may be a combination of uncertainty in both. ∆ is therefore used to avoid discounting models which are close to the observations but do not encompass them. A larger value of ∆ provides greater leniency toward differences in modelled and observed values, at the cost of reduced model precision.

125 Bias (𝑏) accounts for uncertainty in modelled and observed values through a systematic shift of all observations. Systematic differences may occur due to the observations not being representative of ecological condition throughout the entire case study area, or a consistent under or over estimation of condition in either the observations or expert models. In this case study, a bias is applied to account for a difference in resolution between modelled and observed condition scores, where observations are at a coarser resolution and hence there is uncertainty in what the equivalent value is at a finer resolution. This is explained in greater detail in Section 6.3.5.3.

It is recognised that the total uncertainty is not fully described by these components. However, this analysis enables the consideration of multiple sources of uncertainty derived from model components shown to have a significant impact on results in the sensitivity analysis, as well as knowledge of uncertainty in observations. It also acknowledges that the magnitude of uncertainty is unknown, hence different combinations of expert models, hydrological assumptions, ∆ values and bias are explored.

Taking a given set of upper and lower modelled ecological condition scores (𝑥𝑚), the posterior probability of a particular set of assumptions {𝑒𝑖, 𝑞𝑗, ∆𝑘, 𝑏𝑛} given a set of observed ecological condition using historical data can be described as:



ijkn o



P



x H

(26) where:



,

,

,



ijkn

e q

i j k

b

n







𝑥_𝑜 = observed data,

x

_o

x

_ot

, t1,...,nT

𝐻 = additional model assumptions not explicitly explored (hereafter assumed to be implicit in

P

_ijkn

x

_o



)

Given a systematic bias bn is used to adjust the observed data by a specified value to

account for differences in resolution between modelled and observed condition scores, Equation 26 is rewritten as:



ijkn o

 

ijkn n



P



x

P



z

(27) where: n o n

z

x

b

126 Bayes Theorem can be used to evaluate different sets of assumptions using Equation 28, to identify which set of assumptions has the highest probability of matching the observed data. Drawing upon conditional probability, Bayes Theorem states:







n ijkn

_{ }

 

ijkn ijkn n n

p z

P

P

z

p z









(28)

𝑃(𝜃𝑖𝑗𝑘𝑛|𝑧𝑛) is the posterior distribution or conditional probability of the model and uncertainty assumptions (𝜃𝑖𝑗𝑘𝑛), given the set of adjusted observations (𝑧𝑛); 𝑝(𝑧𝑛|𝜃𝑖𝑗𝑘𝑛) is the likelihood function which defines the probability distribution used to sample the bias-corrected observations given a particular set of model assumptions; 𝑃(𝜃𝑖𝑗𝑘𝑛) is the prior probability representing existing assumptions/knowledge regarding model performance; and 𝑝(𝑧_𝑛) is the marginal probability density of the bias-corrected observed data.

Given that 𝑝(𝑧𝑛) is independent of 𝑃(𝜃𝑖𝑗𝑘𝑛), Equation 28 can be rewritten as:



ijkn n





n ijkn



 

ijkn

P



z

p z



P



(29)

The likelihood function can be simplified if it is assumed each observation is statistically independent of other observations:



n ijkn



p z





p z

_n1

,...,z

_nT



_ijkn





n1 n nT1: 1

,

ijkn





n nT1: 1 ijkn



p z

z

_



p z

_







nT ijkn





nT 1 ijkn

...



n1 ijkn



p z



p z

_



p z





hence:









 

1 T n

ijkn n nt ijkn ijkn

t

P



z

p z



P







(30)

Equation 30 assumes the difference between modelled and observed values at time t is independent from that at time t-1. This assumption is considered reasonable on practical grounds. First, each set of (observed, modelled) points are a minimum of 29 days apart. Second, there is insufficient data to guide the development of a more complex dependence model. Third,

127 it is believed that the improvement introduced by a dependence model would be small compared with the uncertainties associated with observed and modelled ecological response.

It therefore remains to calculate the likelihood p z



_nt _ijknt



for individual instances of i, j, k

and b, and prior probabilitiesP

 

_ijknt .

In document Quantitative modelling for assessing system trade offs in environmental flow management (Page 140-144)