4.4 Evaluation of methods
4.4.1 Return value distributions and comparison methods
From an engineering perspective, the main inferences from the current analysis are estimates of marginal and conditional return values corresponding to some long return period. In their paper, Jones et al. (2016) use Monte Carlo simulation to ob- tain empirical return level distributions for each case considered. When the return period is small, say 100 years or less, their simulation approach is computationally feasible. On the other hand, when longer return periods are required, Monte Carlo simulation becomes computationally demanding. In these circumstances, numeri- cal integration schemes yield dramatic reductions in the computational complexity of return value estimation (Ross et al., 2017). We implement the latter approach.
Return value integration scheme
The inference procedures introduced in Section 4.3 produce samples from the pos- terior distributions of the model parameters for each dataset. In order to obtain
estimated return values from these results, one needs to find a way to summarise the contribution from each iteration of this chain and across the different simulated datasets for each of the cases introduced in Section 4.4.2. As mentioned above, while Monte Carlo simulation provides a natural framework, it can also be compu- tationally intensive. An alternative approach consists of using numerical methods to obtain a summary distribution of the return levels for each of the cases and models considered, by extracting the information obtained across MCMC runs for each data sample and then across all simulated datasets for the same case (Ross et al., 2017).
Let us first focus on a single MCMC draw from the posterior of the parameters. First we need to subdivide the covariate domain into small subsets (bins), which provide a set of binned covariates common to all the samples of each case consid- ered. Assume we want to divide the covariate domain innb total bins, and denote a covariate bin Sj, for j = 1,2, ..., nb. It is essential to choose a large enough number of bins such that Sj is sufficiently small and we can assume each model parameter φ(xi), ψ(xi), ξ(xi) is constant within the bin, that is for all observed covariates xi ∈ Sj. Let us denote the set of Poisson-GPD parameters to be esti- mated by Θ(x) = {φ(x), ψ(x), ξ(x)}. For each specific bin Sj, let Θj indicate the estimates of these parameters in the bin. For each bin, we can then compute the cumulative distribution function (cdf) of any storm peak event with such a covariate value by using the estimates of the distribution parameters Θj in the
bin. We denote this byF(yp|Θj), which corresponds to the GPD cdf F(yp|Θj) = 1−1 +ξjyp−uj ψj −1/ξj if ξj 6= 0, 1−exp −yp−uj ψj if ξj = 0.
Furthermore, the parameter φj = φ(xj) can be interpreted as the mean number of storm peak events in the covariate bin Sj in a year. We can then obtain the cumulative distribution function FMT(yp|Θj) of the maximum MT observed in a period ofT years inSj as FMT(yp|Θj) = P(MT < yp) (4.4.1) = ∞ X k=0
P(k events in Sj in T years)Pk(size of an event in Sj < yp)
= ∞ X k=0 (T φj)k k! exp (−T φj)×F k(y p|Θj) = exp{−T φj[1−F(yp|Θj)]}.
Eq. 4.4.1 provides estimates of the cdf of theT −year maxima for each of the nb bins.
As mentioned before, it is often the case that “omni-covariate” return values, as well as results for some specific subset of the covariates, are required. In order to obtain these distributions, we need first to exploit the fact that, given some covariates, storm peak events are considered to be independent. By the law of probability, the joint cdf across all covariate values can then be obtained by taking the product of the individual cdf’s for each of the bins. Mathematically, this is
obtained by taking the product over all covariate bins, such that FMT(yp|Θ) = nb Y j=1 FMT(yp|Θj). (4.4.2)
Practically, this corresponds to computing an “omni-covariate” (e.g. omni-directional) cumulative distribution function FMT(yp|Θ) of the overall storm peak maximum
MT over all covariate bins.
We can then repeat the above operations given the posterior samples from each MCMC iteration, denoting the resulting cdf’s as FMT(yp|Θ
i) for i = 1, . . . , n
iter.
Then, we can obtain the posterior expected “omni-covariate” (e.g. omni-directional) cumulative distribution function as
b FMT(yp) = 1 niter niter X i=1 FMT(yp|Θi). (4.4.3)
If we have more then one realisation, such as in the case of a simulation study, we then need to proceed to a second stage of the approach. In this case, we consider the posterior expected “omni-covariate” (e.g. omni-directional) cdf indexed by simulated sample l as FbMl
T(yp). Then, we obtain a summary of the behaviour
across the nsim samples as
b FMsim T(yp) = 1 nsim nsim X l=1 b FMl T(yp). (4.4.4)
Algorithm 3 summarises the whole approach. Let us apply it to Case 2 as an example. For each of the 100 data sample realisations available, we can implement the first stage of this approach on the 20000 MCMC iterations. We then obtain
Algorithm 3 Posterior expected cdf of return values
Stage 1:
Consider the lth observation sample{yl,xl}, forl = 1, . . . , n
sim.
Input:
• The posterior estimates for the GPD and Poisson parameter, denoted as Θi j and evaluated at each bin j = 1, . . . , nb and MCMC iteration i= 1. . . , niter;
• A return period T of interest;
• Some future return level (e.g. Hs) of interest yp;
for each covariate bin j = 1, . . . , nb do
for each MCMC run i= 1. . . , niter do
Compute FMT(yp|Θ i j) as in Eq. 4.4.1 end for Compute FMT(yp|Θ i) as in Eq. 4.4.2 end for
Obtain the posterior expected “omni-covariate” cdf FbMl
T(yp) as in Eq. 4.4.3
Stage 2: . For hindcast datasets, nsim = 1 and Stage 2 is not necessary. If more the one sample realisationl is available:
1. Repeat Stage one for each sample realisation l= 1, . . . , nsim;
2. Compute the summary (average) mean expected posterior “omni-covariate” cdf FbMsim
the corresponding values for Eq. 4.4.3, which give the expected behaviour across the MCMC draws and are shown in grey in Figure 4.4.1. If we proceed to stage 2 of the algorithm, we can obtain the average posterior expected omnidirectional cdf
b FMsim
T(yp), which is shown for the results of the case study in Figure 4.4.4 in Section
4.4.2. This can then be compared to the true underlying return value distribution shown by the black line in both Figure 4.4.1 as well as the omni-directional plots in Figure 4.4.4.
Figure 4.4.1: Posterior expected omnidirectional cdfFbMl
T(yp) (in grey) for each of
the 100 sample realisations of Case 2 shown in Figure 4.4.2, corresponding to a return period of ten times the period of the original sample, with the true return value cdf in solid black.
The same two-stage approach can be used to obtain cumulative distribution func- tions for subsets of the covariates, e.g. for some given directional sector of interest, by taking the product only across a selection of sectors rather than for all nb sec- tors. The sectoral plots (right) in Figures 4.4.4 are the sector-specific equivalent to the omnidirectionalFbMsim
T(yp) curves (left).
Distribution divergence tests
case, for each sample realisation we can compare the empirical cumulative distri- bution function from the fitted model, obtained following the method described above, with the ‘true’ one from the known underlying case. The Kolmogorov- Smirnov (KS) and the Kullback-Leibler (KL) criterion have been used in the lit- erature to quantify the discrepancy between distributions. Given two distribution functions, the KS test uses the maximum vertical distance between their cumula- tive distributions, defined as
Dks(F0, F1) = sup
x
|F1(x) −F0(x)|,
to obtain a measure of their discrepancy. In particular, perfect agreement between F1(x) andF0(x) yields a minimum KS criterion value of zero, and the larger the
value, the more the two distributions differ.
The Kullback-Leibler divergence compares distributions using the average ratio of the logarithms of the density functions
Dkl(F0, F1) = Z ∞ −∞ log f0(x) f1(x) f0(x)dx, (4.4.5)
where there is no upper bound to Dkl(F0, F1) and, similarly to the KS criterion,
perfect agreement yields a minimum KL divergence of zero. In order to compute the KL criterion values, we approximate the integral in Eq. 4.4.5 following the approach of Perez-Cruz (2008).
In Section 4.4.3, we illustrate the performance of the different models considered for both cases introduced in Section 4.4.2. For all the 100 samples in each of these cases, we obtain a posterior predictive cdf, corresponding to the grey curves shown
in Figure 4.4.1. Then, we compare it with the ‘true’ underlying cumulative distri- bution to obtain the corresponding KS and KL divergence, and we use box-plots to summarise these criterion values across the 100 samples. The same approach is then repeated for the each of the directional sectors of interest.