Traditional spatial verification methods

Deterministic forecasts obtained from model 1 and model 2 are checked firstly by us- ing traditional spatial metrics, such as RMSE, MAE and point-wise correlation. The results for all four months and 0-3lt results are presented in Table 6.2. Comparing both models according to the listed metrics, we find that model 1 predicts SPI-1 more accurately than model 2. For model 1, the correlation is higher, by around 0.2, and the errors are almost half lower. The best forecast is achieved for June, with low RMSE and MAE. The month with the second highest accuracy is September, while for Au- gust, the forecast is slightly worse, and the worst forecast is obtained for July, with the largest values for RMSE and MAE and lowest correlation values (Table 6.2). The low- est values are observed in the southern parts of WA (Angola, Zaire; Fig. 6.8), near the Gulf of Guinea and in the eastern parts of the Sudan region. These patterns strongly resemble the areas of high and low correlation between SPI-1 and teleconnection in- dices, identified in Section 6.3. The northern part of WA shows high predictability, as also expected from the teleconnection analysis.

The derived verification parameters for model 2 show a slightly different picture of the distribution of forecast accuracy for boreal summer in WA. In general, model 2 predicts precipitation with the lowest errors in August, with mean values for RMSE of 1.12, MAE of 0.89, and a mean correlation value of 0.54. Thus, the errors are significantly higher, and correlation is much lower than for model 1. Next, in July, both errors are higher (1.20 and 0.95, respectively) and the mean correlation is just 0.46. In September, the results for the errors are similar, with an RMSE of 1.21 and MAE of 0.95; however, the mean correlation is a little higher (0.51). Finally, June has the highest error values (1.65 and 1.03, respectively) and the highest mean correlation, 0.54. Maps of the spatial distribution of correlation coefficients (Fig. 6.9) display an area of low values in the central (continental) part of WA. Having high correlation and high error means the bias of obtained results is also high and we do not catch the mean values good enough, however the bias is not checked explicitly. Again, as described for model 1, the July 2lt case shows very low values compared to other months and lead-times. We also observe more northern areas with low values compared to model 1. However, coastal areas exhibit good forecasts, with correlation coefficients of 0.7 to 0.8.

Next, the outputs of deterministic forecasts are analysed by conditional quantile plots (Wilks 2011) derived for each forecasting month for both models. Here, we use a modified version of classical Q-Q plots, plotting 10/90%, 25/75% quantiles, median and histogram (Fig. 6.10 and 6.11). Conditional quantile plots (CQP) are a convenient method for graphical representation of the joint distribution of forecast and observa- tions. Quantiles are derived from the conditional distributions p (oj | fi) in relation

6.5. Forecast verification 111

Figure 6.8: Point-wise correlation between the forecasts obtained by model 1 and ob- served SPI-1 for June, July, August, September for a) 0lt, b) 1lt, c) 2lt, d) 3lt forecasts.

6.5. Forecast verification 113 to the line of identity, and the lower parts of the figure show the unconditional distri- butions of the forecasts, p( fi) (Murphy et al. 1987). In the best forecast, the red line

characterising the median must be positioned on the main diagonal. Any deviation from this line shows uncertainties in the forecasts and SPI values that are poorly fore- casted. For better comparison, we fix the axis on the figures at the interval [-4, 4] (in the SPI-1 gradation, values of -4 and 4 characterise extreme drought and wet events). Comparing both forecast models, it is clearly seen that model 1 provides acceptable results (except for June 2lt, August 0lt; Fig. 6.10). The median line intersects with the diagonal line for normal events; however, there are forecasts for dry and wet events that significantly deviate from the diagonal line. There are even some cases where the median line becomes almost flat (June 0lt, July 2lt) and these forecasts are less reliable. In general, dry events are more closely following the diagonal line than wet events; however, with different lead-times, various situations can be observed.

We cannot conclude that with a larger lead-time the quality of the forecast is reduced. On the contrary, for August and September, the 3lt forecast presents the current situ- ation more accurately than the 2lt or 1lt forecasts. For model 2 (Fig. 6.11), it can be clearly seen that the forecasts are much further separated from the diagonal line than in model 1. The intensity of dry and wet events is also much more strongly overestimated by model 2 than by model 1. From the figures it is hard to select "good" and "bad" forecasts, since they all look more or less similar and present low accuracy.

Additionally, we present comparison maps with observed and deterministic forecasts for one randomly selected case with forecast values of all lead-times for both model 1 and model 2 (Fig. 6.12). The left panel presents the observed SPI pattern and de- terministic forecasts for both schemes with 0-3lt are presented on the right side. In June 1984 we clearly observed an intense and extreme drought in WA. Both models predict the actual drought and wet events accurately. However, the spatial distribution is not accurate, especially for model 2. Moreover, 0lt and 1lt forecasts from model 2 underestimate wet events and produce forecasts in the wrong areas. More accurate visualisations of deterministic forecasts are presented by model 1, where spatial locali- sation and intensity of dry and wet events are presented reasonably well. Less accuracy is achieved for 2lt, where both wet and dry events are underestimated. Conversely, for the 3lt forecast wet events are predicted in the southern part of the study region with high accuracy for both spatial distribution and intensity.

In summary, traditional forecast verification methods show that model 1 produces de- terministic forecasts more accurately, compared to the more classical model 2. That means, using combinations of predictors and obtaining ensembles of forecast equa- tions for each forecast region can increase the probability of getting better forecasts compared to the use of only one deterministic model.