Fractions Skill Score

Spatial verification methods have recently become popular in the meteorological com- munity (Radanovics et al. 2018). To keep up with the state-of-the-art we compute

Figure 6.10: Conditional quantile plots for model 1: a) 0lt, b) 1lt, c) 2lt, d) 3lt fore- casts. The red solid line presents the median, the green dashed line the 25th and 75th percentiles, the blue dashed line the 10th and 90th percentile; the blue histogram shows the frequency of forecasted values.

6.5. Forecast verification 115

Figure 6.12: Example results of observed SPI and deterministic forecasts of spatial SPI-1 patterns for June 1984 using model 1 (top 4 panels) and model 2 (bottom 4 panels) for a) 0lt, b) 1lt, c) 2lt, d) 3lt forecasts.

6.5. Forecast verification 117

Table 6.2: Summary of forecast verification parameters. Model 1

0lt 1lt 2lt 3lt

Month RMSE MAE Corr RMSE MAE Corr RMSE MAE Corr RMSE MAE Corr June 0.7004 0.5348 0.7285 0.7111 0.5473 0.7109 0.7227 0.5510 0.6808 0.7129 0.5463 0.7033 July 0.7699 0.5786 0.6372 0.7682 0.5801 0.6655 0.8767 0.6653 0.3855 0.7668 0.5788 0.6596 August 0.7978 0.6129 0.6488 0.7366 0.5686 0.7163 0.7612 0.5925 0.6603 0.7489 0.5784 0.7007 September 0.7536 0.5914 0.6675 0.7408 0.5769 0.6930 0.7386 0.5762 0.6921 0.7274 0.5696 0.7193 Model 2 0lt 1lt 2lt 3lt June 1.0592 0.8463 0.5629 1.0592 0.8463 0.5629 3.4357 1.6214 0.4806 1.0503 0.8404 0.5567 July 1.0847 0.8620 0.5199 1.1843 0.9427 0.5011 1.4205 1.1226 0.3010 1.0966 0.8721 0.5191 August 1.2495 0.9879 0.5103 1.1139 0.8949 0.5481 0.9993 0.7887 0.5474 1.1253 0.8987 0.5343 September 1.0198 0.8189 0.5147 1.1056 0.8741 0.5500 1.2301 0.9807 0.5132 1.4883 1.1328 0.4848

Fractions Skill Score (FSS) in our study to check the obtained deterministic forecast for both models. Most of the spatial verification characteristics are available in R within the package SpatialVx6. For FSS calculations we use a combination of the SpatialVxpackage and R code provided by Dr. M. Hoff (DWD, Germany) for better visualisation.

The FSS calculations require to identify thresholds and scales (numbers of neighbour- hood boxes), where the model produces different scores. In deterministic forecasts, it is important to identify intensity thresholds of dry and wet events (from -1 to -2 for dry events and from +1 to +2 for wet events). As neighbourhood size, we take the highest value equal to 8 (in our case the WA domain is containing 13 by 18 grid points, if we take n = 8, it fully covers the study area). We obtained the FSS for dry and wet events for deterministic forecasts for both models, 0-3lt, and the years 1982-2016. FSS can be obtained for each year separately or for a whole period of interest, depending on the specific purposes.

The results of FSS calculations are shown in Figures 6.13, A4, A5 and A6 as a function of dry/wet intensity threshold (x-axis) and spatial scale (y axis). Within these plots, the numbers present the FSS score values; dark red, red and orange colors indicate good performance, whereas all colors below the blue dividing line represent poorer perfor- mance. The edge between those two areas (indicating the range of "useful forecasts") can be calculated using the formula FSS > 0.5 + f_obs/2, where f_obs is the fraction of observed events in the full domain (Mittermaier et al. 2010). Values below the blue line in the FSS diagrams display the skill that would be achieved using traditional grid box matching and a low threshold. All other values of n (from 2 to 8 in this case) on the y-axis present FSS scores for upscaled boxes. Following this logic, it is easy to achieve high skills for the forecast with n = 8, since all domains are taken into account and the higher n, the higher the FSS. Comparing the forecast for different box sizes helps to overcome the biggest problem of weather forecast verification — the double penalty

problem (see Section 3.5.1), and thus the fact that traditional techniques consider two errors — the first when the forecasted event did not occur at the corresponding loca- tion, and the second when the event happened where it was not forecasted (Gilleland et al. 2009). Therefore, matching observed and forecast values for large box sizes decreases the chance of double penalty and increases the accuracy of the forecast. Forecasted dry events presented in Figures 6.13 and A5 for both models show high FSS values with similar qualities, as found using traditional verification methods. The FSS values for model 1 are in most cases (except for July), especially for box sizes 1 and 2, above the blue threshold. Only in July, the forecasts for all four lead-times show much lower values, with FSS below the usefulness criterion (July, 2lt). For other box sizes, we found acceptable scores. August and September forecasts are more accurate than June and July, however, FSS values in June are also high, especially for 1lt and 3lt forecasts. The highest accuracy can be observed again for the September 3lt forecast. FSS values for model 2 are slightly higher, and values are equally distributed among all months and lead-times; however, none of the cases covers the first two box sizes. That behaviour allows us to conclude that model 1 shows higher accuracy compared to model 2 for dry events.

FSS values for wet events are presented in Figs. A4 and A6. The FSS values for wet events have similar behaviour and distribution as for dry events. According to Figure A4, FSS values are higher in August and September, whereas slightly worse forecasts are obtained in June, with the lowest values presented in July. The 2lt forecast is again useful for boxes with sizes n = 1, 2, as we noticed using other verification characteristics. For wet events, different thresholds have different accuracy. Boxes with sizes n = 1, 2 show low FSS, with values in the gradation below the value of useful forecasts. However, as the box size increases, FSS rises for thresholds (of the SPI-1 values) from 1 to 1.6 and in June, 0lt for all thresholds. The model does not produce a score for thresholds above 1.6, meaning that the model can better predict low intensity wet events compared to such with high intensity.

Wet events in model 2 have lower FSS values, and the boxes with sizes n=1, 2 for all months and lead-times also show skills below the usefulness threshold. July shows bad forecast accuracy compared to other months, but without large differences among lead-times. The highest accuracy is obtained in September, while the accuracies for June and August are similar.

We conclude that the FSS is a useful tool to verify the quality of forecasts for SPI gra- dations and helps the user becoming aware of incorrect event gradation. An advantage of the SPI (particular using SPI in FSS analysis, instead of pure precipitation) is its gradation system. Thus, the main challenge for a forecaster is to predict the grada- tion correctly (not predict an exact value), which is increasing the chance of obtain- ing a high accuracy of a forecast. Generally, the FSS is better when verifying larger domains with many grid points; otherwise, with smaller domains, the results must be interpreted more carefully. The FSS helps to overcome the problem of a double penalty and to identify the frequency of its occurrence. This score reflects the general behaviour of the forecast accuracy obtained from traditional methods and can be used

6.5. Forecast verification 119

Figure 6.13: Neighbourhood verification scores using the FSS method for dry events for the period 1982-2016 verified against observed SPI for the same period. a) 0lt, b) 1lt, 2) 2lt, 3) 3lt, for model 1.

as an extra verification tool.

6.5.2

Probabilistic forecast

Probabilistic forecast comparison of both regression models for WA is done using ROC curves, attribute diagrams and CRPS, since they accurately quantify the skills of probabilistic forecasts. All these characteristics are calculated for each month and lead-time to show the deviation in skills when moving from 0 until 3lt during the June- September season for both forecasting models.