Top PDF Evaluating density forecasts: a comment

One point of MW’s criticism is that GBR are concerned with forecasting of white noise instead of more relevant forecasting of dependent time series. However, what is really important is that GBR do not pay due attention to the conditional nature of calibration. 3 MW define “complete calibration” as independence and uniformity of the probability integral transform (PIT) values of a time series. In- deed, it is very true that the available history of a time series (including the PIT values) is an important information for judging calibration of forecasts of this series, but in general this view is unnecessarily narrow. The notion of forecast calibration is not about time series properties, it is about conditioning on forecasting information.

The issues described in those working papers stem from the fact that the prediction produced by a density forecasting model can rarely be compared to the true generating distribution in real world problems. Instead, only a single instance of the generating distribution, the actual outcome, is available to the forecaster to optimize and evaluate their model. Conventional diagnostics for evaluating point predictions such as the root-mean-squared error (RMSE) and others fail to assess probabilistic predictions. Furthermore, the ranking of different density forecasting models is difficult because a ranking depends on the loss function of the user (Diebold et al. 1998). For example, a user’s loss function could be non-linear and/or asymmetric. In such cases the mean and variance of the forecast densities are not sufficient to rank predictive models. For example, a user with an asymmetric loss function would be particularly affected by the accuracy of a model’s predictions of the skew in the conditional densities. Diebold et al. (1999) suggests that the problem of ranking density forecasts can be solved by assuming that the correct density is always preferred to an incorrect density forecast. Using the true density as a point of reference it is possible to rank densities relative to the true densities to determine the best models to use. Therefore, in the absence of a well defined loss function, the best model is the one that approximates the true density as well as possible. Diebold et al. (1998) go on to suggest the probability integral transform (PIT) as a suitable means of evaluating density forecasts in this way.

A common application for evaluation techniques is model selection. Here, the goal is to determine the best model set up to use for your final prediction model. In this sim- ple example we show how it is useful to plot a Pareto evaluation plot to determine the optimal models. For this experiment we used a Mixture Density Network (MDN) to make our density forecasts[27]. MDNs are an adaptation of the multi-layer perceptron that can accurately estimate conditional probability density functions by outputting a Gaussian Mixture Model (GMM). Like most neural networks there are a number of variables that must be decided upon by the modeler before the model can be trained. The two most important variables to be selected with this type of model are the number of hidden units in the network architecture and the number of Gaussians to be included in the GMM. In our experiment we use the Pareto optimality plot described above to determine the best model set up in terms of hidden units and Gaussian components for a simple inverse problem. Target variables, t, are uniformly drawn from the interval [0,1] and the input variables, x, are generated by x = t + 0.3 sin(2πt) + ǫ and ǫ is uniform noise drawn from the interval [-0.1, 0.1] 6 . We created a training and test set of 1,000

information to produce a satisfactory forecasting exchange rate model. While previous research on term structure models has analyzed forecasting performance focusing primarily on accuracy evaluations based on point forecasts, several authors have recently emphasized the importance of evaluating the forecast accuracy of economic models on the basis of density - as opposed to point - forecasting performance. Especially when evaluating nonlinear models, which are capable of producing non-normal forecast densities, it would seem appropriate to consider a model’s density forecasting performance. This is indeed the primary objective of the empirical work undertaken in this paper, where we carry out density forecasting tests on the linear VECM and the MS-VECM of the term structure of forward premia as well as on a random walk exchange rate model. We then investigate some of the implications of our density forecasting results for exchange rate risk management.

This paper considers a vector autoregressive model (VAR) model with stochas- tic volatility which appeals to the Inverse Wishart distribution. Dramatic changes in macroeconomic time series volatility pose a challenge to contemporary VAR forecasting models. Traditionally, the conditional volatility of such models had been assumed constant over time or allowed for breaks across long time periods. More recent work, however, has improved forecasts by allowing the conditional volatility to be completely time variant by specifying the VAR innovation vari- ance as a distinct discrete time process. For example, Clark (2011) specifies the elements of the covariance matrix process of the VAR innovations as linear func- tions of independent nonstationary processes. Unfortunately, there is no empirical reason to believe that the VAR innovation volatility processes of macroeconomic growth series are nonstationary, nor that the volatility dynamics of each series are structured in this way. This suggests that a more robust specification on the volatil- ity process—one that both easily captures volatility spill-over across time series and exhibits stationary behaviour—should improve density forecasts, especially over the long-run forecasting horizon. In this respect, we employ a latent Inverse Wishart autoregressive stochastic volatility specification on the conditional vari- ance equation of a Bayesian VAR, with U.S. macroeconomic time series data, in evaluating Bayesian forecast efficiency against a competing specification by Clark (2011).

The full-sample SEF data in Tables 1 and 2 show a greater tendency towards similarly “favourable” scenarios in respect of GDP growth than inflation, with this again increasing as the forecast horizon (or question number) increases. However in the two-year-ahead inflation forecasts the deviations are more evenly balanced, suggesting that, in an inflation targeting regime, a favourable scenario is one in which the official target is achieved in the medium term, and this may lead to a positive or negative adjustment to an initial forecast. The aggregate data for the regular respondent subsample show the same patterns, although these data mask considerable variation across the 19 individual rows of each table. We note that the interpretation of Engelberg et al. (2008) that “forecasters who skew their point predictions tend to present rosy scenarios” implicitly uses the density forecast as the base for the

Nsoesie et al. [16] reviewed different studies in the field of forecasting influenza outbreaks and presented the features used to evaluate the performance of proposed methods. Eleven of the sixteen forecasting methods stud- ied by the authors predicted daily/weekly case counts [16]. Some of the studies used various distance functions or errors as a measure of closeness between the predicted and observed time-series. For example, Viboud et al. [17], Aguirre and Gonzalez [18], and Jiang et al. [19] used cor- relation coefficients to calculate the accuracy of daily or weekly forecasts of influenza case counts. Other studies evaluated the precision and “closeness” of predicted activ- ities to observed values using different statistical measures of error such as root-mean-square-error (RMSE), per- centage error [19, 20], etc. However, defining a good distance function which demonstrates closeness between the surveillance and predicted epidemic curves is still a challenge. Moreover, the distance function provides a gen- eral comparison between the two time-series and ignores the epidemiological relevance between them, which are more significant and meaningful from the epidemiolo- gist perspective; these features could be better criteria to compare epidemic curves together rather than simple distance error. Cha [21] provided a survey on different distance/similarity functions for calculating the closeness between two time-series or discrete probability density functions. Some other studies have analyzed the overlap or difference between the predicted and observed weekly activities by graphical inspection [22]. Epidemic peak is one of the most important quantities of interest in an out- break, and its magnitude and timing are important from the perspective of health service providers. Consequently, accurately predicting the peak has been the goal of some

By evaluating the SETAR models over the entire forecasting sample we have found that none of the models was able to produce ‘good’ density and interval forecasts in general, while the density and interval forecasts produced by the GARCH model were correctly conditionally calibrated at each level of the evaluation study. The correct calibration or not of the various regions of the density has been illustrated by cumulative probability plots of the probability integral transforms against the uniform (0,1), and also assessed by the X 2 goodness-of-fit test and its individual components. The decomposition of the goodness-of-fit test into individual components has enabled us to explore possible directions of departures more closely, indicating major departures for the SETAR models with respect to scale and kurtosis.

The forecast evaluation results in the previous section suggest that the forecasts from the model by Smets and Wouters (2007) are more precise than those of the other considered DSGE models. How- ever, deviding the evaluation sample into three subsamples, that cover about five years each shows that the Smets & Wouters model does not continuously outperform the other DSGE models. Table 3 shows for three subsamples the model with the lowest RMSE for the different forecast horizons and for output growth, inflation and the federal funds rate, respectively. There is no model that continu- ously performs better than all other models. Even for a given subsample the most precise forecasts for the different variables can be generated by different models. Different frictions in different mod- els seem to be useful for forecasting specific variables in certain periods only, while other frictions are more important for other periods. The problem of instability in the performance and predictive content of different forecasting methods is well known and surveyed in detail in Rossi (2012). Rossi (2012) provides a survey of various contributions that show that combined forecasts are one possibil- ity to overcome such instability in the forecasting performance of nonstructural models. Timmermann (2006) surveys the literature on forecast combinations and concludes that combining multiple fore- casts increases the forecasting accuracy in most cases. Unless one can identify a single model with superior forecasting performance, forecast combinations are useful for diversification reasons as one

The results regarding the impact on forecast accuracy are recorded in table 5. The table records the results for btt and os-forecasts separately. For each forecast horizon h = 1; : : : ; 4 we report: the average accuracy of all the individual forecasts, where we average the squared forecast errors over all respondents and surveys; the number of NC and non-NC forecasts (either btt or os); and the results of replacing the NC forecasts by the arti…cial forecasts. The columns headed ‘btt[os]-ratio MSFE’report the MSFE of the y e forecasts to the MSFE of the reported forecasts. By and large, the adjusted forecasts are generally more accurate than the originals, with the exception of the CPI forecasts, indicating that individuals’ NC-behaviour worsens forecast performance. For CPI in‡ation, on the other hand, the ‘smoothed’counterfactual forecasts are roughly 10% to 15% less accurate. These results are clearly at odds with those for the consensus forecasts for all variables other than the CPI. To investigate further, we calculate the average absolute forecast error over all respondents and surveys (instead of the squared error) to check whether the average measure is being unduly in‡uenced by a few large errors (possibly resulting from idiosyncratic reporting errors, etc): see the columns headed ‘btt[os]-ratio MAE’. There is now less evidence that NC-behaviour clearly harms forecast accuracy, but by and large little evidence for the positive e¤ect found for the median forecasts (except for the CPI).

In the case of forecasting the decisions people make in conflicts, the evidence shows that the forecasts of experts using their unaided judgment are no more accurate than chance. A reasonable default method for these forecasting problems is therefore the equal-likelihood forecast, whereby equal probabilities are allocated to each outcome option. The denominator in the Exhibit 2b formula is a simplification of the Brier score calculation for the equal-likelihood forecast, and k is the number of outcome options.

As a result of the low price regime adminis- tered by OPEC since 1986, the US dependence on oil imports has kept increasing, so that these imports in 1990 accounted for about half of the[r]

Due to the unobserved nature of the true return variation process, one of the most challenging problems in evaluation of volatility forecasts is to find an accurate benchmark proxy for ex - post volatility. This paper uses the Austral- ian equity market ultra-high-frequency data to construct an unbiased ex - post volatility estimator and then use it as a benchmark to evaluate various practic- al volatility forecasting strategies (GARCH class model based). These fore- casting strategies allow for the skewed distribution of innovations and use various estimation windows in addition to the standard GARCH volatility models. In out-of-sample tests, we find that forecasting errors across all model specifications are systematically reduced if using the unbiased ex - post volatil- ity estimator compared with those using the realized volatility based on sparsely sampled intra-day data. In particular, we show that the three bench- mark forecasting models outperform most of the modified strategies with dif- ferent distribution of returns and estimation windows. Comparing the three standard GARCH class models, we find that the asymmetric power ARCH (APARCH) model exhibits the best forecasting power in both normal and fi- nancial turmoil periods, which indicates the ability of APARCH model to capture the leptokurtic returns and stylized features of volatility in the Aus- tralian stock market.

The model we have just discussed mimics the Survey of Professional Forecasters (SPF) data set. The survey is mailed four times a year, on the day after the first (preliminary) release of the NIPA data for previous quarter. Forecasters are asked to return the survey before the middle of each forecasting quarter. Therefore, even though forecasters share common information about previous quarters, they also have some private information gathered during the forecasting quarter. During the 45 - 60 days from the end of previous quarter to when they actually report their forecasts, respondents can obtain partial information about current quarter from many objective sources. Forecasts may also reflect individual experiences in specific markets and forecasters’ beliefs regarding the effects of current “news” on future inflation. 12 Thus the data generating process of SPF is consistent with the simple model we have suggested. This framework provides a guide to compare survey measures from SPF data and time series measures of forecast uncertainty.

Hlavac 4 In a recent research paper,3 health economists David Cutler and Adriana Lleras-Muney analyze data primarily from the National Health Interview Survey NHIS, an annual crosssectio[r]

Although wind turbines are able to turn to face the wind, it has been suggested that the relationship between wind power and wind speed is, to some extent, dependent on the wind direction. Potter et al. (2007) find that the uncertainty in the relationship depends on the wind direction. Nielsen et al. (2006) include a wind direction variable into the relationship to explain turbine wake effects and direction dependent bias of the meteorological forecasts. nche (2006) recognizes that wind direction influences the performance of a wind farm, and so uses it in a wind power prediction model. In Figure 5, we plot wind power against wind speed using different symbols to show the data points for selected wind directions. The plots suggest that the variability in the relationship can depend on wind direction. For Aeolos, south-westerly wind seems to produce a higher degree of variability in the relationship, and for Rokas, south-westerly wind shows higher variability than north-westerly when the wind speed is below about 13 m/s.

This paper analyses and compares two data-driven approaches to perform density forecasts of mixed causal-noncausal autoregressive (hereafter MAR) models. MAR models incorporate both lags and leads of the dependent variable with potentially heavy-tailed errors. The most commonly used distributions for such models in the literature are the Cauchy and Stu- dent’s t -distributions. While being parsimonious, MAR models generate non-linear dynamics such as locally explosive episodes in a strictly station- ary setting (Fries and Zako¨ıan, 2019). So far, the focus has mainly been put on identification and estimation. Hecq, Lieb, and Telg (2016), Hencic and Gouri´eroux (2015) and Lanne, Luoto, and Saikkonen (2012) show that model selection criteria favour the inclusion of noncausal components ex- plaining respectively the observed bubbles in the demand of solar panels in Belgium, in Bitcoin prices and in inflation series. Few papers look at the forecasting aspects. Gouri´eroux and Zako¨ıan (2017) derive theoretical point and density forecasts of purely noncausal MAR(0,1) processes with Cauchy-distributed errors, for which the causal conditional distribution ad- mits closed-form expressions. With some other distributions however, like Student’s t, conditional moments and distribution may not admit closed- form expressions. Lanne, Luoto, and Saikkonen (2012) and Gouri´eroux and Jasiak (2016) developed data-driven estimators to approximate them based

• When there is uncertainty in forecasting, forecasts should be conservative. Uncertainty arises when data contain measurement errors, when the series are unstable, when knowledge about the direction of relationships is uncertain, and when a forecast depends upon forecasts of related (causal) variables. For example, forecasts of no change were found to be more accurate than trend forecasts for annual sales when there was substantial uncertainty in the trend lines (e.g., Schnaars and Bavuso 1986). This principle also implies that forecasts should revert to long- term trends when such trends have been firmly established, do not waver, and there are no firm reasons to suggest that they will change. Finally, trends should be damped toward no-change as the forecast horizon increases.

One of the features in estimation with DSGE models is imposing restrictions on the comovements between macroeconomic variables from the point of view of the DSGE model. The higher forecast performance of the model with the financial friction compared to the model without friction reflects the presence of comovements generated by the friction in the data during the period. It is suggested that the causality of the financial accelerator exists with a higher probability than that of the frictionless model. And we estimated when and the extent to which the comovements generated by both DSGE models change in terms of time series through changes of the time-varying model weights, realizing the optimal combination of density forecasts. These gave us the clues to which conditions in economic situations contribute to changes of the comovements. Furthermore, we conducted a robust check to examine whether a similar dynamic change of the weight is observed when using the dynamic prediction pooling method by Del Negro et al. (2016) in this paper.

High levels of malonaldehyde are found in rancid foods. Malonaldehyde is a decomposition product of polyunsaturated fatty acids. Edible oils are one of the main constituents of the diet used for cooking purposes. Oils with lower values of viscosity and density are highly appreciable to consumers as they signify good quality fresh products. Temperature affects the quality of edible oils. The effect of temperature on the physicochemical characteristics and rancidity of two edible oils (corn and mustard oils) were analyzed. Results revealed that due to the temperature change in the oil there is a notable difference in the spectral band which showed that the proportions of the fatty acids were changed and thus, becoming soured or rancid.