Evaluating Volatility Forecasts

There are a variety of statistics available to evaluate and compare the

are used to evaluate the accuracy of forecasts: viz., the mean squared forecast error (MSFE) and the mean absolute forecast error (MAFE). These measures are defined as:

where y t+h is the realization of the series at time t + h , y4 h is the /z-step ahead forecast of the series using data observed up to and including time t, and T2 is the number of /z-step-ahead forecasts considered. The MSFE provides a quadratic loss function which disproportionately weights large forecast errors more heavily relative to the MAFE and thus may be useful in forecasting situations when large forecast errors are disproportionately more serious than small errors.

We compute the MSFE and the MAFE for each set of one-step-ahead forecasts generated. We then rank each of the thirteen approaches according to which produces the most accurate forecasts. Starting with the MSFE, we first rank the approach that gives the smallest MSFE statistic. Next, we take the second smallest MSFE statistics produced by the next relevant approach. The process continues until all approaches are ranked accordingly. We then repeat the same process with the MAFE statistics for all the volatility modelling approaches. At the end of the exercise, we have two sets o f forecasting performance rankings for T1 to T13. Finally, we

7 For a detailed survey on the popular evaluation measures used in the literature, please refer to Poon

performance of forecasts produced by volatility models.7 In this study, two measures

10 t = \

(5.2)

MAFE (5.3)

compute the MSFE and MAFE for the unconditional variance estimator for comparison purposes.

To compare the predictive accuracy of alternative forecasts, we employ an asymptotic test of the null hypothesis of no difference in the accuracy of two competing forecasts proposed by Diebold and Mariano (1995). This is a convenient test of the null hypothesis that the forecasts from two models do not differ significantly. We assume that the time t loss associated with a forecast i is an arbitrary

A

function of the realization and prediction, g (yt, y it) ; specifically, it is assumed to be a

A

direct function of the forecast error, g (y ,9y u) = g(eit). Under this assumption, the null hypothesis of equal forecast accuracy for two competing forecasts is E[g(ej,)] = E [g (eJ,)] or E [d, ] = 0, where dt = [g(e;,) - g(eJt)] is the loss differential. Thus, the equal accuracy null hypothesis is equivalent to the null hypothesis that the population mean of the loss-differential series is zero.

Let (5.4)

2 /=1

denote the sample mean loss differential over T2 forecasts, and let g{eit)be a general function of forecast errors (e.g. MAFE or MSFE); then the Diebold Mariano (1995) test statistic (henceforth denoted DM) is given by:

DM = - f - . (5.5)

2^ , ( 0)

1 t 2

A

where / </(0)is a consistent estimate of the spectral density of the loss differential at frequency zero. The DM statistic has a standard normal distribution with mean zero and unit variance under the null hypothesis. The loss functions adopted in this study

are the squared function (MSFE) and the absolute function (MAFE). We proceed by comparing the MSFE and MAFE values associated with the forecasts of each of the thirteen approaches, T1 to T13, with the naive model, denoted by M l. Next, we apply the DM asymptotic test for each competing pair of forecasts among the thirteen volatility modelling approaches. This exercise produces 78 forecast quality comparisons.

We then apply the test of forecast encompassing developed by Harvey, Leyboume and Newbold (1998). Forecast encompassing refers to whether or not the forecasts from a competing model contain information that is missing from the forecasts of the original model. If they do not, then the forecasts from the competing model are said to be ‘encompassed’ by the forecasts from the original model. Furthermore, a forecast is considered ‘conditionally efficient’ if the variance of the forecast error from a combination of that forecast and a competing forecast are equal to or greater than the variance of the original forecast error. Therefore, a forecast that is conditionally efficient ‘encompasses’ the competing forecast. Harvey, Leyboume and Newbold (1998) developed an encompassing test based on the fact that if the forecasts from model 1 encompass the forecasts from model 2, then the covariance between eu and eit- ^2t will be negative or zero (eu and e2t are the two sets of forecast errors from model 1 and model 2 respectively). The alternative hypothesis is that the forecasts from model 1 do not encompass those from model 2, in which case the covariance between eu and eu - e2t will be positive. The Harvey, Leyboume and Newbold (1998) test statistic (henceforth denoted HLN) is formulated as follows:

where c - — 2_jCt , ct = eu (eu - e 2t) , and T2 is the number of observed forecasts. The T i t=i

HLN statistic has an asymptotic standard normal distribution. Similar to the steps applied in the DM asymptotic test above, we compare the unconditional variance estimator with each forecast of the individual volatility modelling approaches, T1 to T13. Finally, we proceed by comparing the forecasts for each pair of competing volatility modelling approaches. This again results in 78 forecast quality comparisons.