After all the previous steps we have constructed a statistical forecast but have no real idea of its performance. In our proposed framework judgemental adjustments always comes after, and are based on, the statistical results. Therefore, we desire the statistical results to be as good as possible without manual input. As such, we focus on evaluating the results of the framework without applying judgmental input. In order to properly test performance, we will take several steps to see how well the proposed model works.
4.7.1
Statistical performance
Our main goal is to provide the statistical framework necessary for forecasting and thus we need to know how well the applied methods worked. We will compare accuracy according the MASE for four different forecasting results:
- The current forecasting practice
- The best forecast that resulted from the proposed model before reconciliation - A benchmark combination forecast of ETS, Arima, Tbats and seasonal-naive - The reconciled forecast based on the best results of the proposed model
These 4 collections of forecasts help us determine the value of the proposed method. First, we can compare the results of the proposed approach, before and after reconciliation, to that of the current approach indicating whether more accurate forecasts are actually produced. Then, by including a benchmark of just 4 models we assess the benefits of including a larger flexible selection of models. In order to fairly asses the performance we take the train, test, validate approach from 4.4.4. We fit on 2012-2015 and forecast 2016, then we fit on 2012-2016 and forecast 2017, resulting in three separate measures of accuracy, 2016, 2017 and the average over both. This will teach us whether the proposed method outperforms the current process, does not include too many models and what the effects of reconciliation are.
4.7.2
Individual forecast method performance
We evaluate the best outcomes of the proposed method in Section 4.7.1 but this provides no further insight in the performance of individual methods. We choose to apply a set of multiple models and incur a computational penalty for each we include. Determining whether the included models add accuracy to the forecasts allows us to evaluate if such a large collection of models is necessary. Perhaps some models do not contribute to high performance and could be excluded. Additionally, if only a select number of models are responsible for the best forecasts a, large, ensemble could also be unnecessary. In both cases we require information on the performance of individual methods over the different demand subsets. We propose the following comparisons for each method:
- Best average performance per group vs best average performance while excluding a method - Mean performance per group vs mean performance excluding a method
By comparing the effects a method has on the best performance we can see if there is an immediate impact on overall accuracy. Comparing mean performance lets us see whether the inclusion of a method improves or diminishes overall performance. For instance, we expect that including Croston’s method or iMapa will impact the bottom time series more than it does the higher aggregations. As a result their in- or exclusion might have significant effects on forecast accuracy. By evaluating the performance of each method we can conclude which can be omitted without drastic
72
influence on the results. It will indicate which methods perform significantly better or worse than others as well as provide information on the merits of including more methods.
4.7.3
Total forecast accuracy
All previous comparisons work by comparing a scaled value the MASE in order to say something about the forecast accuracy. This indicates whether improvements in accuracy have been made but not what the actual effect on this is for the organization or the actual numbers. Comparing absolute numbers and errors between groups does not work because of their differences in scale and behaviour but each is a disaggregation of total demand. We propose to sum all forecasts in a group in order to construct 16 different forecasts of total demand. Add the current approach and the reconciled forecast and we can comprehensively compare the differences in accuracy in numbers that are relevant to the organization
4.8
Chapter conclusion
Research question 6: ‘How should the forecasting methods be applied to engineering demand?’ was answered in this chapter by defining our forecasting approach. We combined conclusions from the research goals, the organizational context and theory to define a suitable forecast method. In Chapter 2 we concluded that all subsets contain different information and should be included. In Chapter 3 we found that different models are suited for different data. This resulted in the conclusion that no single model is suitable to forecast all subsets and multiple would be needed. Additionally, we concluded that manual application and tuning of these models would require too much time.
We propose that engineering demand should be forecast with multiple model to use as much of the available information as possible. By combining the forecasts from all models we produce forecasts that combine the different information and produce more accurate forecast. In order to select the best resulting forecast from all these combinations we measure their accuracy with the MASE. This is done for every node in the demand structure resulting in a collection of accurate but incoherent forecasts. These forecast can then be reconciled in order to align all levels of the structure with each other sharing information from the different forecasts over all the levels. With the statistical forecasts in place we can measure and compare performance of different approaches, e.g. current practice vs reconciled forecasts, and determine the most accurate results. Allowing us to conclude whether a more complex approach is useful. Finally, the proposed model then initiates a feedback loop to evaluate the applied models and improve performance where possible.
The proposed framework provides a structured approach to forecasting. There are clear defined steps and trust is placed on the statistical outcomes. An initial setup for applying judgemental forecasting is provided but good judgemental forecasting is dependent on trustworthy statistical forecasts. Therefore we omit further testing in favour of focussing on the statistical framework as our main goal. With the different statistical forecasts in hand we have defined how to compare their results. Chapter 5 is dedicated to evaluating the forecasts and determining the organizational impact of more accurate forecasts.
73
5
Model performance and results
Chapter 4 provides a guide on how to implement the proposed forecasting method. It focusses on producing trustworthy statistical forecasts, evaluating them which leads to, judgemental, reassessment of the model and its forecasts, see Figure 4.1 for a summarized overview. The proposed method is deemed successful if we can improve over forecasts from the current process. Defining and comparing performance also provides an answer to research question 7.
We focus on the results from the statistical framework while not testing judgmental adjustments. Section 3.2 describes how the first step of good forecast has its foundation in an accurate and trustworthy quantitative forecast which is our main focus. As such we discuss the results of the statistical forecasts, compare different methods and describe the organizational impact. We follow the approach from Section 4.7 to evaluate the outcomes. Section 5.1 discusses the results per demand group and compares our method, the current approach, a benchmark combination, and the effects of reconciliation. Section 5.2 discusses the sensitivity of our method to the inclusion or exclusion of single models. Section 5.3 attempts to discuss the organizational implications of increased forecast accuracy and Section 5.4 presents our conclusions about the forecasting results.