In this chapter, twenty-four regression techniques were evaluated on six real-life datasets obtained from major international bank- ing institutions. The average performance of the models in terms of R2 ranged from 4% to 43%, showing that several resulting mod- els have limited explanatory power. These rather weak performance results are quite similar to those obtained in previous LGD forecast- ing studies. Nonetheless, a clear trend can be seen that non-linear techniques, and support vector machines and artificial neural net- works in particular, yield significantly higher model performance than more traditional linear techniques. This suggests the presence of non-linear relations between the independent variables and LGD, contrary to previous benchmarking studies on PD modeling where the difference between linear and non-linear techniques was not that explicit. Therefore, the study clearly demonstrated the potential of applying non-linear techniques to LGD modeling, possibly in the form of first order regression splines so as to yield good predictive performance while offering the advantage of being well interpretable.
3
Backtesting LGD models
”When I see articles with lots of significance tests, I say that the statisticians are p-ing on the research.” -Herman Friedmann (American statistician, 1930-2010)
”The only relevant test of the validity of a hypothesis is comparison of prediction with experience.” -Milton Friedman (American economist, 1912-2006)
The Basel accords require financial institutions to regularly validate their LGD models. This is crucial so banks are not underestimat- ing or overestimating the minimal required capital to protect them against the risks they are facing through their lending policies. The validation of an LGD model typically includes backtesting which is the process of evaluating to which degree the internal model es- timates correspond with the realized observations. Current back- testing practices are limited to solely measuring the similarity be- tween model predictions and realized observations. It is however
not straightforward to determine acceptable performance based on these measurements. Although recent research lead to advanced backtesting methods for PD models, literature on similar backtest- ing methods for LGD models is non-existing. This study addresses this literature gap by proposing a backtesting framework with sta- tistical hypothesis tests to support the validation of LGD models. The proposed statistical hypothesis tests implicitly define reliable reference values to determine acceptable performance and take into account the number of LGD observations which may influence the quality of the backtesting procedure. The workbench of statistical hypothesis tests is applied to an LGD model based on real-life data. Special attention is given to the evaluation of the statistical power of the proposed tests.
3.1
Introduction
Banks are required to regularly validate the internal estimation pro- cess and the internal models so as to prove their soundness to the national regulator (6). The validation of the estimation process in- volves issues like data quality, reporting and problem handling and how the predictive models are used by the bank. The validation of the estimation process is mainly qualitative in nature, although quantitative methods are useful for the examination of data quality. The validation of the models on the other hand includes both the examination of the model design and the predictions it produces. The evaluation of the model design consists of a qualitative review of the statistical techniques and the relevance of the data used to
3.1 Introduction
build the model. The assessment of a model’s predictions typically includes quantitative methods as benchmarking and backtesting. While benchmarking methods evaluate the internal model estimates with external model estimates (88), backtesting methods evaluate the internal model estimates with the actual realized observations. The purpose of backtesting is to evaluate the predictive performance of a model and to assess its time evolution to detect model deterio- ration in a timely manner. An LGD model can experience reduced predictive performance when current loan loss behavior does not reflect previous loan loss behavior anymore on which the model is built. This may lead to an overestimation or underestimation of a bank’s required minimal capital so that its operations can become less profitable or more risky respectively. Although banks are re- quired to validate their models in order to be Basel compliant, the accord does not mention how to perform the validation (6). In ad- dition, recent research has largely focused on advanced methods for backtesting PD models (70, 71, 72) but literature on comparable methods for backtesting LGD models is non-existing.
Current LGD backtesting practices are usually limited to comparing internal LGD predictions and realized LGD observations with error based metrics, correlation based metrics or even classification based metrics (88). It is however not straightforward to determine ac- ceptable performance solely based on these metrics. A single value has little meaning without an appropriate reference value indicating acceptable accuracy. Additionally, these metrics do not take into ac-
count the number of LGD observations. When the portfolio lacks sufficient observations, a few extreme observations can distort the accuracy result and so degrade its reliability. This study proposes a backtesting framework where the model performance on test data is evaluated with respect to the model performance on training data with appropriate statistical hypothesis tests. Hence, an appropriate reference value is introduced while the number of observations is implicitly taken into account.
The remainder of this study is organized as follows. First, a litera- ture review is conducted on empirical LGD studies which focus on the evaluation of the predictive performance of LGD models. Sec- ond, the key idea of the proposed backtesting procedure is explained together with the workbench of appropriate statistical hypothesis tests to evaluate LGD models. Third, the experimental set-up to apply and to evaluate the backtesting framework is described. This involves information about the employed real-life LGD data, the design of a predictive LGD model based on this data, a statistical significance analysis of the measured predictive model performance and a statistical power analysis of the proposed tests based on these performance metrics. Forth, the results of the backtesting procedure applied to a real-life LGD model is reported and discussed.