General Standards of Model Validation

The Basel Committee requires banks to establish robust procedures and methodologies to validate if internal risk models are conceptually suited and if they accurately and adequately represent the material risk.43 Supervisory recognition of risk models under the IRB approach and internal market risk models of the revised capital standards require: (1) an initial approval of the model based on specified minimum requirements, (2) an independent review process established to assess the risk systems and processes, and (3) a supervisory review of the validation process.

The initial approval of risk models takes into account the documentation of risk proce- dures, models and systems, the consistency of type and source of data, the methodological and statistical concept of the risk model, procedural requirements on the estimation and validation of risk measures as well as the independence and qualification of staff engaged in developing and operating risk models.44 _{Guidelines for the model approval are industry}

standards and established results from academic research.

Banks take on the primary responsibility for the review of risk measurement systems and processes. The independent review of the internal risk measurement systems for market risks involves the verification of approval, documentation, a change-control for all risk measurement processes and systems, the integration of risk measurement into the risk management function, the consistency, timeliness and reliability of independent data sources, the accuracy and completeness of exposures, the accuracy in parameters in estimation, risk measurement and validation models, and finally, the quantitative model validation using backtesting.45 _{Backtesting involves all validation techniques that compare}

estimates of risk components to actual outcomes using statistical test theory. Supervisors do not stipulate specific techniques or definite criteria for the validation of risk models, but they require banks to develop the expertise for a self-reliant qualitative and quantitative assessment of applied risk models.

43_{Cf. BCBS (2006c), p. 109ff and p. 202.} 44_{Cf. BCBS (2006c), p. 88ff. and p. 254ff.} 45_{Cf. BCBS (2006c), p. 193.}

The review of internal rating models for the estimation of PD, LGD and EAD is divided into a procedural validation of the rating process and a methodological validation of the risk model. The evaluation of the rating process is qualitative in nature and involves data quality, internal reporting, problem identification and handling, system usage, staff training, and consistent application of the risk model throughout business lines and geo- graphic regions. The methodological validation consists of an assessment of model design and a quantitative assessment of the quality of predictions. The examination of the model design includes a qualitative review of the statistical concept, the relevance of the input data, the way risk factors were selected and whether they are economically meaningful. Quantitative validation involves backtesting and the comparison of estimates to external data sources. Quantitative tests must not vary systematically during the economic cycle.

Requirements similar to those specified for market risk measurement systems and models for the estimation of risk components under the IRB approach are assumed to apply to the validation of internal credit portfolio models as well, although, the use of internal models to set capital requirements for credit risk has not yet been approved and standards for the validation of credit portfolio models are not specified.

The validation procedures conducted by supervisory agencies are specified on a general level to avoid a limitation of the competence of supervisors. Supervisory agencies review the adequacy of risk assessments and derived capital requirements, the compliance with minimum standards and qualifying criteria and the effectiveness of the review of the risk assessment processes.46

The quantitative validation of risk models has two objectives: (1) testing the accuracy of risk measures and risk parameter estimates, and (2) assessing the adequacy of a model to accomplish the overall objectives of the risk measurement process.47

Risk parameters that are typically controlled for the accuracy of estimates are PD, LGD, EAD and credit correlations of individual exposures or exposure classes. Accurate risk models implement a timely and accurate estimation of risk measures. However, statistical tests are limited in their ability to distinguish between accurate and inaccurate models or risk estimates, since the power of a hypothesis test to avoid committing a type II error, i.e. to reject the null hypothesis when it is actually false, cannot be calibrated to unity by a suitable specification of the test. The Basel Committee acknowledges that it is not possible to define a statistical test to correctly identify inaccurate models that prevents giving erroneous negative indications on other accurate models.48 _{According to}

the limitations of hypothesis tests, statistical evidence on the accuracy of a risk model is

46_{Cf. BCBS (2006c), p. 209ff.} 47_{Cf. BCBS (2006c), p. 193.} 48_{Cf. BCBS (1996b), p. 5.}

not mandatory for a model to receive approval from supervisory agencies, because even risk models whose accuracy cannot be verified can nevertheless be considered to provide adequate risk forecasts. The current backtesting procedure for internal market risk models implements this view.

Model adequacy refers to the ability of a model to reach an appropriate decision under uncertainty to meet pre-specified objectives. Pesaran and Smith (1985) stipulate the following criteria to assess the adequacy of a risk model:

• Does the model comply with the requirements of the user? (relevance criteria)

• Does the model contradict to secured knowledge? (consistency criteria)

• Is reliability ensured on a satisfactory statistical level? (statistical adequacy) Credit portfolio models are required to ensure capital adequacy of a bank with respect to the credit risk of the bank’s credit portfolio. The consistency of a model is assumed to be given, if the conceptual design implements established methodologies of risk modelling and if the definition of model parameters coincide with those of parameter estimates. Statistical adequacy requires that the relevance criteria be met on a specified confidence level.

Quantitative methods of assessing the adequacy of credit risk models are sensitivity anal- ysis, stress tests and backtesting. Sensitivity tests of credit portfolio models examine the dependence of capital requirements and risk measures on a change in PD, LGD, EAD, correlations and credit spreads. Furthermore, the impact of a change of risk con- centrations within particular economic sectors, regions or rating classes is assessed. For mark-to-market credit portfolio models, the effect of a change in drift, reversion, volatility or correlation parameters is of interest.

Stress tests examine the effect of adverse circumstances on risk measures and capital adequacy. Stress tests typically take the form of a scenario analysis and represent either an absolute or a relative change of several model parameters and market factors. Instead of the directional change of risk measures as observed in sensitivity analysis, stress tests consider the absolute outcome of a risk measure.

Regarding validation methodologies different approaches are distinguished for default pre- diction models, rating models, and portfolio credit risk models. Default prediction models discriminate between prospectively defaulting and non-defaulting exposures to support credit approval and prolongation decisions. Rating models assess effective and potential credit exposures on a fine-grained scale for risk-adequate pricing, assessment of general and specific credit loss provisions, determination of impairments, economic loss and capi- tal requirements. Risk parameters estimated by rating models comprise PD, LGD, EAD

or correlations. Credit portfolio models provide risk measures to assess the unexpected loss of portfolios and the capital requirements of banks.

Backtesting the statistical adequacy of credit risk models either involves testing the dis- criminatory power of default prediction models, the accuracy of risk parameter estimates, or the adequacy of portfolio credit risk measures to specify capital requirements. Tech- niques for validating the effectiveness of default prediction models are presented in Section 2.4.2. Backtesting risk parameter estimates involves hypothesis tests as outlined in Sec- tion 2.4.3 for the validation of PD estimates. Drift, recursion and diffusion parameters of credit portfolio models have not been considered in backtesting studies yet. Backtesting the adequacy of a credit portfolio model involves verifying whether the actual portfolio credit loss is in line with unexpected loss and corresponding economic and regulatory capital.49

Data limitations are a key impediment to the estimation and validation of credit risk models. The scarcity of data is due to the infrequent nature of default events and long- term time horizons in risk measurement. In consequence, data pooling, proxy data and low-frequency data are frequently used in model estimation and validation. Compared to market risk models, the one-year holding period and the higher quantile of Credit- VaR complicates backtesting. A statistical confidence equal to the backtesting of market risk models would require the excessive number of 250 observations of actual one-year portfolio outcome in backtesting credit portfolio models. To qualify for the advanced IRB approach, banks are required to establish a track record of historical estimates and actual rates of PD, LGD, EAD, as well as rating histories including the dates of any rating review for a time interval of at least five years.50

Corporate loans are typically not marked-to-market, so that risk models cannot be esti- mated from time series of loan valuations and the predictive quality of portfolio models cannot be compared to observed portfolio outcomes. Instead, it is assumed that mark-to- model valuations of credit exposures without observable market prices are derived from updated ratings to determine the credit loss of exposures. Thus, the validation of the credit portfolio model relies on the accuracy of the internal rating system, which is it- self subject to a separate validation process. In this context, the inaccurate recognition of significant credit losses that accumulate in the banking book unnoticed due to mis- specified PD estimates is the major threat to an adequate capital endowment and affects rating-based capital standards as well as credit portfolio risk measures.

49_{The comparison of expected and actual portfolio credit loss does not address the capital adequacy}

purposes but only credit provisioning and pricing.