Statistical Information Criteria

Section 3.2 introduced the BIC and KIC. However, there are many different kinds of information criteria available for the same model selection purpose. These information criteria include the Akaike information criterion (AIC) (Akaike, 1973; Akaike, 1974), the corrected Akaike information criteria (AICc) (Sugiura, 1978; Hurvich and Tsai, 1989), Mallows’ criteria (Mallows, 1973; Hansen, 2007), the Bayesian information criterion (BIC) (Schwarz, 1978), Hannan and Quinn’s information criterion (Hannan and Quinn, 1979), the Kashyap information criterion (KIC) (Kashyap, 1982), the Kullback information criterion (Seghouane and Bekara, 2004), and the focused information criterion (FIC) (Claeskens and Hjort, 2003).

46

Statistical information criteria have been applied to many disciplines. In the groundwater community, the AIC, AICc, BIC, and KIC are often used for model selection and model averaging purposes. Numerous people have tried to clarify the misunderstandings of the AIC, BIC, and KIC; however, they have never being cleared enough because many of those who attempted to clarify the misunderstandings misunderstood them. A long running debate continues in the model selection and model averaging literature. Comparisons of information criteria with biased data make the argument more confusing. In the following sections, three commonly used information criteria, the AIC, BIC, and KIC, will be analyzed, and their applicability to groundwater inverse problems will be discussed.

3.3.1 Akaike Information Criterion

Akaike (1973, 1974) found a formal relationship between Kullback-Leibler information and likelihood theory, where the maximized log-likelihood value was a biased estimate but this bias was approximately equal to m, the number of estimable parameters. The maximum (log-) likelihood method can be used to estimate the values of the parameters. However, it cannot be used to compare different models without some corrections. The reason for such bias is that the same data are used to estimate the parameters and to calculate the log-likelihood.

The AIC can be written as:





( ) ( ) ˆ( ) ( )

AICp 2ln Pr | p , p 2 p

D   D  β  m (3.12)

where the first term is the sample log-likelihood for the pth _{alternative models, and m is the}

number of independent parameters estimated for the pth _{model. The second term may be viewed}

as a penalty for over-parameterization.

The derivation of the AIC involves the notion of loss of information that results from replacing the true parametric values of a model with their maximum likelihood estimates (MLE) from a sample. The AIC, which does not directly involve the sample size, n, has been criticized

47

as lacking properties of consistency (Bozdogan, 1987). To correct this problem, Sugiura (1978) and Hurvich and Tsai (1989) proposed the AICC, which included the sample size n:

c 2 ( 1) AIC =AIC+ - -1 m m n m  . (3.13)

The AIC is favored by many engineers and researchers. A software package for groundwater modeling called JUPITER-API (Banta et al., 2006) advocates using the AIC. However, based on our understanding, the way to calculate the first term of the AIC in JUPITER is biased. They take an additional log on the fitting residual, which makes the value of the first term in the AIC much smaller. This biased value compensates for the smaller penalty term in the AIC, but would cause poor performance in the BIC and KIC.

Generally speaking, the choice of information criteria strongly depends on particular problems. The AIC will favor complex models over simple models when the sample size is big. In the following sections of this chapter, a variance analysis and a dimensional analysis will be performed on statistical information criteria for a better understanding of all the problems and arguments related to information criteria. From mathematical and statistical grounds, most researchers recommend the BIC rather than the AIC.

3.3.2 Bayesian Information Criterion

The BIC is derived under the assumption that the model parameter follows a multivariate normal distribution with mean βˆ( )p and covariance matrix ₁ ( )p 1

D n



 

 F  (Raftery, 1995). The true distribution of β is never known; otherwise, there is no need to estimate the model parameter ( )p

using inverse methods. The multinormal assumption is a fair assumption under most circumstances for unknown model parameters. In the derivation of the KIC, Kashyap (1982) neglected the term of the prior model parameter distribution, which will cause strange (or special) behavior of the KIC (discussed in the next section).

48

The BIC penalizes overparameterization more heavily than is the case for the AIC, especially with large n. This is consistent with the law of parsimony. In groundwater inverse modeling, adding more artificial parameters will make the fitting residual smaller but will create complex models that do not necessarily honor the true hydraulic conductivity distribution and increase parameter uncertainty. The BIC favors simpler models over complex ones if their fittings are same.

When the observation data increase, the AIC and BIC are able to find the true model if the true model is among the candidate models. In practical groundwater problems and other complex systems, however, the true model is impossible to obtain. It is understood that all the models developed are wrong, and the 100% model weight in BMA does not imply a 100% correct model. The model weights are relative model weights among the selected models, not the true model weights over the model space.

3.3.3 Kashyap Information Criterion

The KIC was originally derived in computer science for auto regression moving average (ARMA) model pattern identification problems (Kashyap, 1982). ARMA models are linear regression models that have been well defined and studied. Carrera and Neuman (1986a) and Neuman (2003) applied it in groundwater model selection and model averaging. The KIC ignores _{ln Pr |}









previous section, the prior distribution of model parameters is unknown. Simply neglecting these terms would lose the information from the prior parameter distribution. Under the BIC assumption, the term that contains the Fisher information term can be canceled out. Section 2.6 has shown that the Fisher information is a quadratic form of groundwater head sensitivity. The GP inverse problem is a nonlinear problem and the head sensitivity is a function of model

49

parameters. The third term in KIC, _ln ( )p D

F , therefore is related to the groundwater head

sensitivity and is a function of model parameters. The value of _ln ( )p D

F varies dramatically with

changes in model structure and varies slightly when the model parameter values change. No evidence shows that these changes have positive or negative correlations with the leading term in the KIC. With these features, the KIC can be used to detect changes of model structures (or patterns). However, the KIC is not recommended for model averaging.

Another problem of the KIC is that if the true model parameters really exist, it cannot identify the true model parameters because the term _ln ( )p

D

F does not agree with the leading MLE term. This problem does not exist in the BIC and AIC. Although the KIC is not recommended in this study, the Fisher information is still useful because it relates to the estimated parameter uncertainty. At the end of this chapter, a numerical example will be used to demonstrate the problems of the KIC.