A bootstrap approach to evaluating the performance of Akaike Information Criterion (AIC) and Bayesian Information Criterion (BIC) in selection of an asymmetric price relationship

(1)

A BOOTSTRAP APPROACH TO EVALUATING THE PERFORMANCE OF AKAIKE INFORMATION CRITERION (AIC) AND BAYESIAN

INFORMATION CRITERION (BIC) IN SELECTION OF AN ASYMMETRIC PRICE RELATIONSHIP

Henry de-Graft Acquah* University of Cape Coast,

Department of Agricultural Economics and Extension, Ghana

Abstract: This study addresses the problem of model selection in asymmetric

price transmission models by combining the use of bootstrap methods with information theoretic selection criteria. Subsequently, parametric bootstrap technique is used to select the best model according to Akaike’s Information Criteria (AIC) and Bayesian Information Criteria (BIC). Bootstrap simulation results indicated that the performances of AIC and BIC are affected by the size of the data, the level of asymmetry and the amount of noise in the model used in the application. This study further establishes that the BIC is consistent and outperforms AIC in selecting the correct asymmetric price relationship when the bootstrap sample size is large.

Key words: model selection, Akaike’s Information Criteria (AIC), Bayesian

Information Criteria (BIC), asymmetry, bootstrapping.

Introduction

An asymmetric price relationship is useful for the assessment of the existence of asymmetries in agricultural markets. It can be used to answer some important questions such as whether prices rise faster than they fall. The asymmetric price transmission models are often used to derive the magnitude and direction of asymmetry, which are used as reference points for addressing the asymmetries in the markets.

However, the determination of the asymmetric relationship is perhaps among the most difficult tasks in price transmission analysis. This is because of the existence of many competing asymmetric price transmission models. There are many alternative specifications of asymmetric price transmission models. However, the different asymmetric models can result in quite different conclusions. One therefore needs to have a basis for choosing among the different asymmetric

(2)

relationships. This simulates interest in model selection methods such as Akaike’s Information Criteria (AIC) (Akaike, 1973) and Bayesian Information Criteria (BIC) (Schwarz, 1978) which provides a basis for addressing the model selection problems. However, very little is understood about relative performance of AIC and BIC in an asymmetric price transmission modelling context.

In an attempt to understand the relative performance of AIC and BIC in price analysis, Acquah (2010) presents a method for comparing asymmetric price transmission models and selecting the best model using the AIC and BIC in a Monte Carlo simulation. Acquah (2010) did not consider the use of bootstrap methods to analyse the relative performance of AIC and BIC. However, the issue of understanding the relative performance of AIC and BIC in selecting an asymmetric price relationship in a bootstrap simulation has not yet been investigated. A fundamental question previous studies have not addressed is how well AIC and BIC will perform when bootstrap samples are used in the asymmetric price transmission analysis. In the presence of bootstrap samples, will AIC and BIC point to the true data generating process as observed in previous Monte Carlo studies? Furthermore, unlike the Monte Carlo studies, the bootstrap simulations are based on minimal assumptions and provide more robust results.

This study therefore aims empirically at testing and comparing the ability of AIC and BIC in selecting the true asymmetric price relationship using a bootstrap simulation procedure. A comparison of AIC and BIC will thus contribute to understanding information criteria modelling generally and their empirical performance in price transmission analysis. The true data generating process is known in all experiments and the bootstrap simulations are essential in deriving the model recovery rates of the true model.

Material and Methods

The bootstrap

The term bootstrap was derived from the phrase “to pull oneself up by one’s bootstraps”. The phrase is thought to originate from one of the eighteenth century Adventures of Baron Munchausen by Rudolph Erich Raspe. The Baron had fallen to the bottom of a deep lake. Just when it looked like all was lost, he thought to pick himself up by his own bootstraps. The basic idea of the bootstrap as noted in Efron and Tibshirani (1993) involves repeated random sampling with replacement from the original data, to produce random samples of the same size of the original sample, each of which is referred to as the bootstrap sample. Each sample can be used to compute an estimate of the parameter of interest. ‘With replacement’ means that any observation can be sampled more than once in each bootstrap sample. It is important because sampling without replacement would simply give a random permutation of the original data, with many statistics such as the mean

(3)

being exactly the same. Repeating the process a larger number of times provides the required information on the variability of the estimator, since the standard error is estimated from the standard deviation of the statistics derived from the bootstrap samples.

Parametric Bootstrap

For model based resampling, the conventional fitted values and residuals are first obtained from the observed data. A bootstrap sample of the residuals is then drawn as outlined in step 1. These residuals are then added to the original regression equation (and x values) to generate new bootstrap values for the outcome variable as in step 2. Ordinary least squares are then used to estimate the new bootstrap regression coefficients for this bootstrap sample in step 3.

1. Generate by sampling with replacement from ̂ , … . , ̂ 2. Form

3. Compute from ( , ).

Repeat steps 1 to 3 (resampling of the residuals, adding them to the fitted values and estimating the regression coefficients) lots of times to estimate parameters of interest with the bootstrap samples. This model based on re-sampling is referred to as “parametric bootstrap” where residuals from a parametric model are bootstrapped to give estimates of interest.

Asymmetries and Equilibrium Relationship

The Granger and Lee (1989) Error Correction Model data generating process can be specified as follows:

∆ ∆ _, _,_~ _, ₍₁₎ where y and x are price series of a marketing chain. If y and x are non- stationary series that are cointegrated then there exists an equilibrium relationship between y and x which is defined by an error correction term. The long run dynamics captured by the error correction term are implicitly symmetric. In order to allow for asymmetric adjustments, the error correction term can be partitioned as follows:

, 0

(2)

, 0

(4)

The resulting asymmetric model is defined as

∆ ∆ _, _, ~ 0, (4)

This specification is referred to as the Granger and Lee asymmetric model. Asymmetry is incorporated by allowing the speed of adjustment to differ for the positive and negative components of the Error Correction Term (ECT) since the long run relationship captured by the ECT was implicitly symmetric. Symmetry in equation (4) is tested by determining whether the coefficients ( +

2

β and_β₂−_{) are}

identical (that isH₀:β = β2+ ₂−).

An alternative but a more complex approach to test for price asymmetry is proposed by Von Cramon-Taubadel and Loy (1996). In this approach, asymmetries specified affects the price increases and decreases as well as adjustments to the equilibrium level.

Where

∆

x

_t+ and

x

t

−

∆

are the positive and negative changes in

x

_t and the remaining variables are defined as in equation (5).

2 1 1 2 1 2 1 4, 4,

y

t

β

x

t

β

x

t

β

(

y

x

)

t

β

(

y

x

)

t

ε ε

t t

~

N

(0,

σ

)

+ + − − + + − − − −

∆ =

∆ +

−

+

−

+

(5)

A formal test of the asymmetry hypothesis using the above equation is:

0 : 1

H β₁+ = β − and β = β₂+ ₂−. In this case, a joint F-test can be used to determine symmetry or asymmetry of the price transmission process.

In contrast to Von Cramon-Taubadel and Loy (1996) model specification, Houck (1977) proposed a model in which asymmetries affects price increases and decreases and does not affect adjustments to the equilibrium level. The Houck method can be specified as follows:

2

1 1 5, 5,

y

_t

β

+

x

_t+

β

−

x

_t−

ε ε

_t _t

~

N

(0,

σ

)

∆ =

∆ +

(6) The variables in the model are defined as in equation (5). Symmetry is tested by determining whether the coefficients (β₁+_and

1

β−_{) are identical (that is}

0: 1

H β = β₁+ −).

Model Selection

The fundamental principle of information-theoretic model selection is to select statistical models that simplify description of the data and the model. Specifically, information-theoretic methods emphasize minimizing the amount of information

(5)

required to express the data and the model. This leads to the selection of models that are parsimonious or efficient representation of observed data. Numerous theoretic criteria have been developed. Generally, information-theoretic measure has two components. The first term is equivalent to the negative log-likelihood of the data calculated at the maximum likelihood estimates of the parameters. The second term can be thought of as a penalty for model complexity; it differs between different information-theoretic fit criteria and usually uniquely defines a given criterion. As the aim of information-theoretic model selection is to select parsimonious models, models that minimize the criterion are selected.

Akaike’s Information Criterion (AIC)

A well known information-theoretic criteria is AIC, originally referred to as “an information criterion” (Akaike, 1973). Theoretically, AIC is derived from consideration of the Kullback-Liebler distance. The Kullback-Liebler distance is a function of the ratio of two distributions, and can be thought of as reflecting the efficiency with which one distribution is approximated by another (Barron and Cover, 1991). Specifically, AIC is an estimate of the relative expected Kullback-Liebler distance of a given model from the true model. It can be thought of as measuring the relative inefficiency of approximating the true model by the model of interest. It is defined as:

2 l o g ( ) 2

A I C = − L + p (7)

Where the first term is the negative maximum log-likelihood of the data given the model parameter estimates and the second term p is the number of parameters in the model. Models producing smaller values of AIC can thus be thought of as more efficiently approximating the true model, where the true model is unknown. AIC has entered widespread use, especially within the domain of asymmetric price transmission modeling.

Bayesian Information Criterion (BIC)

BIC is currently among the most commonly used information-theoretic criteria. BIC is usually explained in terms of Bayesian theory, especially as an estimate of the Bayes factor for the comparison of a model to the saturated model. (Schwartz, 1978; Raftery, 1996). BIC is defined as:

2 l o g ( ) l o g ( )

(6)

Where n is the sample size and p is the number of parameters in the model. Models producing smaller values of BIC can thus be thought of as more efficiently identifying the true model where the true model is assumed to be among the models being compared.

A Simulation Study

The aim of the bootstrap simulation study is to investigate the ability of the model selection methods to identify the true model. Drawing from previous studies (Holly et al., 2003) the value of β₁ is set to 0.5 and ( +, ₂−) ( 0.25, 0.75)

2

β β ∈ − − are

considered for the coefficients of the asymmetric error correction terms in the true model. The competing models are fitted to the bootstrap samples and their ability to recover the true model was determined. The recovery rates were derived using 1,000 Bootstrap samples. The data generation process is specified in equation (4) and the data is simulated from the standard error correction model as follows:

1 1 0.5 0.25( ) 0.75( ) t t t t t t t t y x y x +− y x −− ε ∆ = − − − − + (9) t

y and x are generated as I (1) non-stationary variables that are cointegrated. The _t

error correction terms ((yt xt) t₁,(yt xt) t₁)

+ −

− −

− − denote the positive and negative deviations from the equilibrium relationship between y and _t x . _t

In order to examine the effect of the increase in the difference of asymmetric adjustment parameters on model recovery, the study simulated data of sample size 150 with an error size of 1 from the standard asymmetric price transmission model specified in equation 4 and asymmetry values ( +, ₂−) ( 0.25, 0.50)

2

β β ∈ − − or

( 0.25, 0.75)− − are considered for the coefficients of the asymmetric error correction terms.

Results and Discussion

Model recovery rates of the different model selection criteria

The performance of AIC and BIC in recovering the true data generating process (DGP) is evaluated using parametric bootstrap techniques. The bootstrap samples are used to investigate the effect of sample size, noise levels and the level of asymmetry on model selection. The performance of the two model selection methods are compared in terms of their ability to recover the true data generating process (DGP) across various sample size conditions (that is Model Recovery Rates) as illustrated in Table 1. In the following discussion, the standard

(7)

asymmetric error correction model, the complex asymmetric error correction model and the Houck’s model are denoted by SECM, CECM and HKD respectively.

For each model selection method, the model recovery rate defines the percentages of bootstrap samples in which each competing model provides a better model fit than the other competing models. Thousand bootstrap samples are generated from the original data using random sampling with replacement. AIC and BIC performed reasonably well in identifying the true model, though their ability to recover the true asymmetric data generating process (DGP) increases with increase in bootstrap sample size. When the bootstrap sample size was small (upper part of Table 1), the model selection methods recovered at most 79.9%. In large bootstrap samples (lower part of Table 1), the model selection methods recovered at most 97.6%. AIC performs well in small bootstrap samples, but is inconsistent and its performance does not improve in large bootstrap samples. BIC on the other hand is consistent and performance of the Bayesian criteria improves as bootstrap sample size increases.

Table 1. Relative performance of the model selection methods across sample size.

Experiment criterion Model fitted

Methods CECM (%) HKD (%) SECM (DGP) (%)

AIC 18.9 5.0 76.1 BIC 6.9 13.2 79.9 AIC 20.0 0 80.0 BIC 4.0 0.1 95.9 AIC 19.0 0.0 81.0 BIC 2.4 0.0 97.6

Note: Recovery rates based on 1,000 bootstrap replications.

Noticeably, model selection performance improved as bootstrap sample sizes increased. Two obvious trends can be observed with regards to the recovery rates of the true model (DGP) in Table 1. First, recovery rates of the BIC strongly depended on bootstrap sample size. Second, though AIC performed well in the small samples, its performance did not strongly depend on bootstrap sample size. Previous studies on model selection (Ichikawa, 1998; Markon and Krueger, 2004) note that AIC performs relatively well in small samples, but is inconsistent and at larger sample sizes it continued to exhibit a slight tendency to select complex models. BIC, in contrast, appears to perform relatively poorly in small samples, but is consistent and improves in performance with larger sample size. For example, in large samples, BIC performs better by choosing the correct model for 97.6 % of the bootstrap samples, whereas AIC correctly chooses 81% of the bootstrap samples. The findings of the current study are consistent with the Monte Carlo Simulation experiment of Acquah (2010) which finds the AIC to be

1 σ= 50 n= 150 n= σ=1 1 σ = 500 n=

(8)

asymptotically inconsistent and the BIC to be consistent in larger samples. Fundamentally, these results are confirmed in the bootstrap simulation as presented in Table 1.

In order to simulate the effects of noise level on model selection, this study considers three error sizes (σ) ranging relatively from small to large and corresponding to 1.0, 2.0 and 3.0. Using 1,000 bootstrap simulations, data is generated from equation (9) with the different error sizes and a sample size of 150.

The data fitting abilities of alternative models are compared in relation to the true model as the error in the data generating process was increased systematically. The performance of the model selection algorithms analyzed deteriorates with an increasing amount of noise in the true asymmetric price transmission data generating process (SECM) as illustrated in Table 2. BIC outperforms AIC in recovering the true data generating process at lower noise levels (σ = 1 - 2) but at higher noise levels (σ = 3), AIC outperforms BIC.

Table 2. Relative performance of the selection methods across error size.

AIC 14.1 20.7 65.2 BIC 1.6 50.2 48.2 AIC 18.7 4.9 76.4 BIC 3.1 18.7 78.2 AIC 20 0.0 80.0 BIC 4.0 0.1 95.9

The study further investigated the extent to which sample size and stochastic variance concurrently influenced model selection performance in bootstrap samples. Bootstrap simulation results reveal that a small error and large bootstrap sample improve recovery of the true asymmetric data generating process and vice versa. With a small bootstrap sample of 50 and an error size of 2.0, the true data generating process was recovered at least 41.1% of the time by the model selection criteria as illustrated in Table 3.

On the other hand, with a relatively large bootstrap sample of 150 and error size of 0.5 at least 80.0% of the correct model was recovered across all the model selection methods as indicated in Table 3. The model recovery rates of the model selection methods are derived under combined conditions of a small bootstrap sample size of 50 and large error size of 2 (that is, Unstable conditions) and a relatively large bootstrap sample size of 150 and a small error size of 0.5 (that is, Stable conditions). Under stable conditions, model selection performance or recovery rates improve for the true model.

150 n= _σ₌₃ 150 n= σ=2 150 n= σ=1

(9)

Table 3. Effects of sample size and stochastic variance on model recovery.

AIC 11.9 34.1 54

BIC 2.7 56.2 41.1

AIC 20.0 0.0 80.0

BIC 4.0 0.0 96.0

Table 4 illustrates how the different model selection methods exhibit different relative performance in recovering the true model at different levels of asymmetry. An increase in the difference between the asymmetric adjustments parameters from 0.25 to 0.50 led to improvement in the model recovery rates of the true asymmetric data generating process by the model selection methods. Noticeably, recovery rates of the Bayesian criteria respond more strongly to increases in the difference between the asymmetric adjustments parameters for the true model.

In short, another factor which may influence model selection or the recovery of the true data generating process is the difference in asymmetric adjustment parameters as illustrated.

Table 4. Effects of the level of asymmetry on model recovery.

An important characteristic of the current study is that they generally echo previous theoretical and empirical work on the performance of model selection methods in other applications. The bootstrap simulation results establish that AIC and BIC do identify the true asymmetric data generating process in the presence of bootstrap samples. This is consistent with Acquah (2010) Monte Carlo experimentation results which indicated that BIC and AIC clearly identify the true data generating process in asymmetric price transmission modeling framework.

The recovery of the correct model using bootstrap samples declined with increases in noise levels. This finding suggests that the amount of noise in the asymmetric data generating process is influential for the purposes of model selection. Similarly, previous studies (Gheissari and Bab-Hadiashar, 2003; Yang,

2 0 .2 5 + − 2 β − β = AIC 19.8 0.2 80.0 BIC 3.8 2.2 94.0 2 0 .5 0 + − 2 β − β = AIC 20.0 0.00 81.0 BIC 4.0 0.1 95.9 2 σ= n=50 150 n= σ=0.5

(10)

2003) also find that the performance of AIC and BIC declines with increases in the amount of noise in the data generating model. Within the asymmetric price transmission modeling context, AIC outperforms BIC when there is a high amount of noise in the true model. This observation is consistent with Chen et al. (2007) who note the propensity of BIC to perform worse than AIC at high noise levels in a factorial analysis.

The results of the current study demonstrate the usefulness of parametric bootstrap methods in asymmetric price transmission model selection. Bootstrap simulation results suggested that AIC performs relatively well in small bootstrap samples, but is inconsistent and does not improve performance in large bootstrap samples. Alternatively, BIC appears to perform relatively poorly in small bootstrap samples, but is consistent and improves in performance with large bootstrap samples in the price transmission modeling framework. This is consistent with the Monte Carlo simulation results of Acquah (2010) which suggest that generally AIC should be preferred in smaller samples, whilst BIC should be preferred in larger samples in the price transmission modeling framework.

With regards to the level of asymmetry, the results indicated that the ability of the model selection methods to identify the true data generating process depends on the difference in asymmetric adjustments speeds. In a Monte Carlo simulation, Acquah (2010), observed that the difference in asymmetric adjustment parameters from 0.25 to 0.50 has a positive effect on the ability of the model selection methods to recover the true model. Bootstrap simulation results suggest that on the basis of the recovery rates of the true model, BIC should be preferred to AIC in applications in which the data has strong levels of asymmetry. Using bootstrap samples in the asymmetric price transmission modeling framework, this study sheds light on the performance of the model selection methods. Fundamentally, I demonstrate that in the presence of bootstrap samples, the BIC and AIC point to the correct model.

Conclusion

The model selection criteria examined clearly point to the true asymmetric model out of different competing models. Fundamentally, the results demonstrate the usefulness of combining the use of bootstrap methods with information theoretic selection criteria to identify the true asymmetric price transmission model. The bootstrap simulation results indicate that the sample sizes, level of asymmetry and noise levels, are important in the selection of the true asymmetric model. Larger bootstrap sample sizes or lower noise levels improve the ability of AIC and BIC to point to the correct asymmetric price transmission model. Under unstable conditions such as small bootstrap sample and large noise levels, AIC outperforms BIC. An increase in the difference between the asymmetric

(11)

adjustments parameters improves model recovery rates of the true asymmetric data generating process by the model selection methods. The bootstrap comparison provided contributes to knowledge and understanding of the relative performance of the Akaike’s Information Criteria and the Bayesian Information Criteria in selecting an asymmetric price transmission model in the presence of bootstrap samples. Future research will depart from parametric assumptions and investigate model selection in asymmetric price transmission context using non-parametric bootstrapping.

References

Acquah, H.D. (2010): Comparison of Akaike information criteria (AIC) and Bayesian information criteria (BIC) in selection of asymmetric price relationships. Journal of Development and Agricultural Economics 2(1):001-006.

Akaike, H. (1973): Information theory and an extension of the maximum likelihood principle. In: B.N. Petrov, F. Csaki (eds.) 2nd_{International Symposium on Information Theory. Akademiai} Kiado, Budapest, pp. 267-281.

Barron, A.R., Cover, M.T. (1991): Minimum complexity density estimation. IEEE Transactions on Information Theory 37(4):1034-1054.

Chen, L., Giannakouros, P., Yang, Y. (2007): Model combining in factorial data analysis. Journal of Statistical Planning and Inference 137(9):2920-2934.

Efron, B., Tibshirani, R. (1993): An introduction to the bootstrap, Chapman and Hall, London, U.K. Gheissari, N., Bab-Hadiashar, A. (2004): Effect of noise on model selection criteria in visual

applications. Proceedings of the 17th International Conference onPattern Recognition vol. 2, pp. 229-232.

Granger, C.W.J., Lee, T.H. (1989): Investigation of production, sales and inventory relationships using multicointegration and non-symmetric error correction models. Journal of Applied Economics 4:135-159.

Holly, S., Turner, P., Weeks, M., (2003): Asymmetric adjustment and bias in estimation of an equilibrium relationship from a co-integrating regression. Computational Economics 21:195-202.

Houck, J.P., (1977): An approach to specifying and estimating nonreversible functions. American Journal of Agricultural Economics 59:570-572.

Ichikawa, M. (1988): Empirical assessments of AIC procedure for model selection in factor analysis. Behaviormetrika 24:33-40.

Markon, K.E., Krueger, R.F. (2004): An empirical comparison of information - theoretic selection criteria for multivariate behavior genetic models. Behavior Genetics 34(6):593-609.

Raftery, A.E. (1996): Approximate Bayes factors and accounting for model uncertainty in generalized linear models. Biometrika 83:251-266.

Schwarz, G. (1978): Estimating the dimension of a model. Annals of Statistics 6:461-464.

Von Cramon-Taubadel, S., Loy, J.P. (1996): Price asymmetry in the international wheat market: Comment. Canadian Journal of Agricultural Economics 44:311-317.

Yang, Y. (2003): Regression with multiple candidate models: Selecting or mixing? Statistica Sinica 13:783-809.

Received: October 20, 2012 Accepted: March 11, 2013

(12)

BOOTSTRAP PRISTUP VREDNOVANJA PERFORMANSI AKAIKEOVOG

INFORMACIONOG KRITERIJUMA (AIC) I BAJESOVOG INFORMACIONOG KRITERIJUMA (BIC) U IZBORU ASIMETRIČNOG ODNOSA CENA

Henry de-Graft Acquah* Univerzitet u Kejp Koustu,

Odsek za agroekonomiju i savetodavstvo, Gana

Ovaj rad se bavi problemom izbora modela u modelima asimetričnog prenosa cena kombinovanjem upotrebe bootstrap metoda sa informacionim kriterijumima za teoretski izbor. Parametarska bootstrap tehnika je korišćena kako bi se izabrao najbolji model prema Akaikeovom informacionom kriterijumu (AIC) i Bajesovom informacionom kriterijumu (BIC). Rezultati bootstrap simulacije su ukazali da su učinci AIC i BIC uslovljeni veličinom podataka, nivoom asimetrije i šuma u modelu koji se koristi u aplikaciji. Ovaj rad dalje utvrđuje da je BIC dosledan i da nadmašuje AIC pri izboru ispravnog asimetričnog odnosa cena kada je veličina

bootstrap uzorka velika.

Ključne reči: izbor modela, Akaikeov informacioni kriterijum (AIC), Bajesov

informacioni kriterijum (BIC), asimetrija, bootstrap simulacija.

Primljeno: 20. oktobra 2012. Odobreno: 11. marta 2013.