The Criteria for Adjustment Algorithm Evaluation

This chapter argues that the choice of adjustment algorithm should be based on its effect on the statistical properties of the model results, rather than on criteria relating to the adjusted data matrix.28 Model results are the heart of the modelling exercise: they are often used to inform the policy process, and erroneous model results can have potentially serious repercussions in the real economy. Thus, modellers are concerned with minimizing the likelihood of such errors, and the choices made in the modelling process, including the choice of adjustment algorithm, are undertaken to meet this objective.

Underlying the wish of modellers to minimize the error in their model results is the idea that some 'true' model result exists, hut its value is unknown and unobservable. Instead, the modeller estimates the value of the model results from a single unbalanced raw data set and a set of elasticities as inputs. This process is often

28 Others (see, for example, Schneider and Zenios 1990) also consider practical criteria such as the ease of implementation, and the computer time required to balance a matrix. As computing technology improves, these practical considerations are likely to diminish in

Chapter 4: The Adjustment Algorithm Choice

considered to be deterministic since applied general equilibrium models admit no explicit stochastic structure. If the adjustment algorithm, the elasticity specification, the model structure and the policy experiment are all held constant, a given set of raw data will yield a unique model solution (assuming an absence of multiple equilibria) and a unique set of model results. Because only one raw data set is ever balanced and then used to calibrate the model, no statistical considerations arise.

The problem with this view is that it ignores the fact that the data inputs are random variables which are subject to measurement errors. The unbalanced data set used by the modeller is generated by a random realization from the probability distribution for each random element. If the modeller were to undertake the production and consumption surveys again so that the values for the raw data would be measured a second time, the second raw data set would very likely be different from the first - it would be generated by a second random realization from each variable's probability density function. In principle, the modeller could generate a large sample of random unbalanced raw data sets. If each of these data samples were to be balanced with a specific adjustment algorithm and then used to calibrate the model, each would lead to a set of model results so that the sample of input data would lead to a sample of model results. Because the model results are generated using random variables as inputs, they too are random variables.

The process o f deriving estimates of the model results from the sample of unbalanced data sets is one of applying a mapping from one set of random variables, the unbalanced data, into another set, the model results. In this case the mapping rule is complex. It consists o f the data adjustment procedure, the model, and the policy

Chapter 4: The Adjustment Algorithm Choice

experiment. If the model and policy experiment components of this rule are held constant, however, the choice of an adjustment algorithm determines the statistical properties of the distribution of the model results derived from a given sample of unadjusted data. The distributions of the model results derived using various adjustment algorithms will each have means that are estimators of the true model results, and variances that capture the dispersion in those model results. An adjustment algorithm which performs well will yield a distribution for the model results that has an unbiased mean and a low dispersion.

If R denotes the true model result and Rp is the model result from using adjustment algorithm p, then the bias of Rp, denoted by Rp, is given by the difference between the expected value of the model result and the true model result.

Bp = £(Rp) - R. (4.1)

If two adjustment algorithms yield unbiased, or low-bias estimates of the mean value for the model results, the one which yields a lower dispersion in the distribution of the model results will be preferred. The mean square error of the model results derived using adjustment algorithm p, MSE(Rp), accommodates a tradeoff between dispersion and bias and is given by the expectation of the square of the difference between the estimate o f the model result and the true model result,

Chapter 4: The Adjustment Algorithm Choice

Thus, a preferred adjustment algorithm yields distributions for the model results that have unbiased mean values and low mean square errors.

The question which then arises is whether the adjustment algorithms that are commonly used by applied general equilibrium modellers differ significantly in their effect on the statistical properties of the model results. If they do not, then the choice of algorithm is peripheral to the modelling process, but if they do, then modellers have an incentive to exercise care in their choice of adjustment algorithm. The experiments in Section 4.4 of this chapter show that adjustment algorithms can differ substantially in their performance. Given a choice of adjustment algorithms and a desire to minimize the likelihood of error in the model results, the modeller should thus choose the adjustment algorithm which minimizes the bias and/or the mean square error in the distributions of the model results. Even though the modeller uses only one unadjusted data set, the probability of minimizing the error in the model results from that single data set is smaller using an adjustment algorithm that performs well than one that performs poorly.

4.2.1 Evaluation Criteria in Previous Work

The criteria for choosing an adjustment algorithm here contrast with previous work in the social accounting and input-output literature, where the focus has been the effect of the adjustment algorithm on the properties of the balanced matrix, instead o f the model results (see, for example, Khan, 1993; Schneider and Zenios, 1990; Gunliik-§enesen and Bates, 1988; Parikh, 1979; Lynch, 1976; Maliziaand Bond, 1974). Modellers are, of

Chapter 4: The Adjustment Algorithm Choice

course, more concerned with how the choice of algorithm affects the model results than with its implications for the adjusted matrix per se. Furthermore, this work in the social accounting literature is of limited value for applied general equilibrium modellers because the statistical properties of the adjusted matrix arising from the use of a particular adjustment algorithm do not necessarily transmit to the model results derived from that adjusted data - an adjustment algorithm that yields an unbiased adjusted data matrix may not yield unbiased estimates for the model results.

This lack of equivalence exists because the adjusted data are mapped through the model into the model results. It is evident from the meaning of statistical bias. Consider an algorithm, p, that yields an adjusted, unbiased matrix, Ap. If A is the true matrix o f balanced data, then by definition, £(AP) = A. Under the expectation operator, this unbiasedness does not imply that the expectation of a function of A* will be unbiased. Specifically, it does not imply that for all functions F , E (F(AP)) = F((A)), particularly when F represents the equations of a highly non-linear general equilibrium model.

Social accountants have been forced to adopt model-neutral evaluation criteria because input-output tables and social accounting matrices have many potential applications, and are not associated with a specific set of model results. In contrast, the adjusted data matrix in an applied general equilibrium model, the BED, is constructed as an input to a specific model so that a mapping exists between an adjustment algorithm and the final model results. This link allows the effects of the adjustment algorithm on the statistical properties of the model results to form the basis of the algorithm choice in applied general equilibrium modelling.

Chapter 4: The Adjustment Algorithm Choice