Sensitivity Analysis

Because of the uncertainties in the numerical specification of a model, modellers typically test the robustness of their results using some form of sensitivity analysis. The overwhelming majority of these analyses focus on the effects of the choice of model elasticities. The most common approach to sensitivity analysis is termed 'limited sensitivity analysis' by Wigle (1991). In limited sensitivity analysis, the modeller subjectively identifies the important model elasticities, so that a trade modeller might, for example, list import demand elasticities as the subjects of sensitivity analysis. The central values of these key parameters are then perturbed by a 'reasonable' amount, the model is solved, and the results are reported for the alternative elasticity configurations. The process is repeated for several values o f the key elasticities. While this procedure can give some sense of whether model results are fragile, it provides no meaningful quantitative measure of robustness.

More rigorous statistical sensitivity analysis procedures have also been developed. Wigle (1991) discusses two classes of systematic elasticity sensitivity analysis used in reporting applied general equilibrium model results, both of which require the modeller to assign probabilities to alternative elasticity configurations. Conditional systematic sensitivity analysis (CSSA) infers the distribution of the model results by computing a series of solutions as each elasticity is varied while the others

remain constant. Unconditional systematic sensitivity analysis (USSA) computes model results over the entire grid of possible elasticity configurations. USSA is the most thorough and therefore, the more preferable response to criticisms o f elasticity specification, but for most models the computational requirements of such a procedure are prohibitive.10

Pagan and Shannon (1985) develop an approximation method for performing unlimited systematic sensitivity analysis. Instead of solving the model for each point in the elasticity space explicitly, their procedure analyzes the effects of altering elasticity parameters in a region surrounding the model solution. Because their sensitivity procedure relies on calculations made using a linear approximation of the model solution, which is a function of the elasticity parameters, the computational requirements are considerably less than in unconditional systematic sensitivity analysis, while the procedure retains the flexibility to examine the effects of simultaneous elasticity variations. The Pagan-Shannon approximation procedure is applied in Pagan and Shannon (1985, 1987) and Wigle (1991).

Other sensitivity procedures, in which modellers map a priori information about

elasticity probabilities into the model results, have also been developed. Harrison and

Vinod (1992) and Harrison et al. (1992), develop and apply a global sensitivity analysis

procedure in which the model is solved for a sample of elasticity configurations. Their procedure relies on sampling from discrete representations of what are usually continuous elasticity probability density functions. DeVuyst and Preckel (1997) argue

10 Wigle (1991) calculates that a USSA using 5 values for each elasticity in an 18 elasticity parameter model would require more than 3 trillion model solutions.

that the methodology of Harrison and Vinod introduces an identifiable source of bias into the sampling procedure and propose an alternative way of finding discrete approximations to the continuous probability density functions, based on Gaussian quadrature. In both approaches, the model results are weighted by the probability of each elasticity configuration used in their derivation. Repeated sampling allows the modellers to build expected values and confidence intervals for the model results.

Sensitivity analysis completes the modelling process. It represents a way for modellers to address some of the weaknesses inherent in a methodology that requires subjective judgement at many junctures, and which employs estimates for a large number o f diverse data points.

2.S The Econometric Critique of Calibration

The calibration methodology used for applied general equilibrium models has been criticized in Jorgenson (1984) and more recently by McKitrick (1995, 1998) on several grounds. They argue that the data pre-adjustments in the process of implementing calibration introduce untraceablc bias into the data and hence, into the parameters and ultimately the model results. The use of a benchmark year for calibration also enters their critique since any anomalies in the economy for that year can be transmitted to the calibrated parameter values, and hence, to the model results. They highlight the inadequacies of the elasticity estimates in applied models, and argue that the reliance on CES and Cobb-Douglas functional forms is restrictive and unrealistic. This restrictive class of functional forms precludes complementarities, and incorporates

elasticities of substitution which are independent of prices and which thus, unrealistically constrain behavioural responses in counterfactual simulations. McKitrick's (1998) illustration that a model's functional structure has a large effect on results highlights the shortcomings of relying on these functional forms.

Jorgenson and McKitrick's proposed alternative is the simultaneous estimation of all of a model's elasticities and share parameters using time series data. This approach allows elasticity estimation which is fully consistent with the definitions of variables employed in the model, and does not require the use o f restrictive functional forms. The statistical basis of estimation isolates systematic effects from random noise, and the use of unadjusted time series data precludes the introduction of pre-adjustment bias.

Explicit econometric approaches to applied general equilibrium modelling have thus far been limited to a handful of papers: Clements (1980), Jorgenson (1984), Jorgenson, Slesnick and Wilcoxen (1992), McKitrick (1995), and McKitrick (1998). Most of these econometric general equilibrium models, however, estimate model subsystems rather than incorporating the full set of cross-equation equilibrium restrictions.

If estimation is superior to calibration in so many ways, why it has not been more widely adopted? One issue is the difficulty in imposing the equilibrium solution concept, which is central to general equilibrium analysis, as a series of cross-equation restrictions in estimation. Another is the paucity of time series data on the variables of interest for the questions that are addressed in calibrated models. The estimation of large dimensional models, or models which focus on variables that are not measured in

national accounts data, may be intractable. The effort required to generate the single observation required for calibration can itself be formidable, and extending the process to include time series observations may be close to impossible. For example, modellers must frequently update an earlier year's input-output matrix as an approximation to that of the benchmark year, because most countries do not produce annual input-output tables."

The econometric approach also precludes the use of some simplifying techniques commonly employed in applied general equilibrium models. One such technique is the Harberger (1962) convention, whereby the units of quantities defined in the model are given by that quantity which sells for one unit o f currency in the base period. This convention allows the modeller the simplification of representing heterogenous quantities in a homogenous manner, both in data and in the model. For example, if labour inputs were to be measured as hours worked, some correction would have to be made for different levels of labour efficiency and skill. The use of this assumption also reduces the number of variables required in the model; the modeller need only collect data in value terms, rather than in separate price and quantity terms. Such a convention, however, creates time-dependent units that make the interpretation of the results of counterfactual policy simulations a somewhat delicate issue. How, for example, should the modeller interpret a 10% increase in the price of a non-electrical

machinery aggregate in the counterfactual equilibrium? How should labour of different 11

11 Furthermore, where they are produced, they are often generated by updating a previous year's table rather than by undertaking new production surveys.

skills be aggregated, when compared to a cost o f one unit in the benchmark equilibrium? Such a convention makes time series estimation virtually impossible.

Faced with the weaknesses presented in the econometric critique, why do policy modellers persist with their work? The answer lies in the lack of practical alternatives. Policies will be decided with or without numerical input. Modellers' underlying belief is that imperfect analysis is better than no analysis. To contribute to debate on the social issues of the day a modeller must make the best use o f the available information, rather than refraining from any analysis until every parameter is definitively tied down. Modelling is a way of harnessing available information to contribute to policy making by raising the level of debate - an argument which would clearly be rejected by those whose advocate an exclusively positivist approach to research in the social sciences and to producing policy recommendations.