Study of Risk Aversion”
1.2.2 Variable Selection
As was shown in the last section, HLT’s enormous treatment effect for Ungraded Coins as Final Outcomes is the result of experimental data in which there is no behavioral response to changes in stimuli and a stochastic choice model that, in the absence of a possibility to attribute it to choice errors, takes this absence to be evidence of extreme risk aversion. Results for the other treatment effects, while identified by data from treatments in which subjects do, however noisily, respond to incentives, are dependent on model specification and, in particular, on the set of controls for individual characteristics included in the model.
Figure 1.5 shows point estimates and 95% confidence intervals for the four treat- ment effects and the effects of three of the seven individual characteristics included in HLT’s original specification —–Female, Coin dealer and Single and never mar- ried. It does so for specifications that contain the four treatment indicators and any possible combination of the three individual characteristics, for a total of seven specifications. For comparison, the Figure also shows estimates from HLT’s original specification.
Note that in what one might consider a minimal specification, one that contains only the four treatment indicators, all estimated effects are small and none are statistically significantly different from zero. In fact, for this model an omnibus F-test cannot reject the null (χ2(4) = 2.79, p = 0.59).
Only after the addition of controls for demographic characteristics does this change. It is, of course, entirely correct to control for individual characteristics. This not being a linear model, the estimated treatment effects would otherwise suffer from omitted variable bias despite being orthogonal by design. Which con- trols to include, however, is not an innocuous choice. Even within the small subset
9In the money treatment subjects were told the monetary value of the four prizes. In the graded
coins treatment subjects were presented with four coins as prizes, all of which had an attached certificate attesting to the coin’s condition. In the ungraded coins treatment the same four coins were used. The certificates, however, had been removed by the experimenters. The coins would therefore have looked identical unless examined.
Constan t Coins As Final Outcomes Ungraded Coins As Final Outcomes Frame to sk ew RA lo w er Frame to sk ew RA higher Female Single and nev er married Coin dealer 0.10 -0.04 0.10 0.20 0.12 0.21 1.05 0.95 ** 0.19 0.09 0.19 0.21 0.19 0.22 0.24 -0.16 ** ** ** 0.85 2.02 0.79 0.99 1.21 0.96 11.42 3.97 ** ** -0.15 -0.20 -0.15 0.07 0.04 0.09 0.90 0.76 ** -0.10 -0.19 -0.11 0.07 -0.00 0.08 0.17 0.14 -0.20 -0.13 -0.75 -1.26 ** -0.03 0.02 0.83 0.84 ** ** -0.38 -0.32 -0.39 -1.23 -0.98 full model (HLT estimates) treatments + female + single + dealer treatments + single + dealer treatments + female + dealer treatments + dealer treatments + single treatments + female treatments only -2 0 2 4 -2 0 2 4 -2 0 2 4 -2 0 2 4 -2 0 2 4 -2 0 2 4 -2 0 2 4 -2 0 2 4 Estimate
Confidence intervals based on clustered standard errors. Noise: Constant not shown for all models. College education or higher, Ever owned Morgan Silver dollars, Dealer × coins, Affiliated with a grading company, Noise: Constant and Noise: Female not shown for full model. “treatments + female + single” now shown because the specification shows the pathology described in Section 1.2.2
**: p < 0.05
Figure 1.5: Point estimates and 95% confidence intervals for models involving treatment indicators and the three individual characteristics Female, Single and never married and Coin dealer
of possible specifications shown in Figure 1.5, the statistical significance, magni- tude and even sign of all coefficient except for the one on Ungraded Coins as Final Outcomes is dependent on the specification.
Including all three individual characteristics in the model ––– call this the re- duced specification — yields estimates that are broadly similar to those in HLT’s original specification and a log-likelihood that is only 4.10 points lower10. Esti-
10It is unclear how these two models ought to be formally compared. A Wald test for a joint
restriction on HLT’s original specification at HLT’s parameter vector cannot reject the null (χ2(5) = 2.31, p = 0.80). However, this test assumes that HLT’s parameter vector is likelihood
mated parameters for the individual characteristics that remain in the model are very similar to those in the original specification. The picture for the treatment effects, however, looks very different. Relative to the original estimates, the treat- ment effect of Coins as Final Outcomes reverses sign and becomes statistically significantly different from zero (point estimate: 0.24, p = 0.02). Under this speci- fication, in other words, using “natural” instead of artificial prizes has a statistically significant influence on elicited risk preferences. The estimated treatment effect of Ungraded Coins as Final Outcomes is qualitatively similar to that in the orig- inal specification, that is, it continues to suffer from the issue detailed in the last section. Lastly, in HLT’s original specification the cross-treatment in which prob- abilities in the multiple price lists are skewed towards values that would produce lower measured risk aversion if subjects had a tendency to switch in the middle of the list is estimated toraise the CRRA coefficient by 0.756 but is not statistically significantly different from zero at the 5% level. In the reduced specification, the treatment effect is estimated to be 0.90 and is now highly statistically significant (p = 0.002).
What about specifications with other sets of demographic controls? It is at this point that one must confront the unfortunate reality that the model is not identified for most of the specifications that contain more than three individual characteris- tics. Table 1.A2 in the Appendix shows the results of numerical maximization for specifications involving all possible combinations of the seven individual character- istics included in HLT’s original specification.11 For many of the specifications the
model shows groups of parameters moving towards infinity in opposite directions. This phenomenon is similar to the (quasi-)complete separation which is some- times encountered in standard binary choice models when a combination of vari- ables allows the model to predict the dependent variable perfectly for a subset of observations. In such cases the likelihood does not have a maximum because predicted choice probabilities under the model can be driven ever closer to zero
11HLT’s original specification does not use all variables available in the dataset. The dataset
also contains a number of other variables: An experimenter effect, information about the years of experience in the coin and paper money market, the number of shows attended and the number of coins graded in a year, age, more fine-grained educational attainment, income level, the marital status, size of the household, information about whether and how much a subject smokes, which day of the 3-day show the experiment was conducted on, and whether a participant only deals in graded or ungraded coins.
or one by choosing more extreme parameter values. Similarly, adding individual characteristics to HLT’s model sometimes gives it enough flexibility to drive the coefficient of risk aversion of a group of subjects who are identical on the included variables either towards positive infinity, which drives choice probabilities under the model towards 0.5, or towards negative infinity, which drives choice proba- bilities for the riskier option B to one. In both case no likelihood maximum can exist because choices can always be moved closer to 0.5 by increasing the CRRA coefficient or closer to 1 by decreasing it.
HLT’s data contain observations of subjects who choose the risky option B throughout and subjects who choose very unsystematically. For the former group the model can gain likelihood by assigning to the group a negative CRRA coef- ficient while for the latter group it can gain likelihood by assigning the group a very large, positive CRRA coefficient. As long as the specification is relatively sparse each cell in the partition is unlikely to contain only subjects who make such “extreme” choices and estimated coefficients and CRRA parameters will be “reasonable”. Adding controls for individual characteristics, however, allows the models to partition subjects ever more finely and on HLT’s data the partition soon becomes fine enough to isolate extreme subjects. Once this happens, the model can drive the coefficient of one of the individual characteristics shared by these subjects towards infinity in either direction. The coefficients on some of the other characteristics, meanwhile, move in the opposite direction so the CRRA coefficient of subjects who share some but not all the characteristic does not also move.
On HLT’s data such estimates are common and the model is easily pushed to- wards them. Starting from the minimal specification that contains only treatment indicators and in which the choice error does not differ by gender adding just one demographic variable can already produce extreme results. By the time the model contains four individual characteristics only 5 of the 35 specification still have well-defined likelihood maxima. By five individual characteristics, these results are universal (for details, see Appendix Section 1.A4).
The same problem ails HLT’s original specification: HLT use a Newton-Raphson algorithm to find the estimates reported in the paper, at a log-likelihood of −1486.285. All other maximization algorithms offered by Stata do not converge on a likeli-
hood maximum. For the BHHH algorithm, however, intermediate solution can- didates achieve log-likelihoods above −1460. Generalized simulated annealing, which explores the parameter space randomly and therefore does not rely on the problem being globally concave finds a solution that is better still (log-likelihood: −1438.880). This draws into question the original estimates and inference based upon them in their entirety.
Luckily, the reduced specification that contains only three of the individual characteristics does not suffer from this pathology. For it, the likelihood maximum seems to be well-defined aside from the issue with the Ungraded Coins as Final Outcomes coefficient discussed in Section 1.2.1.12
All in all, the positive treatment effect of using Coins as Final Outcomes in the reduced specification is not anomalous. In fact, it’s HLT’s estimate of a negative treatment effect that is. It is positive and of roughly equal magnitude for all spec- ifications shown in Figure 1.5 and of the 64 specifications in which estimates do not show signs of the pathology described above, none feature a negative point estimate for Coins as Final Outcomes. Its statistical significance, however, does depend on which set of individual characteristics the model contains. The same is true for the effect of the treatment designed to lower measured risk aversion by skewing the multiple price list. This effect was already sizable in HLT’s specifica- tion but not statistically different from zero. In the reduced specification the effect is similarly large and statistically significant, but this is the only one of the models shown in Figure 1.5 for which either are true. In most the effect is of variable sign, small and not statistically significant.