Simulations 2 & 3 – Positive and Negative

The second and third set of simulations compare the performance of the four models under conditions of positive and negative dependence in both the endogenous sample selection and endogenous treatment cases. Data with positive dependence are generated using all three dependence models, while data with negative dependence are generated using the CMP specification. All four count models are used to produce estimates from each of the three data configurations with positive dependence, in order to determine which model performs best when it is not the true data generating distribution.

The NB is not considered in analyzing data with negative dependence since it collapses to the Poisson in this instance.

The ATE is selected to be a “moderate” effect size of approximately 25% of the mean, and the mean remains at 3. The dispersion is selected so that the conditional variance is approximately 7.3. We target this variance with all three data generating processes in order to keep the comparison data sets as similar as possible. Results are presented in Tables 5-10.

Table 5

CMP Generated Data with Endogenous Treatment and Positive Dependence (ν = 0.281)¹²

βp =

NB Generated Data with Endogenous Treatment and Positive Dependence (ν = 25.00)

βp =

One takeaway from all of the simulations with positive dependence is that in the case of endogenous treatment the standard Poisson with heterogeneity performs poorly when there is additional dependence. The Poisson has the worst bias in estimating

12 Estimates of CMP coefficients produced by the NB, RGP, and Poisson models are multiplied by 0.281 post-estimation in order to rescale them for comparison with the CMP. This cannot be done in reverse since the “true” dispersion parameter from the CMP is unknown if the data are not generated by a CMP process.

treatment effects, even predicting a large effect in the wrong direction in the case of the CMP. In all three cases the Poisson has the worst fit statistics, and the LR test correctly rejects it in favor of any of the three dependence models. The one advantage the Poisson possesses is that it has the lowest variance of the estimates. However, this cannot

overcome its primary weakness of large and potentially catastrophic bias.

Table 7

RGP Generated Data with Endogenous Treatment and Positive Dependence (ν = .025)

βp =

CMP Generated Data with Endogenous Sample Selection and Positive Dependence (ν = 0.281)

Table 9

NB Generated Data with Endogenous Sample Selection and Positive Dependence (ν = 25.00)

RGP Generated Data with Endogenous Sample Selection and Positive Dependence (ν = 0.025)

The CMP performs well in all three cases. The CMP has the best fit statistics when it is the correctly specified distribution, and also when the data follow an NB distribution. In the case of RGP data, the CMP has fit statistics comparable to the true RGP model. The CMP also has a lower variance of the estimator than the RGP and NB models, resulting in a smaller MSE even when the data follow a NB or RGP

specification. Although the CMP has somewhat high bias with NB distributed data, the bias of the NB model is even worse with CMP data. The CMP also performs better on RGP data than the RGP performs with CMP data, although when both models are pitted against NB data, the RGP has the lowest bias of the two, landing in a virtual tie with the true NB model. As in the case of simulations 1-4, the CMP does a relatively poor job fitting the coefficients compared to the NB and RGP models, although the Poisson fares the worst in this regard.

Although the CMP is arguably the superior model in the case of endogenous treatment, this advantage does not hold in the case of sample selection. The CMP performs best across the board when it is the true data generating process, but the bias of the treatment effect estimates, and the mean squared error are the worst of all four models when the data are NB or RGP distributed. Moreover, the CMP does substantially worse fitting the data than any of the other three models. Contrary to the endogenous treatment case, the NB and RGP models perform fairly well (and nearly identically) when they do not reflect the true data generating process.

The Poisson performs reasonably well in the CMP case, but is still the worst by every metric, indicating that it may have trouble handling dependence when it re-enters the conditional mean as it does in CMP distributed data. The Poisson fares better with RGP and NB data, actually having the least bias and smallest MSE in the NB case, and fitting the data better than even the CMP in both cases. Overall, the NB model edges the RGP slightly in fit and bias over the three models, although the two models are virtually interchangeable.

In case of negative dependence the mean remains at 3 and the ATE remains at 25% of the mean. However, the presence of the heterogeneity presents a problem, since it overdisperses the data relative to a basic specification. In the case of the CMP, since the dependence parameter enters the conditional mean (including the heterogeneity term), it takes an extreme level of negative dependence (approaching a binary outcome) to make the conditional variance less than the conditional mean. Rather, the model presented for simulation represents a case where individual heterogeneity increases the variance of the model, but negative dependence decreases the variance, leaving the model with net overdispersion.

One interesting finding of the present study is the failure of the RGP model in the case of negative dependence with overdispersion induced by individual heterogeneity.

The restrictions on the RGP force the dispersion parameter ν to be such that ν >

min(-1/max(λ),-1/max(Y)) when the dependence is negative. This is typically not a problem, but the mix of net overdispersion with negative dependence results in such large values of Y and/or λ that ν is constrained to be virtually 0. Although in theory this should collapse the model to the Poisson, it rather prevents the model from achieving concavity and it is unable to converge consistently in this case. As such, we do not consider the RGP for comparison with the CMP or Poisson models in the case of negative

dependence.

In the ET case, the Poisson performed fairly poorly in terms of bias of the ATE estimate and the MSE. As expected the CMP performed much better, but still had a fairly large bias and MSE in comparison to data with positive dependence. Both models

performed better in the sample selection case, and the Poisson was actually more accurate in estimating the coefficient values. However, the CMP still fit the data better, rejecting the null hypothesis of a Poisson specification, in addition to having both lower bias and MSE of the ATE. (See Tables 11 and 12.)

Table 11

CMP Generated Data with Endogenous Treatment and Negative Dependence (ν = 4.50)

βp =

CMP Generated Data with Endogenous Sample Selection and Negative Dependence (ν = 4.50)

The results of the simulations did not overwhelmingly favor one model over the others. All four models performed comparably well at estimating both “large” and

“small” treatment effects when the data follow a Poisson distribution. This result is not surprising given that all three of the dispersion models nest the standard Poisson.