4.5 Results for the General Linear Model
4.5.4 Predictive Simulation
We have already reported that the average total effect estimator obtained from the predictive simulation ap-proach is unbiased (see paragraph 3.2.3 on page 61). The computed measures of efficiency for the estima-tion of the average total effect, i. e., the mean squared errors, were comparable for the predictive simulaestima-tion approach and the regression estimate approach as well.32
Type-I-Error Rate The rejection frequencies for tests of the hypothesis ATE = 0 based on predictive sim-ulations as suggested by Gelman and Hill (2007) are clearly higher than the nominal 5 % level for all studied conditions with serious interaction effects (see the distribution of the rejection frequencies over all con-ditions of simulation study I for equal group sizes in Figure 4.21). Obviously, this is not an effect of small sample sizes, because the asymmetric distribution of observed rejection frequencies does not even disap-pear for conditions with large sample sizes (see the right tails in the distributions for N = 400 and N = 1000 in Figure 4.21). Although the average total effect estimator is unbiased, the test statistic (the t-test as well as the normal approximation) produce acceptable rejection frequencies only for small interaction effects γ11≤ 1, and only for simulated conditions with equal group sizes.
29As Figure 4.19 also shows, the absolute bias is largest for conditions with the highest value of the dependency between X and Z (R2
X |Z= 0.75) and the smallest sample sizes (N = 100).
30See the additional Figure 32 on page 43 and Figure 33 on page 44 of the digital appendix.
31See the additional Figure 34 on page 45 of the digital appendix.
32See Figure 4.15 and Figure 4.16 above as well as the additional Figure 26 on page 37 of the digital appendix.
4.5 Results for the General Linear Model 142
Type-I-Error Rate for the Hypothesis ATE = 0
Distribution of Rejection Frequencies for Predictive Simulations Equal Group Size [P(X = 1) = 0.5], Grouped by Sample Size
0 5 10 15 20
0.00.20.40.6
N = 100
Rejection Rate
0 5 10 15 20
0.00.20.40.6
N = 250
Rejection Rate
0 5 10 15 20
0.00.20.40.6
N = 400
Rejection Rate
0 5 10 15 20
0.00.20.40.6
N = 1000
Rejection Rate Normal Approximation t−Test
Figure 4.21: Type-I-error rate: Distribution of the rejection frequencies for the predictive simulation approach (normal approximation vs. t–test), grouped by sample size N [P(X = 1) = 0.5]
A direct comparison of the empirical type-I-error rates obtained for tests of the hypothesis of no average total effect based on the (unadjusted) GLH / mean-centering approach to the empirical type-I-error rates observed for the test statistic based on the standard type-I-errors obtained by predictive simulation (t–test)33 is presented as level plot in Figure 4.22. Obviously, the simulation-based procedure produces only very small improvements of the empirical type-I-error rate. These benefits are likely to vanish when the power of the resulting test statistic (instead of the type-I-error rate) is considered.
Bias of the Standard Error of the ATE–Estimator The distributions of RB£S.E.( dd ATE10)¤
for the predic-tive simulation approach, approximated as histograms, are summarized in Figure 4.23, grouped by sample size and group size. For equal group sizes [P(X = 1) = 0.5] and for treatment groups smaller than control groups [P(X = 1) = 0.2], standard errors are strongly underestimated. For some conditions with a treatment probability of P(X = 1) = 0.8, the effect due to heterogeneity of between-group residual variances and the consequences of the inappropriately handled stochasticity of the covariates cancel each other out.34
33In contrast to the regression estimation approach discussed in the previous subsection, no meaningful difference between the t -test and the normal approximation can be observed. As the additional Figure 35 on page 46, Figure 36 on page 47 and Figure 37 on page 48 of the digital appendix demonstrate, the observed inflation of the empirical type-I-error rates are directly connected to the values of the interaction parameter γ11used for generating the data.
34This is obvious from a comparison of the results presented in the additional Figure 38 on page 49, Figure 39 on page 50 and Figure 40 on page 51 of the digital appendix.
4.5 Results for the General Linear Model 143
Type-I-Error Rate for the Hypothesis ATE = 0
Predictive Simulations (t–Test) vs. GLH / Mean-Centering Approach with Estimated Mean of the Covariate [R2X |Z= 0.75, N = 400 and γ01= 5]
GLH / Mean-Centering (Estimated Mean of the Covariate)
N=400 , RX|Z
Figure 4.22: Type-I-error rate: Level plots for the predictive simulation approach vs. the GLH / mean-centering approach (estimated mean of the covariate) [R2X |Z= 0.75, N = 400 and γ01= 5]
Summary The predictive simulation approach did not differ substantially from the GLH / mean-centering approach. Although Gelman and Hill (2007) suggest the simulation-based procedure for inference about nonlinear predictions, the unconditional variance of the ATE–estimator is underestimated for equal group
4.5 Results for the General Linear Model 144
Relative Bias RB£
S.E.( dd ATE10)¤
of the Standard Error of the ATE–Estimator Predictive Simulation, Grouped by Sample Size and Group Size
N=100 and P(X=1)=0.2
−0.4 −0.2 0.0 0.2 0.4
050100150 mean: −0.313
N=100 and P(X=1)=0.5
−0.4 −0.2 0.0 0.2 0.4
050100150 mean: −0.255
N=100 and P(X=1)=0.8
−0.4 −0.2 0.0 0.2 0.4
050100150 mean: 0.031
N=250 and P(X=1)=0.2
−0.4 −0.2 0.0 0.2 0.4
050100150 mean: −0.312
N=250 and P(X=1)=0.5
−0.4 −0.2 0.0 0.2 0.4
050100150 mean: −0.262
N=250 and P(X=1)=0.8
−0.4 −0.2 0.0 0.2 0.4
050100150 mean: 0.021
N=400 and P(X=1)=0.2
−0.4 −0.2 0.0 0.2 0.4
050100150 mean: −0.310
N=400 and P(X=1)=0.5
−0.4 −0.2 0.0 0.2 0.4
050100150 mean: −0.262
N=400 and P(X=1)=0.8
−0.4 −0.2 0.0 0.2 0.4
050100150 mean: 0.015
N=1000 and P(X=1)=0.2
−0.4 −0.2 0.0 0.2 0.4
050100150 mean: −0.310
N=1000 and P(X=1)=0.5
−0.4 −0.2 0.0 0.2 0.4
050100150 mean: −0.263
N=1000 and P(X=1)=0.8
−0.4 −0.2 0.0 0.2 0.4
050100150 mean: 0.015
Figure 4.23: Relative bias of the standard error of the ATE–estimator: Histograms for the predictive simulation approach, grouped by sample size N and group size P(X = 1)
sizes and unequal group sizes with P(X = 1) = 0.2 and biased for P(X = 1) = 0.8. Hence, the predictive simulation approach will not be included in the second part of the simulation study.
4.5.5 Summary
In line with Flory (2004) and Nagengast (2006) we found inflated empirical type-I-error rates for tests of the hypothesis ATE = 0 obtained from the general linear hypothesis. Furthermore, we demonstrated that mean-centering of covariates with estimated means does not change the statistical properties of the ATE–
estimator. In contrast to previous simulation studies, we manipulated the individual (total) effect’s vari-ability as an additional factor in the simulation design. We were thus able to disentangle the consequences