Checks on normality
Visual inspection of histograms for the assignment and outcome variables showed
that both variables were relatively normally distributed and did not have any substantial
skewness or kurtosis (see graphs in APPENDIX A). Tabachnick and Fidell (2001)
recommend assessing the distributions visually because with very large samples such as
the one used here even small deviations from normality will lead to significant inference
tests on deviations in normality, skewness, and kurtosis. Both distributions had slightly
elongated positive tails, but these represented a very small number of cases. There was a
minor tendency for MATHSS_1 to be slightly heavier on the right side of the distribution
and for MATHSS_2 to be slightly heavier on the left side of the distribution, but overall
the two distributions appeared to be quite normal. No deviations appeared large enough
to merit concern.
Checks on linearity
A critical part of the analysis of RD involves appropriately modeling the
functional form of the relationship between the assignment and outcome variables,
because a misspecification of functional form can lead to spurious discontinuities. For the
purposes of this simulation it was important to determine that the form of the relationship
was indeed linear to ensure that no spurious discontinuities were inadvertently
In order to assess this, a scatterplot of the relationship between the assignment and
outcome variables was examined. Based on visual inspection the relationship appeared to
be linear with no obvious flexion points which would indicate an underlying curvilinear
relationship, though there is a slight fanning of scores at the upper end of the distribution.
This relationship can be seen in Figure 5 below.
Figure 5. Scatterplot of relationship between outcome and assignment variables
Linearity was further assessed by using a graph of the studentized residuals from
indicated that they were well distributed around zero and did not suggest any
curvilinearity in the relationship between these two variables (for a graph of the residuals
see APPENDIX A).
Estimates from the null model: The sharp RD
The null model forms the basis of comparison for the other models. The null
model is a random coefficient model where all cases were assigned to groups solely on
the basis of the assignment variable. This model contains no fuzziness – it is equivalent to
a perfect sharp RD with no misassignment. Because there is no actual effect of group
(program vs. control) in the base data, estimates from the null model should not be
statistically different from zero. As can be seen in Table 10 the average estimate for the
effect of group does not differ significantly from zero (Z=0.22, p=0.63) and has a
negligible effect size (d = 0.004).
Table 10. Results from the sharp RD
Coefficient SE t Value p Value d
INTERCEPT 487.60 0.47 1,046.52 <0.001 --
MATHSS_1 0.93 0.01 147.05 <0.001 --
Z (Program) 0.22 0.46 0.48 0.63 0.004
Effects of random misassignment
Before assessing the effects of biased misassignment a set of simulations were run
to confirm the parameter estimates from models in which misassignment of cases was
random with respect to the assignment variable and therefore is random with respect to
group membership as well. Estimates from the simulations with random misassignment
are represented graphically below; a full table of results can be found in APPENDIX B.
The results of the random misassignment were not as expected. There was a clear
curvilinear pattern to the results and several of the estimates showed significant bias (see
Figure 6).
Within the region 0.67 to 1.37 standard deviations from the mean fuzziness of
60% or greater tended to introduce significant amounts of bias. This relationship is not
found at the upper or lower bounds of the region of fuzziness or for lower percentages of
fuzziness, where the amount of bias is not significantly different from zero. The absolute
value of the largest deviation from zero was relatively small, reaching a maximum of
2.14 points with an effect size of 0.04. Nevertheless, given that there was a clear
curvilinear pattern and significant bias which theoretically should not have existed,
Figure 6. Random model: effects of fuzziness on program estimates (by distance from mean)
The reason for the significantly biased parameter estimates is not immediately
clear. The program effect in this dataset is known to be zero – the cutoff was arbitrarily
set and no program was associated with the cutoff. The random coefficient model based
0.22, p = 0.63), it was only when fuzziness was introduced that significant bias began to
occur. Given these findings, a number of possible reasons for the biased parameter
estimates were assessed.
The most obvious problem is that there could have been an issue with the
sampling method used which created bias on the outcome variable between the cases
sampled and the region they were sampled from, thus deviating from randomness and
leading to biased parameter estimates. This possibility was tested by comparing the
average mean of the samples to the mean of the region of fuzziness from which they were
drawn. The sample means were found to be very close approximations of both the
assignment and outcome variables in the region from which they were drawn; there was
no substantial evidence of departures from randomness in the samples themselves.
Given that the sampling method was confirmed to be random this suggests that
the bias is an artifact of the dataset. In order to test this, the same random misassignment
model was run on SIMDATA. The results clearly showed that no bias resulted from
random misassignment on this dataset – neither the region of fuzziness nor the percent of
fuzzy data were associated with any significant bias to the parameter estimates (see
Figure 7. Results of random misassignment on SIMDATA
This strongly suggested that there was something particular about the
characteristics of the real student data which was causing the biased estimates. Given that
the real student data appears to be the problem, one possible issue is that the model was
improperly specified. The literature on RD clearly notes that a model which does not
outcome variables can result in spurious discontinuities. Preliminary analyses on the
dataset had indicated no substantial departures from linearity and an examination of the
scatterplot and residuals had not suggested the presence of curvilinearity. However,
additional analyses were conducted in order to determine whether there was a more
subtle effect occurring which had not been apparent during the screening process.
The model used in the simulations only included a main effect of MATHSS_1
and program group; no interaction term was included because it was known that no
program had been implemented on these students and it was therefore theoretically
impossible for the program term (Z) to interact with MATHSS_1. However, for the
purposes of exploring the bias cased by random fuzziness a model was fit to the ‘sharp’
RD data which included the linear, quadratic, and cubic terms, and their interactions.
Shadish et al. (2002) suggest including terms at least two orders higher than any
suspected curvilinearity in the data and note that overfitting the model should not bias
estimates though it does lower power; with the large sample sized used here power was
not a concern. The results of the overfitted model are presented in Table 11.
Although the program effect term (Z) is considerably larger than in the previous
model it is still only marginally significant in this model (Z = -1.32, p = 0.08) and by the
common standard of p = 0.05 it would not be considered significantly biased. However, it
is much larger and closer to significance than the Z term from the model which contained
Table 11. Fully fitted model with curvilinear terms
Solution for Fixed Effects
Effect Coefficient SE DF t Value p Value
INTERCEPT 487.77 0.62 463 783.50 <0.001 MATHSS_1 0.90 0.03 26,357 30.18 <0.001 Z -1.32 0.75 26,357 -1.77 0.076 MATHSS_1*Z -0.26 0.06 26,357 -4.61 <0.001 QUADRATIC 0.0011 <0.01 26,357 2.54 0.011 CUBIC <-0.0001 <0.01 26,357 -5.93 <0.001 QUADRATIC*Z -0.0094 <0.01 26,357 -8.08 <0.001 CUBIC*Z <-0.0001 <0.01 26,357 -6.16 <0.001
Although all of the curvilinear terms and their interactions are significant the
coefficient terms are very small, with none exceeding an absolute value of 0.009.23 The
linear interaction term had a more substantial coefficient than the curvilinear terms and
was significant (MATHSS_1*Z = -0.25, p<0.001), indicating that the slope of the
assignment variable differed on the two sides of the cutoff. Analysis of model fit
indicated that including the curvilinear and interaction terms only explained
approximately an additional 1% of the variance at level-1 compared to a model which
23
Analysis of a separate model showed that the exclusion of the linear interaction term did not substantively change the estimates of the curvilinear terms.
included only MATHSS_1 and Z. So given that a model using only the linear main effect
term produces biased estimates, and the curvilinear terms are significant but small in
absolute terms, how do we determine whether it is important to include these in the
model?
One way to do this is to compare the outcomes of a model including the
curvilinear terms with a model that does not, and to capitalize on Trochim’s (1990)
concept of pattern-matching. When misassignment is totally random the predicted pattern
of program estimates is that they do not different significantly from zero regardless of the
percent of fuzzy data or region of fuzziness. Therefore, when a given model produces
unbiased parameter estimates with the expected pattern after random misassignment is
applied we can conclude that the model has produced the ‘appropriate’ pattern of results.
Using these results as our baseline we can compare the results of using a model with only
the main effect terms and a model which includes curvilinear terms to determine which
model produces the most ‘appropriate’ pattern. Comparison of results from the main
effect versus curvilinear model clearly shows that the curvilinear model does not provide
better estimates of the program effect parameter than the model with only MATHSS_1
and Z. When curvilinear terms are included the pattern deviates from that which would
be expected and shows an upward slope for all percent of fuzzy data values across the
regions of fuzziness. Both the estimates of absolute program effect and effect size are
nearly as large as those from the models which include only main effect terms (see Figure
Figure 8. Random model: effects of fuzziness on program estimates with curvilinear terms
Taken together, these results suggest several conclusions. First, the main effect
model appears to capture essentially all of the variance captured by a model with
curvilinear terms while considerably improving parsimony. Second, even though the
improves model fit very slightly, it makes no theoretical sense to add this term because
there was no actual program. The significance of the term suggests some underlying
oddities of the dataset, and adding the interaction term would therefore essentially be
modeling the specific characteristics of the data (thereby potentially reducing the
generalizability and validity of the findings), rather than adopting a theory-driven
approach to modeling. Third, modeling the curvilinear terms results in a fairly negligible
increase in model fit and results in a pattern of parameter estimates which does not match
what would have been predicted a priori given a theoretical understanding of the effects
of random misassignment. The results of the model with curvilinear terms, though
different from those obtained using the main effect model, do not appear to be
substantively better than those from the more parsimonious model. The biased parameter
effects resulting from random misassignment point to abnormalities in the base data
which are not well explained by the curvilinear terms. Overall, these findings suggest that
the main effect model is actually the most appropriate model to be used and it was
therefore retained for simulations assessing the effects of biased misassignment.
Effects of biased misassignment
This section will focus on the effects of biased misassignment which are based on
the outcome variable (MATHSS_2). The next section will discuss how different
misassignment variables affect these results. Because they are so extensive, the full tables
with the results for the program effect estimates can be found in APPENDIX C.
This analysis begins by considering the results at an aggregate level in order to
relates to how bias is impacted by the ratio of fuzzy to non-fuzzy cases within a region.
The next section then considers the effects of the region of fuzziness when the ratios of
fuzzy to non-fuzzy cases are the same. The third section explores in greater detail the
cumulative effect of the parameters, and the specific levels of bias which result from
different combinations of the parameters. A fourth section explores the question of how
changes in the size of the region affect bias when the absolute number of fuzzy cases is
held constant (as opposed to the ratio of fuzzy cases). The fifth section explores the
effects on bias when the variable which determines misassignment is less highly
correlated with the outcome (so that there is a smaller degree of selection bias). The final
section explores whether a tie-breaking experiment yields better results in the presence of
fuzzy data than an analysis using all available cases. In looking at the results presented
here it is useful to remember that program estimates which exceed a value of about -0.91
or an effect size of about -0.02 are usually statistically significant due to the large sample
size. Because there is in actuality no program effect in this dataset, and because the
results from the null model do not differ significantly from zero (Z = 0.22, p=0.63), the
program effect parameter can be compared to the true value of zero for the purposes of
assessing the magnitude of any bias.
Percent of fuzzy data
Considering the value of the program effect estimate with reference to the total
percent of fuzzy data facilitates an understanding of the general trend in how the percent
of fuzzy data impacts bias in program estimates. Recall that for a given percent of
because fuzzy data is specified as a function of the number of cases in the region. The
question being asked in this analysis is: in general, how does an increase in the ratio of
fuzzy to non-fuzzy data impact estimates of the program parameter?
It can be clearly seen in Figure 9 that as the percent of fuzzy data increases the
bias in the program estimate tends to increase as well. At very low percentages of fuzzy
data (i.e., 2.5% or 5%) the bias in the program estimate tends to be relatively small, but
with as little as 10% fuzzy data there is a substantial increase in bias which continues to
Figure 9. Biased model: effects of fuzziness on program estimates (by percent of fuzzy data)
To summarize the extent to which increasing the percent of fuzzy data creates
biased estimates we can consider the percent of estimates which were significantly biased
with these estimates. Because these results are calculated across all regions of fuzziness
they simply represent the overall trend of how increases in the percent of fuzzy data
impacts bias in estimates of the program effect. As can be seen in Table 12, when the
percent of fuzzy data is very small (i.e., 2.5% or 5%) it is less likely that estimates will be
significantly biased, but as the percent of fuzzy data increases to 10% or more it becomes
likely that significantly biased estimates will occur. However, examination of the average
effect size for each percent of fuzzy data parameter shows that when fuzzy data is equal
to only 2.5% or 5% of a region the effect size is extremely small (not exceeding -0.02).
When the percent of fuzzy data is equivalent to 10% or 20% the effect size is still
relatively small (-.05 to -.10), but it quickly increases to a value of -0.25 for 50% fuzzy
data.
Table 12. Relationship between percent of fuzzy data and significant program estimates
Significant estimates
Percent fuzzy
data Mean Effect Size N
a % 2.5% -0.01 523 37% 5% -0.02 920 66% 10% -0.05 1,161 83% 20% -0.10 1,300 93% 30% -0.15 1,307 93%
Significant estimates
Percent fuzzy
data Mean Effect Size N
a
%
40% -0.19 1,400 100%
50% -0.25 1,400 100%
a
There are a total of 1,400 possible significance tests for each percent of fuzzy data value, 100 for each of the 14 region of fuzziness values.
Overall, as the percent of fuzzy data in a region increases so do both the
likelihood of getting significantly biased estimates and the effect size of the estimate.
However, for very small percentages of data the effect size is very small.
Region of fuzziness
Another way to consider the effect of misassignment is with regards to the region
of fuzziness (i.e., the distance below the cutoff within which the fuzzy data occurs). This
provides a sense of the way in which the extent to which the spread of the fuzzy data
relative to the assignment variable influences bias and provides a complimentary view to
assessing bias from the perspective of the percentage of fuzzy data in a region. Recall that
as the size of the region increases so does the number of fuzzy cases (because the amount
of fuzziness is determined as a percentage of the total cases in the region). Therefore,
increases in the size of the region and the number of fuzzy cases are confounded. The
analysis presented here addresses the question: given the same ratios of fuzzy to non-
fuzzy cases, how does the size of the region in which fuzzy data is distributed relate to
is the effect of increasing the size of the region of fuzziness while holding the number of
fuzzy cases constant, but this is addressed separately in a subsequent section. The
advantages of conceptualizing fuzzy data as a ratio of the region are twofold. First, this
allows us to explore the effects of fuzzy data in a very small area in a simpler way. It is
more challenging to think about the percent of fuzzy data as being 0.01% of the total, but
is conceptually much clearer when specified as 20% of the cases within the region 0.05
standard deviations wide. Second, structuring the analysis in this way allows us to
explore whether having a consistent ratio of fuzzy to non-fuzzy cases has an equal effect