Missing data can present several difficulties, reducing the analytical sample size and, consequently, power, as well as potentially introducing bias in the results when teachers with and without missing data differ systematically on their responses to focal measures of interest. The latter can be especially difficult, if not impossible, to detect. However, a thorough analysis of missing data precedes the main analyses in this study in an attempt to address some of these concerns.
Missing Data at Level Two (Schools)
The first step in exploring missing data was to examine unit-level missingness of school leader survey data. The school-leader survey would have been a potential source for measuring hypothesized school-level mediators of interest in this study. However, year-two surveys were completed by school leaders in 60 of the 67 schools in the sample, and each of the 7 nonrespondents were from different matched pairs. If school-leader data were used to measure hypothesized school-level mediators, all analyses should exclude
the responding matched-pair school leader for each of the nonresponding school leaders in an attempt to maintain internal validity through treatment- and control-school balance. This would further reduce the available year-two school leader sample to 53, making the use of the school leader data for level-two (school-level) measures of leadership and culture problematic. Not only would it reduce the year-two school leader sample, but any multilevel analysis would also eliminate teachers in these 14 schools. This would reduce the sample of year-two teachers by 20 percent (from 616 to 492) and have implications for validity and statistical power.
This reduction in sample size was the justification for using aggregate year-two teacher survey data to generate the school-mean school-level mediators of interest in this study. This strategy ensures that there are valid values for each of the hypothesized school-level mediators for all 67 schools (i.e., no missing level-two data). The issues associated with this method were discussed in chapter three and judged to be outweighed by the benefits. Values for all other cluster-level (level-two) covariates – the treatment indicator and indicators for member district, data collection wave, and schools in the Chelsea three-school “pair” – are known for all schools.
Missing Data at Level One (Teachers)
All year-two measures of mediator and outcome scales or indices were calculated for teachers who responded to 80 percent or more of the items in the set. This rule was set after examining the distribution of missing data. Across the items sets, respondents were generally missing either one item in a set or all items in a set. Therefore, choosing an 80
129
percent rule versus a 50 percent rule made very little difference.3 As a result, the choice
was made to use the higher standard – at least 80 percent nonmissing responses – in the calculation of scale scores. An exception was made for any scale or index comprised of four or fewer items; in these cases, teachers must have responded to all items to generate a nonmissing composite score. In cases where these conditions were not met, the mean scale or index score is missing. Using this 80 percent rule, the number of teachers with missing scale or index values ranges from 0 to 44 (at most, about 7 percent of the teacher sample).
Imputation of missing items or scale/index scores was not performed for two reasons. First, most cases of missing scale or index scores are missing because teachers skipped every individual item in the set used to construct the scale or index. This would make imputation more difficult and would require modeling a mean scale or index score using other survey information. Second, both the proportions of teachers who are missing on any given teacher-level scale or index, or from any regression model, is less than 10 percent. When the proportion of missing data for a particular variable or scale is less than 10 percent and the number of cases with no missing data is large enough to support the selected analysis technique, missing data can generally be ignored as it is unlikely to introduce bias into the study (Hair, Black, Babin, & Anderson, 2010; Dong & Peng, 2013).
3 For 6 of 11 scales, there is no difference in the number of cases for which a valid composite scale measure
is generated using the 50 versus 80 percent rules. The remaining 5 scales would gain between 1 and 4 additional teachers (out of the possible 616) using a 50 percent rule.
A significant proportion of teachers are missing baseline data and, therefore, the main models in the study do not control for baseline measures of the outcomes of interest. Models in appendix B show results for the subset of teachers with nonmissing baseline data. Because other level-one covariates are measured in year two, the proportion of missing data is much smaller.4 Only 2 percent of teachers are missing data on their
highest degree and years of teaching experience; these data are also not imputed. In sum, the use of teacher-level responses to generate aggregate school-level measures and the universal availability of school- and district-level covariates ensured that there was no missing data at level two. While there is missing data at level one, no imputation was performed since the rate of missingness was unlikely to introduce bias in the results. In a given analysis, individual teachers with missing values for teacher-level variables specified in the model are dropped.