Using multi-level modeling allows one to study effects that vary by groups. It also allows estimation of group level averages. One cannot see crossed level effects any other way. Regular regression ignores the average variation between groups. In addition, individual regression can experience sampling problems and lack generalization. A dissimilarity index model is another option, but it does not appropriately take into account the spatial patterning of the three levels and would ignore the variation across levels.
Four sets (eight total) of multi-level regression models are conducted. Each set includes one model for communication arts MAP tests and one model for mathematics MAP tests. The first set is a three year model that covers the third through tenth or eleventh grade students’ MAP test scores between 2006 and 2008. After 2008, the requirement of high school students to take the MAP test ends. Tenth and eleventh grade students are not included in the five year models. The second set of models is a five year model that includes the third through eighth grade students’ MAP test scores between 2006 and 2010. The third and fourth sets of models are time interacted models. These four models provide a better assessment of the transfer program’s effect on student achievement from 2006 to 2010. Students who are in the third grade in 2006 make up the third set of models. Students who are in the fourth grade in 2006 make up the fourth set of models.
All models have a hierarchical leveling structure where individual students are nested within school buildings and those school buildings are nested within school districts. The
following chapter explains the construction process of the models and the tests for the best model fit. It also discusses the tests that ensure the data are statistically significant from each other and
that there is no overlap in data. Graphs illustrate the variation in achievement based on participation over time.
To construct the hierarchical linear model, there is a three step process. The first step constructs the base model. This model computes the estimate mean score of all observations and identifies the standard deviations at all three levels. This model only includes the dependent variable. The second model predicts growth of achievement across student types by including the independent variables. It allows for random intercepts and slopes, but does not include predictor variables at the second level (school building level) and third level (school district level). This tracks standard growth. The third and final model construction includes the predictor and control variables for identifying the average growth in achievement. The participation and predictor variables are not included for level two (school building) and level three (school district) of the model.
In the base model, the estimate of the mean score for each mathematics’ and
communication arts’ test scores determine the standard deviations at the school district, school building, and student levels. The intra-class correlations are also determined at the district level and student level to detect achievement between two students in the same district, and between two measurements on the same child in the same school. This allows one to see how much variation in scores are attributed to the school district level predictors and school building level predictors, and how much are attributed to the student. It is ideal to see more variation at the student level than the school district or school building level because lower levels are closer to the observed occurrence (in this case, the MAP test scores).
To test for the best fit of my model, the Aikaike information criterion (AIC) and Bayesian information criterion (BIC) indicators are examined. The AIC and BIC indicators are two measures that compare maximum likelihood models by looking at the number of
estimated parameters (AIC and BIC) and the number of observations (only BIC). The BIC indicator is considered to be a more stringent measure. In general, smaller AIC and BIC results are preferred because they indicate a better fitting model. Three sets of AIC and BIC results are assessed to determine the appropriate model for assessment. These results are located in the next chapter with each individual model’s regression results.
First, an empty model records the AIC and BIC indicators. The empty model only includes the dependent variable. In the second assessment, the AIC and BIC indicators of the model are recorded with the dependent variable, independent variable (student type dummy variables) and the three level model specification. The third AIC and BIC indicator
assessment involves including the dependent variable, independent variables, other control and predictor variables, and the three level model specification. As the inclusion of more components in the model occurs, the AIC and BIC indicators begin to decrease in size. The indicators produce a very high value each time. This is due to the high number of
observations in the models. Though the values were high, they continually decrease when testing each new model. This confirms the fit of the third model is appropriate for the assessment.
For both Communication Arts and Mathematics three year models and five year models, a three-level hierarchical linear (HLM) model (four models total) is conducted to assess the variation of student achievement on the MAP test across the five types of student participation. After running each model, an assessment of the level of variation explained at the student, school building, and school district level is taken. This assessment is performed for each year within each of the four models. The following chapter explains the level of variation for each model.
To provide a visual of achievement based on participation, a plot of the lines for the five types of student participation is graphed. These graphs show the average rate of achievement on the MAP test based on the type of participation across all students. More specifically, these graphs illustrate the gaps in student achievement across student types during these five school years.
The three year models and the five year models are very similar in design. The only difference between the two is the exclusion of the students in high school grade levels in the five year model. After 2008, high school students no longer took the MAP test. Therefore, the communication arts five year assessment covers the estimated mean scores of all students who complete the MAP test in third through eighth grades between 2006 and 2010. Two major effects occur with the exclusion of the high school students. One, there is a substantial decrease in the number of observations at the student and school building levels. This is due to not observing the high school level school buildings. Two, the average estimate of the mean score is lower. This can be due to the higher range of scores high school students are able to attain on the MAP test in comparison to the lower grade levels.
Two time models are conducted for each of the two test subjects. The first time model uses the observations of students who are third graders in the 2005-2006 school year. These students are seventh graders by the 2009-2010 school year. The second time model uses the observations of students who are fourth graders in the 2005-2006 school year. These students are eighth graders by the 2009-2010 school year. In this model, the time variable is centered to ensure a mean of zero across the school years and then is interacted with the student participation dummy variables. After running each of the models, the gap of MAP
The time interaction in the third and fourth sets of models measures the growth of achievement over time. A centered school year tracks this growth. To do this the school years are recoded so the years fall on a continuum from -2 to 2, where a test score from 2006 receives - 2 and a test score from 2010 receives a 2 (the other three school years are in between). This step creates a constant for time and allows the variable to be centered on zero. The constant term represents year 2008. White county students remain the constant. By obtaining the correlation of the random effects, the model identifies the correlation between the student’s achievement in 2008 and his/her rate of growth. Also, the correlation between a district’s achievement around fifth grade and the rate of change per year is identified. Again, it is expected that most of the variance between student achievements is at the student level rather than at the school building or school district levels, but the variation in rate of change per year is expected to remain about the same at all three levels.
Each dependent variable is interacted with the centered school year in order to assess the rate of change over time at the school district, school building, and student level. This tests the effect of the length of time a student participates in the transfer program on student achievement. The time interacted student aids in identifying the number of times a student is a particular type of student over the five years.