School-Level Random Effects

Of the 100 papers published between 2013 and April 2015, 45 were correlational and in 7 of these, the main findings related to school-level random effects, (i.e. school value-added). Of particular relevance are a group of methodological studies into value-added measures. These studies are reviewed in detail in the next chapter along with other important methodological studies which were published outside of SESI or JREE. As a result, this section simply summarises their focus and conclusion, as follows: First, Dumay et al. (2013) examined the stability over time of value-added measures using different designs. They conclude that the low level of stability found ‘poses a significant challenge to the conventionally accepted view that we can make a generalized evaluation of how effective a school is, based on cross- sectional data from a single cohort’ (Dumay et al., 2013, pp.78-79). Another methodological study examining the stability of value-added estimates over time is Ferrão and Couto (2013) in which value-added estimates were produced for cohorts across three years and 9 grades in compulsory education in Portugal. Ferrão and Couto (2013, p.186) concluded that “the findings reveal a systematic pattern of educational units’ performance is more than just randomness.” At this – rather low – threshold for value, they conclude that Portugal should include a VA indicator into its system of evaluation. The two other methodological studies covered in this review are Televantou et al. (2015) and Lenkeit (2013). The first of these examined the impact of measurement error on value-added estimates, finding that traditional approaches to estimating school effects are positively biased, giving rise to ‘phantom’ compositional effects associated with school average achievement, for instance (see Chapter 4, Section 4.2.2). The second study compared value-added measures with estimates produced using growth models and cross-sectional estimates – so-called ‘contextualised attainment models’. The latter were found to be ‘adequate substitutes’ (Lenkeit, 2013, p.39) for measures including prior attainment, although their predictive power is lower and correlations in some cases only moderate.

What can be seen from these studies is that bias, stability, the strength of causal inferences and the specification of value-added models are all current concerns in educational effectiveness research. Another key point is that, of the 7 EER research papers

which have been reviewed which focus on school-level value-added scores, 4 are methodological studies examining the properties of value-added scores rather than attempting to draw conclusions using them. Only 3 (or 3%) of the studies specifically base conclusions on school-level value-added scores. These are as follows:

First, Noyes (2013) looked at school effects on mathematics performance using the National Pupil Database as well as the effect on subsequent outcomes such as participation at higher levels of mathematics. Noyes compared official school contextualised value-added (CVA) with CVA scores for mathematics only, created for the study. The latter are also used to examine how effective schools are in encouraging pupils to progress to more advanced study. Schools are found to have a ‘very real effect’ on mathematics progress to age 16 and on post-16 participation but that there was little correlation between these two (Noyes, 2013, p.101). The overall school CVA scores and mathematics CVA scores are found to show a ‘considerable degree of variation’ (p. 95), suggesting differential effectiveness between school departments.

The second study basing findings on school-level random effects is Isac et al. (2013) who compared outcomes across schools in different countries. They studied school and system effects on outcomes relating to citizenship education such as civic knowledge, civic attitudes and intended civic behaviour. They found only very small differences in attitudinal non-academic outcomes between schools but found some differences between systems and between schools in relation to civic knowledge. Like many studies reviewed in the previous section, they note the difficulties of using a correlational design and recognise that this prevents causal conclusions being drawn.

A final study considered here is Sammons et al. (2012). This paper investigated the impact of early disadvantage and the role of schools in ameliorating this. A multilevel structural equation model (which they note is a form of value-added statistical analysis) is used to track associations between self-regulation, academic attainment, early disadvantage and academic effectiveness over time, where a CVA measure is used to estimate the academic effectiveness of the schools. They found lasting effects of early disadvantage on regulatory skills throughout primary school. They also found associations between their measure of academic effectiveness and the academic attainment and self-regulation of children experiencing early disadvantage. Sammons et al. (2012, pp.15-16) drew a causal conclusion, stating that ‘what this paper has shown is that more academically effective

primary schools can make a significant difference to the academic attainment and self- regulation of children who experienced more disadvantages early on in life (before age 5 years)’

These three studies show that value-added is in active use within educational effectiveness research. They also continue to demonstrate differences in emphasis and caution in causal interpretations of the results: some authors explicitly rule out causal interpretations, while others present evidence as showing the ‘effects’ of schools and educational effectiveness factors, recognising but not being deterred by the threats to validity noted above.