Simple Policy Dummy Analysis

The initial model is a panel regression with a simple [0,1] policy dummy for the years in which the policy is in operation; the results are displayed for reading, writing and maths at each level, in model #1 columns of Tables 2.11, 2.12 and 2.13 respectively. Looking at the tables we see that, despite the introduction of the policy at two different time points, we cannot identify a significant effect of the policy introduction on reading, writing or maths results at either level.

49

At level 2B or higher the estimated policy effect is almost zero for reading, while for writing the coefficient is +0.255 indicating that the policy had the effect of increasing the percentage of children attaining level 2B or higher by 0.255%-points, though it is not significantly different from zero. Similarly for maths, the estimated coefficient of -0.250 suggests that the policy reduces attainment at level 2B or higher by a quarter of one percentage point though again it was not significant. The year dummies are highly significant in all years for writing and in two of the four later years for reading and for maths, and when this common year variation is removed, it appears that there is not sufficient variation in results within LEAs to identify the effect of the policy.

The story is similar for level 3 or higher results, each of the year dummies are significant at the 1% level for each subject, and once this common variation is removed, the remaining variation in results is not sufficient to identify the policy effect.

Appendix A.3 shows the results of the model for the robustness check samples and confirms that we do not get any significant coefficients for either level in any of the samples, though there is generally a consistency to the point estimates for each subject and level across the samples.

The second specification of the policy evaluation model allows the policy effect to differ according to whether the LEA was one of the poorer LEAs that had the policy implemented in the first phase of its introduction. The results for this model can be seen in Tables 2.11, 2.12 and 2.13 but under the model #2 columns. We can see that for level 2B or higher reading and writing, we continue to fail to identify policy effects in either group of LEAs. It is interesting that while not significant estimates, for writing at this level there is a positive effect of the policy for each group of LEAs, whereas for reading the effect is estimated to be negative for the poorer LEAs but positive for the better off LEAs. This pattern for reading is also witnessed in the maths results at level 2B or higher, but for maths while the positive effect for the better off LEAs is not statistically significant, the negative effect for the poorer LEAs is significant at the 5% level. The estimate of -0.772 suggests that in the poorer areas, the policy led to a 0.772%-point fall in the percentage of children attaining level 2B or higher in KS1 maths. To put this in context, the average attainment at this level in the poorer LEAs is 70.79% with a within standard deviation of 2.02%-points. Therefore the policy effect is equal to a reduction of approximately 1/3 of a within standard deviation. Table A.6 in Appendix A.3 shows that this result is consistent across the samples, remaining of similar magnitude and significant at the 5% level and in one sample at the 1% level.

Turning to the level 3 or higher results for model #2, Table 2.11 shows that for reading there is a positive effect of the policy for each group and in the case of the poorer LEAs there is a marginally significant (p=0.112) coefficient of 0.425 estimated. This suggests a 0.425%-point increase in the policy years for these poorer LEAs. The mean attainment in reading at this level in these LEAs over the period is 23.41 with a within standard deviation of 1.90, therefore the policy effect is equivalent to approximately one fifth of a within standard deviation of results for these LEAs.

Similarly for writing level 3 or higher we see from Table 2.12, that there is a marginally significant (p=0.107) coefficient of -0.728 estimated for the policy effect in the poorer LEAs. This suggests that the policy decreased the average percentage of children in the poorer LEAs’ maintained schools attaining level 3 or higher in writing by 0.728%-points, which is approxi- mately one quarter of a within standard deviation (2.76%-points), with the mean being 11.39%. The effect for the better off LEAs is positive but not significant.

Table 2.13 shows that for maths at level 3 or higher, while the policy effect for the better off LEAs is not significantly different from zero, the policy effect coefficient estimated for the poorer LEAs is -0.825 and is significant at the 5% level. This suggests that the policy led to a 0.825%-point reduction in the percentage of children in the poorer LEAs’ maintained schools attaining level 3 or higher in KS1 maths, which is approximately one fifth of a within standard deviation (4.03%-points), while the mean is 23.50%.

Appendix Tables A.7, A.8 and A.9 show that these results are robust to the choice of sample and in many cases strengthen in significance in the alternative samples. Moreover, the Appendix A.3 Tables A.10 to A.15 show that all of these results remain when we exclude the 2006 data on account of the change in assessment method introduced in 2006.

It appears that there is a significant policy effect on results but that it is only in evidence when we allow the policy to have different effects in the two groups of LEAs, and the effect is only in the poorer LEAs. While there is a slightly significant positive effect on reading level 3 or higher in the poorer LEAs, for writing and maths it is a significant negative effect of comparable or greater relative magnitude. The negative policy effect on maths results in the poorer LEAs is also in evidence in attainment at level 2B or higher. These are interesting results and slightly alarming in that the policy was designed to be a benefit to the poorer LEAs in particular and it is in these areas that the policy is having a significant and mainly negative impact on results. To investigate further I estimate the more complex model, exploiting variation in the actual

take-up rates to examine the way in which this policy is working.