Using older siblings

Another way of obtaining a distribution of characteristics of individuals in pilot areas who were unaffected by EMA, is to examine the post-16 participation decisions of the older siblings of our existing set of 16-year-olds. This allows a comparison of participation in post- compulsory education between pilots and controls with a reasonably full set of individual controls, prior to the introduction of EMA in those areas, to identify any unobserved area effects.

For the results derived in this way to be valid, a number of strong assumptions are required, including that:

1. The sample of older siblings is representative of the population at large, at least in the dimensions that determine continuation in post-compulsory education;

2. Differences in schooling between older and younger siblings are the same between pilot and comparison areas;

3. Differences in the macroeconomic environment at the time when the older siblings were at the relevant age affect education participation decisions in pilot and control areas in a similar way;

4. The family has not moved between LEAs since the older sibling(s) was (were) making their educational choices;

5. Changes in background characteristics that determine choices are such that the impact on aggregate participation change is the same in pilot and control areas; and

6. Unobserved characteristics of the older siblings that determine their educational choices do not vary systematically between pilot and control areas.

Differences in birth order8_{and parental age can be controlled for within the data.}

Requirements 1, 4 and 5, however, cannot be adequately taken account of in the methodology due to lack of data. Additionally, requirement 6 may not be fulfilled. Most seriously, the gender of those siblings who have already left the household (who make up 55 per cent of all siblings) is not known.9

This is potentially a very serious omission, as educational choices of males and females tend to be differently determined. Further information about each

sibling’s ability, as measured by GCSE results, is not available to control for possible differences between pilots and control areas on these characteristics.10 _{Nevertheless, the} exercise may still be informative and provide a further check of robustness of the results.

Information on the post-16 education decisions of older siblings can be used in two different ways, each with different underlying assumptions and each generalisable to a different degree.

1. The first methodology automatically matches those young people with an older sibling in the pilot areas to the closest young person with an older sibling in the control areas. This means that each pilot and control area young person is matched to his or her own older sibling only. Individuals with more than one older sibling, are matched to the next youngest sibling.11

The education decisions of young people and their older siblings in pilot areas can then be compared with those of young people and their older siblings in control areas. This methodology has the advantage of controlling for all household characteristics that affect education decisions and that do not change over time. The disadvantage is that the analysis only provides the impact of EMA on those with older siblings – among whom participation in post-16 education tends to be lower. 2. The second methodology matches all young people to an older sibling who looks most

similar to them on the basis of their characteristics. This may or may not be their own biological sibling. Some of these characteristics will be different between older and younger siblings – for example, parents’ ages.

4.5 Conclusion

The quantitative evaluation of the impact of EMA involves matching individuals in control areas with similar individuals in the pilot areas to estimate the impact EMA has had on initial decisions to remain in full-time education post-16. Matching is based on the assumption that all differences relevant to school participation between those in a treatment (pilot) area and those in a control area can be accounted for by controlling for observable characteristics in the data. The participation rate of individuals in a control area with the same set of

characteristics as those in the pilot area estimates the participation rate that the respondents in the pilot area would have had, had they not been subjected to the policy.

In order to examine the incremental effect of EMA on participation and to make detailed comparisons between variants, it is necessary to place more structure on this simple matching procedure. This is done by looking at the determinants of education participation using a regression model and including estimated EMA entitlement in this model. The structure of this model allows the effects of policy changes to be simulated.

For matching to work, it is crucial that no factor (relevant to the outcome variable of interest), other than the observed characteristics controlled for, varies significantly between the pilot or the control areas. If this seems likely not to be true, then a procedure needs to be developed that can eliminate these unobserved area effects. This chapter has discussed a number of ways that could be used to try to difference out these unobserved effects and, hence, check the robustness of the matching results. The results of all this quantitative work are discussed in Chapter 5.

5

The Impact of EMA