The design
If the aim of an evaluation is interpreted as one of attributing causal effects between the intervention and outcomes, our design has a number of weaknesses that result from the inability to control for confounders. We mitigated this in so far as we were able by our design for the quantitative elements drawing on best practice for designing studies of natural experiments.81In particular we published a protocol for the quantitative component
of the study92(see Appendix 7), specifying hypotheses and the subgroup analyses planned. We identified
secondary data sets that provided before-and-after data (for travel modes), and some time series data (for injuries and assaults). Our use of comparisons with adults in London provided proxy‘non-exposed’ controls, as changes in young people relative to adults are unlikely to be the result of general changes in, say, bus provision in the capital unless these other changes specifically affect young people disproportionately. We have also used proxy ‘dose–response’ measures, such as comparisons between inner and outer London and between boroughs with greater or lesser uptake of free bus travel, where possible. Finally, we developed a logic model to help
In line with Medical Research Council (MRC) guidance on complex interventions,87we included an
economic evaluation to assess the costs and benefits of this intervention, from a societal perspective. This is inevitably limited in that not all costs and benefits can be monetised, and many (such as reduced future dependence on private car travel) are too far in the future to include. However, in the context of increased scrutiny of concessionary fare policies, in almost all scenarios the benefits outweigh the costs – the only exception is a sensitivity test for a hypothetical case in which the additional capacity costs of bus operation were assumed to have increased substantially.
There are of course unknown confounders, and the more general problem of picking up‘signal from noise’ when looking in isolation at individual changes within complex systems. To offset these weaknesses and build a fuller picture of the impact of the intervention on public health as a whole, we have utilised a multimethod approach which has built up an assessment of public health impacts in an iterative way. To link process measures (transport mode change) to health outcomes, we conducted a systematic review of prospective studies on active travel to assess the strength of evidence on whether or not increasing the amount of active travel in the population was likely to benefit health.
Using secondary data sets
The use of secondary data sets made a before and after evaluation feasible, given that there were existing data sets relating to key outcomes of interest. However, there are always limitations in using data for purposes other than that for which they were designed. Here, the travel diary data available for London, although more detailed and extensive than NTS data, only related to term-time weekday travel for our pre-intervention period. Given that young people’s travel behaviour differs across the year and week58we
may have therefore underestimated both positive and negative effects of the scheme on outcomes relating to mode change. The use of qualitative data allowed us to offset these weaknesses to some extent by providing some data on other journeys.
The data set on RTIs, STATS19, has some well documented limitations in terms of completeness due to the under-reporting and under-recording of traffic collisions.142However, this only presents a threat to validity
of our results if data completeness has changed over time disproportionately for the target group in relation to our comparator groups, which is unlikely.
Hospital Episode Statistics data, available at a national level, enabled a comparison to made between the incidence of assault injuries in London and those occurring elsewhere in England. HES data record the age of the patient and therefore allowed age-specific rates to be derived. Furthermore, HES data are more likely to be complete over time and so the chance of reporting bias is reduced, unlike data collected on crime occurring on the transport system (e.g. bus incident reports). However, the coding of location within the external cause of injury code is not complete for a substantial proportion of records. This meant that a detailed analysis of assaults according to the place of occurrence (e.g. on the bus, in the street) was not possible. As the hypothesised pathway between the intervention and incidence of assaults did not specifically refer to assaults on the buses, but rather that young people’s increased travel would leave them vulnerable to assaults in general, the lack of location was not a major limitation.
The population: ‘young people’
Our analysis largely addressed the implications for public health of‘young people’ as a whole, rather than attempting to differentiate the effects by gender, deprivation, or other variables. In London, there are, for instance, known differences in road injury rates by gender, deprivation,143and ethnicity,43and on the
likelihood of walking for different kinds of journey by ethnicity.58It is plausible that these demographic
factors will therefore modify the effects of the scheme on outcomes such as transport mode change and RTI. Our focus in this study was on the health effects at a population level, and we used subgroup analyses for sensitivity tests and for estimating dose–response proxies only. That is, our aim has been to use these subgroup analyses to strengthen the credibility of claims made about the population as a whole, rather than to identify subgroup differences per se. A similar approach was taken to the qualitative data,
except where location (e.g. inner or outer London) or gender has relevance for the interpretation of our findings. We sampled for heterogeneity in terms of residence, gender, ethnicity, age and transport modes available, in order to ensure that our sample was not representative of only a narrow sample of the population, but have not sought to analyse our qualitative data comparatively across these characteristics, and a larger sample would be needed to explore differential impacts of the scheme across populations. We are aware that there are likely to be large differences in how issues such as‘independence’ or ‘risk’ are discussed across gender144or age, with those aged 12 years likely to differ from the older age groups
included. Our aim in this study was not to add to the literature on these differences, but more research is needed on how effects likely to be important for the public health are distributed across populations.
The comparator: adults aged 25–60 years
To ensure an adequately powered comparison, we have included all adults aged 25–60 years as the main comparator group (i.e. those‘non-exposed’ to the intervention). Although this is a pragmatic choice, including only those with no direct experience of free travel as young people, it does have some limitations in that it includes a larger range of ages than the intervention group. Additionally, as a‘control’ group, there are limitations in that adults aged 25–60 years have been subject to some interventions which are likely to have affected their travel choices disproportionately compared with the‘intervention’ group (such as schemes to encourage cycling to work). We have also been unable to assess whether or not the intervention has shifted bus travel by adults to London underground train services, which is one hypothetical outcome of higher bus use by young people.
Strengths of natural experiments
One difficulty facing researchers evaluating complex social interventions is that the intervention effects are moderated by the context.91We therefore provided details of the context of this intervention and have
described how this affects the studyfindings.
We suggest, for instance, that the general context of improved bus provision, which also affects adult bus use, and is thus‘controlled for’ to some extent in our design, is also an important precondition for the effects we see from the intervention. This is because it‘normalises’ bus transport for the wider population, while also making it viable for young people to exercise peer-based preferences for travel. Without good transport provision, even if all young people had free travel, bus travel would not necessarily be
experienced as a‘normal’ and reasonable way to travel, as this also required that other Londoners were using buses, and that buses were relatively accessible and efficient. This was evident in the accounts of young people with disabilities: free travel was not, in the absence of accessible and good transport, a contributor to social inclusion. However, the fact that most young people were making frequent use of the bus service also has some (if marginal) consequent effects on bus use by adults, as reported in the
occasional stories in groups of family outings that are now possible because young members get free travel. This in turn reinforces the normalisation of bus travel as a mode of transport for the whole population. A natural experiment enables a description of what happens in‘realistic’ policy environments (i.e. ones in which a mix of interventions is likely to be implemented simultaneously).
Finally, an important strength of a natural experiment is that the importance of the system as a whole can be appreciated. One keyfinding of this study may not have been identified with a randomised controlled design, should this have been theoretically possible. This is that the structural properties of this transport system have effects which are not simply‘additive’ at an individual level. Specifically, many of the effects of the scheme which have been identified in the qualitative findings arise not from the fact that bus travel is free, but from the fact that it is universally free for young people.145This is what is sometimes termed an
‘emergent property’ of the system (i.e. a property of the system as a whole that arises from the interaction of individual parts but cannot be predicted simply by summing the activity of those individual parts).146,147
Had the scheme been restricted to particular types of journeys (e.g. the journey to school) or specific groups (e.g. low-income families), many of the effects evident in young people’s accounts may not have been realised. We have described in detail how the effects of the scheme are in part a result of the fact
nature of young people’s travel choices and become the default mode. The exceptions demonstrate that these effects only hold when transport is both available and accessible (as young people with disabilities do not enjoy them) and suggest that free bus travel, rather than improvements in general in the bus service, are responsible (as those who have no Oyster card did not benefit).
An evaluation of a design which focused on the individual benefits of free travel (for instance by only offering the scheme in some boroughs, or to some income groups) may not have identified this.