6.3.1 What is the value of taking a longer term approach to impact evaluation?
Arguably, regardless of outcome, it will always be of value to allow for a taking into account a number of years of firm performance post intervention, ideally four to five years as found above. Even if no long-term effects are found, and a scheme appears only significant in the short- term, the clear policy implication is that the scheme in question may well have provided a short-
- 170 -
term boost to the support group of firms, but that the effect is fading in the mid- to long-term. The other way around, where long-term effects are shown by an evaluation, it is clear that a short-term evaluation could have underestimated the full economic impact of a scheme. Potentially, it would have estimated the impact as entirely insignificant, if the short-term evaluation’s cut-off comes before a scheme’s impact had sufficiently developed for a significant impact to develop. There is no doubt about the vital importance of timing in evaluation.
6.3.2 The implications for practice
The resulting challenges for business support evaluators are diverse: - Explain the choice of timing for evaluation
The evidence of this study suggests that significant assistance effects appear mainly for the two to three year period, using conventional methods, depending on the measure used, which in this thesis is employment versus turnover. A period of five years of post intervention data would have captured the main effects here. If a number of similar studies existed, authors may refer back to studies of comparable schemes for their assumptions around timing of their evaluation. However, given the variation of impact observed by length of time, for robust evaluation the answer can only be to evaluate outcomes for a number time periods, as done herein. Of course, the longer the time period assessed, the more likely it is that other effects may distort the findings – the attribution problem. It is not unlikely that firms assisted in one way may also benefit of other assistance in parallel. Changes to firm characteristics are increasingly likely to distort findings over time. This will always limit the maximum sensible time horizon for analysis but the question is where to apply the cut-off? The time period included in evaluation needs to reflect the magnitude of the scheme’s impact – which of course can only be really known after evaluation.
- 171 -
- Explain the required time needed to funders and those requesting fast understanding of the scheme
Whilst the first point around timing is of rather theoretical nature, it is also arguably detached from the practical requirements of the ‘real world’. The demand for support evaluations has a clear focus on rapid evidence; a five year evaluation would not fit within a single government term, including the time it takes to first design a support programme and to implement it. It would be naïve to ignore the push by policy-makers and funders for short-term analysis. However, as done with the BLO Survey data used for this thesis, such relatively short-term studies may be carried out initially to provide a first understanding of impact, and can then be subsequently extended longitudinally. That, of course, assumes that there would still be a market for findings on schemes being some five years old and possibly long abandoned. It also raises the question of the availability of funding for such evaluations with a potentially limited audience.
- Highlight to the reader what the results actually tell
Citations referring to “successful” schemes need to embed evidence into more context. Rather than suggesting, ‘Scheme XY was successful’, an addition that ‘A one-year impact assessment suggests Scheme’s XY effectiveness early on [and this evaluation is robust/not so robust given its methodological approach]’ would highlight that the results are only true for when measurement was taken. Particularly the grey literature, produced for policy-making purposes, often provides little context on findings. The awareness must be raised that most existing evidence cannot universally confirm (or reject) a scheme’s impact, it can only do so in the context of the timing of its applied measures. Of course, this also works in reverse, with users of evaluation results urged to be looking at the detailed context in which any particular scheme was deemed to be working or not.
- 172 -
- Device a strategy that provides the required, potentially longitudinal, data
This is mainly a question of resource and between the choice of primary and secondary methods for data acquisition. Rich longitudinal data on firms is hardly at all available through secondary sources, at least in the UK84. This thesis proposes
one potential approach, where an extensive survey covering multiple firm characteristics gets linked to certain performance measures, reducing the need for resource-intensive primary data collection. Programme design could include automatic transmission of the inventory of assisted firms, allowing for that data to be stored and returned to later, even if the programme office has long been dissolved. However, comparable availability of official data as in this thesis may be limited by geography but also topic (see the linking of the BLO Survey to the UK Innovation Survey). Given data linking success rates (=attrition), the original sample needs to be of sufficient size.