However, for all the obvious benefits and undeniable important contribution made by the Six
Steps framework, the overriding concern is that it (or the SMS) fails to reach far enough for
truly robust evaluation. There appears to be a real danger that the evaluation community possibly has lulled itself into some false sense of certainty around the robustness of its findings.
This is not a novel concern. Possible extensions to the Six Steps framework have been suggested previously. Roper and Hart (2003) provided a tracker study of UK Business Link assisted firms in 1996, using 1994-2000 data. Following Storey’s (1998) suggestions, they included a non-assisted control group and accounted for selection bias, referring to PACEC (1998) and Cosh et al. (1996) in this context. On the basis of their findings, they expressively underline “the value of striving to achieve methodological paradise”. For the tracker study this included accounting for various factors beyond actual assistance. Guided by previous research – for example Storey (1994), Barkham et al. (1996), Roper and Hewitt-Dundas (2001; in Roper and Hart, 2003) – market conditions, business strategy, characteristics of the owner-manager and firm are all emphasised as influencing firm performance (e.g. employment, turnover or productivity growth). The Six Steps framework would, on paper, rank an evaluation that excluded such environmental factors as high as Roper and Hart’s (2003) study that does include those environmental factors. Arguably, the inclusion of such environmental factors simply represents good econometric practice and does not require specific mention in an evaluation framework. The very same could, however, be argued around using the counterfactual and addressing potential selection bias, both aspects that the evaluation frameworks specifically make reference to. These also certainly represent nothing more but good econometric practice.
- 37 -
What was found to be of particular interest in Hart and Roper (2003) is their call for government to continue prioritising well maintained databases of “clients” (that is, firms that interacted with support outlets). The authors are clear in expressing the contribution such solid data makes to robust evaluation – arguing that without the “longitudinal databases specifically constructed for this purpose” they would have been unable to distinguish assistance and selection effects. This reference to a longitudinal database highlights an interesting aspect – that of time.
2.4.1 Evaluation objectives and the role of time
Storey (1998) attaches considerable weight to the importance of clear objectives for a policy intervention which later serve as reference points for evaluation. As previously referred to, Doran (1981) provided the most widely known criteria for defining good, or more appropriately, smart objectives. Smart objectives, Doran (1981) argued, are to be specific, measureable, assignable (later often stated as achievable), realistic and timed objectives. Storey (1998), and subsequently Cook et al. (2008), the OECD (2007) and the World Bank (2009) all call for well- defined objectives to be put in place ex-ante. This would then allow later evaluation to exactly estimate the desired programme effects.
Considering the individual SMART criteria, it is clear that for good evaluation specific objectives are called for. As these specific objectives are expressively required to allow later evaluation, they are also intended to be of measurable nature. Depending on the chosen definition of the letter ‘A’ in SMART, business support programme objectives are both to be assignable and possibly achievable – the latter as proven or disproven by the subsequent evaluation carried out. The former, assignable, is implied by the nature of these programme objectives: The interest lies in the impact of support provided, as such the objectives are seen as the consequence of the support. Objectives may be more or less realistic, this can only be based on previous evidence of similar schemes, and it is unlikely that policy-makers or evaluators would set themselves too aspirational objectives – unless they would receive some sort of gain from doing so, of course. Certainly though, business support evaluations have shown that not
- 38 -
all programmes make a positive contribution, but whether that is due to honest error or unrealistic objectives (out of whatever motivation, and around impact on the assessed measures) remains unknown.
Simplistically said, the interest here is with the T of SMART in objective setting. When are the defined objectives to be achieved? What is the duration of the desired effects? Evaluations appear to be carried out on the basis of the belief that at the point of evaluation, programme effects have been achieved. None of the literature presented in Section 2.6 justifies its choice of timing, or states that it makes an assumption of impact having been achieved. It must be made implicitly, as otherwise an evaluation would be of little sense. Programme impact may not be measureable if assessed too early, or it may fade in the long-term. And even if measureable, would the strength of impact be constant over time? There may be instances, for example, where programmes have an initial positive or negative effect on performance, and the reverse in the longer term. King and Behrman (2009) provide one of the few examples of a specific call for considering the impact timescales as part of the evaluation design.
Because of the importance of impact having been realised (and continuing) just as the evaluation is being carried out, this implicit assumption of evaluating at the right point in time should certainly be explained by evaluators. Why would the chosen point of time for evaluation be reasonable in the context of their scheme assessed? In fact, no evaluations – based on the author’s review of the evidence – appear to provide such rationale for their timing.
The question whether the role of timing of an (business support) evaluation is rightfully ignored, or highlights a certain naivety or ignorance by researchers and those commissioning research, is inevitable. The answer is likely to be linked two central concerns faced by evaluators. Along with data availability, political considerations and pressures play an important role here, and this is discussed in Section 2.7.
- 39 -