Isolating th e effects of micro-level policy instruments

Im pact evaluation has an integral role to play in th e policy making process in these

circumstances. Evaluation cannot be left at th e end o f th e line but instead, it has to

be a key e lem en t o f initial policy form ulation (OECD, 2008). M o rto n (2009) notes

that:

Im pact evalu atio n encourages a shift in th e focus o f policy evalu atio n fro m th e past to th e fu tu re and p ro m o tes th e use o f evalu atio n fo r policy learning ra th e r th an simply fo r cost c o n ta in m e n t o r control, (p .4).

54 | P a g e

Once th e policy is operational, all organisations and individuals responsible for

delivery have to be aw are th a t evaluation is to take place. It is recom m ended th a t

m onitoring and fo rm ative and sum m ative evaluation should be integrated into the

program m e design (W alker & W isem an, 2006).

Once th e evaluation has been undertaken it should be used as the basis for dialogue

w ith policy makers, w ith th e objective o f delivering b e tte r policy. The outcom e o f th e

evaluation can then becom e an input into th e debate on th e most appropriate ways

fo r governm ents and SMEs to co-evolve to pursue econom ic grow th (Breznitz, 2007,

2012). Policy form ulation can thus becom e 'evidence-based'. Policy can thus be

im proved through a process o f'ra d ic a l increm entalism ' (Eisenhardt & Brown, 1998).

55 | P a g e

Figure2.3: Mappingmicro-level stateinterventionin the growth-orientedfirmin the small state

1X3JU03 aieis liews

s j r u e j a i n s a j e i s | |e l u s t - «=

Souce: Storey, 1994; Smallbone & Wyer, 2006, 2012; Davidssoneta\.,2010, Hill & Jones, 2009; White, 2009; Mayne, 2001, 2012- adaptedbythe author.

This approach emphasises th e need for a clear high-level view on SME policy's long

te rm strategic direction. N ear term initiatives or experim ents can then be

contextualised by this longer view. W ith o u t this blueprint th e near term initiatives

begin to lose coherence (Brown & Hagel, 2003) and a smorgasbord o f policy

initiatives ensues (Bennett, 2012). In tandem w ith this Simonian (Simon, 1968)

'in crem ental' learning approach, th e state can also involve itself in 'm arket making'

i.e. putting in place th e conditions for new m arkets and acts as co-ordinator or

facilitato r w h ere th e re are obvious co-ordination failure (Sissons & Thompson, 2012).

This requires policy makers to have superior research and diagnostic skills, be pro­

active and forw ard looking, but also effective at working w ith business and

institutions (Link & Link, 2009).

A ttem p tin g to evaluate th e effects o f program m es is complex and difficult as th ere

are myriad influences on and determ inants o f the perform ance o f an SME - other

than th a t of program m e participation (Figure 2.3). In essence, only w hen all known

factors (including chance) are controlled for can th e im pact o f the program m e be

accurately estim ated. Given the paucity of cum ulative knowledge in the firm growth

space and th e lack o f agreed specificity around the num ber o f key variables, it is little

w o n d er th a t th e re is controversy in th e im pact evaluation field over the most

appropriate approaches and m ethodologies.

The OECD (2000) identified seven headings under which policy interventions could

be assessed. These are: Rationale, Additionality, Appropriateness, Superiority,

Systemic Efficiency, Own Efficiency and Adaptive Efficiency. Each of these areas is

im portant in its own right and should receive th e appropriate attention if designing

policy instrum ents but th e most im portant is th e concept o f Additionality. This is

defined as th e true im pact o f th e schem e/program m e. Theoretically this makes great

sense and whilst it is not always easy to quantify, it is likely to be reflected in a

m easure such as additional output, or growth in em p lo ym en t or growth in sales,

exports, profits or assets th a t can be attrib u ted to th e existence o f th e policy. In

o th e r words activity th a t would not have taken place w ith o u t th e policy program m e

but is a ttrib u tab le to th e firm participating in th e program m e. Figure 2.4 (Oldsman,

2002) is a simplified view o f th e process. It shows, fo r any given outcom e, th a t policy

im pact can be considered as th e difference betw een th e observed outcom e w ith the

intervention and w h a t would have happened w ith o u t th e intervention

-th e counterfactual. Figure 2.4 shows -these tw o outcom es diverging after -the tim e

w hen th e policy is im plem ented.

This illustration is most applicable to 'Hard' support program m es (financial support)

but is clearly m ore challenging to measure th e effects o f 'soft' supports such as

consultancy, training, m anagem ent developm ent etc. The diagram also does not

show th a t th e state intervention can also result in a negative outcom e fo r th e firm or

cohort o f firms in a program m e.

Figure 2 .4 The Impact of an Intervention

Outcome

Observed outcome with intervention

Impact

What would have happened without intervention

Time

Source: Oldsm an (2 0 0 2 )

Storey (2000) developed an approach to try an overcom e th e lim itations o f the

evaluation m ethodologies o f th a t tim e. His six stage approach attem pts to evaluate

th e im pact o f specific program m es on firm behaviour and perform ance. This

approach was subsequently adopted by th e OECD (2004) and has now become the

recom m ended approach o f th e OECD (2008). W hilst recom m ended as 'best practice'

by th e OECD, no exem plar case or empirical studies has yet been published to

dem o n strate th e appropriateness o f th e approach. Its usefulness remains an open

question until such tim e as an em pirical research base has developed. It also appears

to be m ore appropriate fo r estim ating th e shorter te rm effects o f programmes.

Rigorous evaluation seeks, by some means, to contrast the views or actions with

those o f non-recipients in order to present th e counter fa c tu a l (W hite, 2010). The

difference betw een actual changes and th e counter factual is viewed as th e im pact of

th e policy (See Figure 2.4 above). The OECD recom m ended approach is applicable to

discrete policy instrum ents only and does not measure th e effects o f m ultiple

instrum ents or take an econom y-w ide approach. Im pact evaluation, as discussed

here, is inappropriate for applying to the broader m acro-level research questions

which are typically addressed by statistical or macro - economic m odelling (M o rto n ,

2009).

Storey's approach to evaluating m icro-interventions introduced th e notion o f using

mixed research m ethods in evaluation. Nevertheless Curran (2000) highlights the

m ethodological problems encountered w hen attem p tin g to evaluate SME small

business policies and support through this m ethodology. On a broader level th e net

contribution o f th e policy or program m e has to be offset ultim ately against

deadw eight (defined as desired outcom es which would have resulted even if the

policy or program m e had never been initiated) and displacem ent (defined as th e

result o f th e policy or program m e w here o th er firms not involved cease to trad e or

have low er sales or em ploym ent or suffer higher costs) (ibid).

Curran claims th a t th e most appropriate evaluation design to control fo r these

myriad external influences on SME program m es are versions of th e 'm atched control

sam ple' approach to establish th e counterfactual - essentially a tru e experim ental

approach (W h ite, 2009). Storey (1998) how ever does acknowledge th e difficulty of

'm atching' firms given th e myriad factors to consider in relation to th e characteristics

o f th e firm , th e characteristics o f th e en tre p re n e u r/m a n a g e m e n t/o w n e rs h ip , the

nature o f th e business strategy and th e external environm ental factors facing the

firm ( Smallbone & W yer, 2012). Even firms in th e same sector and locality may serve

very d ifferen t m arkets (Curran & Blackburn, 1994). The m ethodological problems

above are com pounded by issues around sample fram ing and response errors and

selection bias. Valid comparison betw een assisted firms and o th er firms can be

affected by adm inistrative selection, self-selection or m oral hazard (Storey, 1988;

Bennett, 1997). Taken to g e th e r the problems o f sampling, response bias, self­

selection and establishing control samples, make rigorous im pact evaluation

extrem ely difficult (Curran, 2000).

The lim itations o f quantitative experim ental techniques alone can be offset to some

extent by using qualitative techniques such as depth interviews, focus groups and

case studies. In addition to adding richness and depth to th e overall findings, these

m ethodologies can also help identify th e firms behavioural and organisational

changes attrib u ted to th e program m e or policy. This approach would appear, as far

as possible, to o ffer a m ethodological approach to th e m easurem ent o f th e strategic,

operational (increased sales, profits, em ploym ent) and behavioural changes (change

in organisational cognitive and absorptive capacities) postulated to happen in Figure

In document Indigenous Firm Performance in a Small Late Developing State : A Case-study of the Role and Contribution of Public Venture Capital in Ireland. (Page 76-83)