Selecting Appropriate Indicators

An important step in any impact evaluation after deciding on a respective research design is to – possibly with the help of programme staff and key stakeholders – think through the possible outcomes of the programme to be evaluated. In order to determine how effective a programme is, one needs to have some idea on how well outcomes are being achieved and on how, i.e. with which indicator, to measure them. The concept of indicators has already been introduced in section 4.2.1. While monitoring indicators assess the progress of a programme towards the achievement of its objectives, indicators for impact evaluation may be described as a measur- able approximation of the effects one is attempting to achieve. They translate general concepts regarding the intervention and its expected effects into specific measures that can be inter- preted and that usually address criteria that will be used to judge the intervention.

Most important for assessing a programme’s effects are result and impact indicators. While

result indicators measure the most immediate impact for the direct addressee of a programme,

impact indicators represent the consequences of the programme beyond its direct and imme- diate interaction with the addressees, i.e. the socio-economic changes which can be attributed to the programme (see also section 4.2.1 for a definition of the different types of indicators). The indicators to be selected for each outcome depend on the evaluation team’s perspective about what is the most accurate measure of the stated outcomes and on the resources available for data collection.

The starting point for selecting appropriate indicators for evaluating the effects of programmes targeted at the promotion of female entrepreneurship are the general evaluation criteria to be applied in the evaluation (see section 3.1.3). After the evaluation criteria have been translated into specificevaluation questions that specifically relate to the programme under investigation (see section 3.1.4), each evaluation question has to be made operational; i.e. measurable by defining corresponding indicators and selecting the according data collection methods (see section 4.3.3 for an overview of different techniques for data collection). Ideally the result and impact indicators selected for evaluating a programme’s effects logically relate to the output indicators established for monitoring the programme’s progress in order to allow causality analy- sis as demonstrated in Box 9.

Box 9: The relation between output, result and impact indicators

result

number of female entrepreneurs who have

gained a qualification

output

number of training courses financed for female entrepreneurs

impact

survival rate enterprises run by women who par- ticipated in the training

result

number of female entrepreneurs who have

gained a qualification

output

number of training courses financed for female entrepreneurs

impact

survival rate enterprises run by women who par- ticipated in the training

With reference to the causality chain illustrated in Box 9, training for female entrepreneurs leads to higher qualification levels among women participating in the training course and in conse- quence to a higher survival rate among the enterprises owned by these women. However, there may well be found a growing number of female entrepreneurs taking training courses in the monitoring data, but if these courses do not correspond to the needs of these women, the inter- vention may not contribute to higher entrepreneurial skill levels among female entrepreneurs at the result level and, thus, to a higher survival rate of enterprises run by women at the impact level. This would imply that the hidden assumptions of the programme’s planners about the causal links between the programme and its supposed effects, i.e. the intervention logic of the programme (see section 4.1.2), cannot be verified. Table 2 in section 4.2.1 provides an example of a set of result and impact indicators for evaluating different types of measures promoting female entrepreneurship interrelating to the respective output indicators.

However, particularly when evaluating programmes targeted at the promotion of female entre- preneurship, individual client effects should be determined in addition to the socio-economic effects of a programme tackled in Table 2. Often the objectives of a programme may be ex- pressed in terms of service delivery or system objectives only (e.g. reducing inequalities bet- ween male and female entrepreneurs), rather than in terms of individual client objectives, stating how clients’ lives will improve as a result of the programme (e.g. gaining life and business skills necessary to be self-reliant and economically independent). Yet, when thinking about the purpose of social and human services programmes, it has to be realised that one of the most important outcomes is the effect on individuals/participants. Evaluating individual client effects amounts to answering the question:

What difference does the programme make in the lives of those served?

The types of effects one may attempt to achieve with programmes promoting female entrepre- neurship at the individual client level include changes in circumstances, status, quality of life, attitude or behaviour, knowledge, skills, etc. These types of effects should be analysed in the scope of all impact evaluations of programmes targeted at the promotion of female entrepre- neurship - although they might not be stated as explicit objectives of the intervention - and should therefore be considered, when elaborating corresponding evaluation indicators. Further- more, certain programmes often produce outcomes that were not listed as objectives when the programme was launched. Impact evaluation should also attempt to discover these unanticipa- ted or unexpected effects by elaborating according indicators.