Comparative analysis

Covered measures by the evaluation methods

The studied evaluation methods in chapter 3 refer to individual measures of axis 1, 2, and 4 of the EU rural development policy or to Rural Development Pro grammes. Evaluation of axis 3 measures is usually part of the evaluation of the whole Rural Development Programme. In addition, evaluation methods cover the CAP, EU Structural Policy, the Nordic Aid Scheme, Federal Policy Programmes in the US and national environmental and nature management measures.

95

Table 4.1 Classification of evaluation methods according to their methodological approach

Approach Method

Evaluation of the EU Rural Development Programmes 20072013 Evaluation of the Rural Development Programme 20002006 (the Netherlands)

Evaluation of the Rural Development Programme 20002006 (Flanders) Expost evaluation of LEADER II programmes 19941999

Midterm evaluation LEADER+ (20002006) in the Netherlands Evaluation of Cohesion policy programmes in Objective 1 and 2 regions

Evaluation of the Less Favoured Area measure in the EU25 Evaluation of the setaside measure

1.CMEF type

approach (indicators and evaluation questions) (9)

Evaluation of the Nordic Aid schemes Evaluation of the LFA policy in Austria

Evaluation of the Dutch national policy for management of wintering goose populations

Evaluation of the nature management measures in the Netherlands Evaluation of the greenhouse horticulture policy in the Netherlands 2.Tally

approach (5)

Programme Assessment Rating Tool (PART)

Nonparametric propensity score matching approach for evaluating agrienvironmental and LFA measures

3.Econometric

approach (2)

Regression model on farm meadow birds Interregional Social Accounting Matrix (SAM) Regional Social Accounting Matrix (SAM)

4.Modelling

approach (3)

LEITAP

Mixedmethod case study

Evaluation of the measure for setting up of young farmers in the Netherlands

5.Mixed case study

approach (3)

Evaluation of the territorial environmental policy in the Netherlands Time period and location

The methods are usually applied for a time period varying from two to seven years. However, the PART evaluation of the US Federal Policy Programmes is a continuous assessment in each calendar year, whereas the evaluation of the farm meadow bird policy in the Netherlands covered a 12 year period and that

96

of the EUwide evaluation of the LFA policy about a 30 year period. The evalu ated location refers to the EU in five studied methods and to the Netherlands in eight methods. Individual EU15 Member States or groups of Member States constitute the evaluated location in the other methods.

Ex ante, midterm and ex post evaluation

The three evaluation methods with a modelling approach were used for ex ante evaluation whereas the US PART method was used to support ongoing evalua tion. Fourteen evaluation methods were used for ex post evaluation and three for midterm evaluation. However, like the CMEF framework for the evaluation of the RDP 20072013, these methods can be applied for all types of evaluation during the policy cycle: ex ante, midterm and ex post evaluation.

Overall design of the evaluation methods

The classification of evaluation methods (Table 4.1) reflects the various types of designs of the evaluation methods. A CMEF type approach, consisting of a combination of a hierarchy of indicators and preponderating qualitative evalua tion questions, is nowadays the officially prescribed approach for the evaluation of common EU policies like the CAP and the Cohesion Policy. A pure quantitative approach for analysing whether quantitative objectives have been achieved is applied in the evaluation methods using tally, econometrics and modelling. A combination of quantitative and qualitative analysis is applied in the mixed casestudy approach, which aims to assess both direct and indirect results of the policy intervention, often in case study areas. Although there is some simi larity between the methods in the CMEF type approach and the mixed case study approach, the methods in the latter group show much more flexibility in the selection of indicators and questions than the CMEF type approach meth ods, which uses a prescribed list of indicators and common questions.

Intervention logic

The intervention logic describes the logical link between the policy intervention and its results. Evaluation methods grouped in the CMEF type approach broadly use small variations of the intervention logic used in the evaluation of the EU Rural Development Policy 20072013 (Figure 4.1). The intervention logic starts with identifying the needs, which describe the socioeconomic or environmental requirements to which the measure or programme should respond. Then the policy response is developed through a 'hierarchy of objectives', from general to specific to operational objectives. The operational objectives constitute the

97

Figure 4.1 Intervention logic of the Common Monitoring and Evaluation Framework (CMEF)

Source: OECD (2009b).

relevant goals and determine the rules of implementation, i.e. the input (financial resources) at farm or firm level. These inputs will generate a chain of outputs, results (immediate effects) and impacts (contribution to the achievement of the overall objectives of the programme).

The methods classified in the tally approach do not explicitly describe an intervention logic, but they assume implicitly that the policy intervention contrib utes to the achievement of the policy objective(s). Sometimes, it is acknowl edged that external influences might affect the objective as well. Methods in the econometric approach do also not employ an intervention logic; they rather aim

98

to explore the impact of participating in a measure by comparing the develop ment of indicators of participants and nonparticipants.

In the Social Accounting Matrices, belonging to the modelling approach, pol icy interventions are transmitted as a monetary injection into the matrix repre senting the economy. As a next step, the diffusion pattern of this injection in terms of generated output, firm and household income and employment is ana lysed. The intervention logic in LEITAP operates in a similar way: in this model, the objective of each measure is linked to a driving variable of the model. The impact is determined by comparing the value of the driving variable before and after the policy intervention.

Two of the methods classified in the mixed casestudy approach do not ex plicitly describe an intervention logic between the policy intervention and its im pact. The overall idea of these methods is rather to explore the impact of the intervention, taking into account the complexity of the context in which it is ap plied. The evaluation method applied on the territorial environmental policy, the third method in the mixed casestudy approach employs a policy impact chain as intervention logic, in which the financial input of the policy measure results in projects contributing to the improvement of the environmental quality.

Measurement of the impact

For assessing whether the policy objectives have been achieved, the methods in the CMEF type approach analyse data and answer the evaluation questions. In the methods grouped into the tally approach and the econometric approach, the impact of the policy intervention is measured by data analysis, whereas in the methods in the modelling approach the models where used. Methods in the mixed casestudy approach follow different ways for measuring the impact: quantitative and qualitative analysis of how the policy intervention interacts with the underlying features of the rural economy (mixedmethod case study), interviews and surveys (evaluation of the setting up of young farmers) and data analysis (evaluation of the territorial environmental policy).

Direct and indirect results of the policy intervention

Evaluation methods can be directed at either assessing direct results of the pol icy intervention or can also use a more comprehensive approach by considering indirect results of the intervention as well. It appears that evaluation methods in the CMEF type approach consider both direct and indirect results of the policy intervention, whereas evaluation methods in the tally and econometric approach only focus at direct results. Methods in the modelling approach take only ac

99 count of the indirect results, as the models are not able to calculate direct re

sults. Two methods in the mixed casestudy approach look at both direct and indirect results, whereas the evaluation method of the setting up of young farm ers takes direct results into account.

Data needed for the evaluation methods

Evaluation methods in the CMEF type approach need data for the used indica tors and additional qualitative information for answering those evaluation ques tions which cannot be addressed by the indicators. Evaluation methods in the tally and econometric approach use farm accounting, farm structure and envi ronmental data, and sometimes data from monitoring databases. Evaluation methods in the modelling approach are the most data demanding: the Social Accounting Matrices need data on all economic transactions in the studied ar eas, data on the amounts of funds allocated to the policy instrument(s) and its distribution over the different users within the economy, whereas LEITAP needs data on many economic variables in order to describe the whole world econ omy. Data needs for the evaluation methods in the mixed casestudy approach vary: the mixedmethod case study needs data on the case study region and data and qualitative information on the local economy, the evaluation method for setting up young farmers needs farm accounting and farm structure data and in formation on investment behaviour, whereas the evaluation method of the terri torial environmental policy needs environmental data and data from the monitoring system.

On the whole, most data for the evaluation methods could be derived from existing data sources and monitoring systems. For the evaluation methods in the CMEF type approach, the mixedmethod case study and the evaluation method for setting up young farmers, additional data collection by means of surveys, interviews and workshops is needed, especially for answering the more qualitative evaluation questions. In the evaluation method of nature management measures field work in a sample of meadows is needed. The evaluation method of the territorial environmental policy is the only method discussed in this report which  according to our interpretation  experienced difficulties due to lack of data.

Strengths and weaknesses

For assessing the strengths and weaknesses of the evaluation methods, we have listed main properties of evaluation methods and explored whether these properties are applicable to the five groups of methods (Table 4.2). It appears

100

that a striking difference can be revealed between the evaluation methods in the CMEF type, tally, econometric and modelling approach on the one hand, and those in the mixed casestudy approach on the other hand: methods in the first four groups in particular identify absolute effects of the policy intervention, whereas the mixed casestudy approach tends to focus on the relative effects and features in the context of the policy intervention.

On the whole, the impact of the policy intervention is measured at the ap propriate territorial level for the methods in the tally, econometric and mixed casestudy approach. This is only partly the case for methods in the CMEF type and modelling approach. Methods in the CMEF type approach tend to measure the impact at regional or national level, which might be satisfying as far as terri torial policies are evaluated. However, when sectoral polices are evaluated, it should be preferred to measure the impact at farm or local level, as the impact of such measures is often only felt locally, and fades away in the total amount of actions at the regional or national level. Within the modelling approach, SAMs are able to capture the impact at the right level, whereas LEITAP is only able to identify impacts at the relatively high national level. Methods in the CMEF type, tally and econometric approach and LEITAP can be applied to the whole EU ter ritory, whereas SAMs and methods in the mixed casestudy approach are re stricted to case study areas.

Considering the amount of needed data for the evaluation method and the way in which these data have to be processed and analysed, it could be noted that the methods in the tally and mixed casestudy approach are rather easy to apply for evaluators, whereas methods in the econometric and modelling ap proach require specific skills of the evaluator and methods in the CMEF type approach are rather time consuming due to its huge number of indicators and evaluation questions.

101

Table 4.2 Assessment of main properties of the evaluation methods

C M E F t y p e a p p ro a c h T a ll y a p p ro a c h E c o n o m e tr ic a p p ro a c h M o d e ll in g a p 2 p ro a c h M ix e d c a se st u d y a p p ro a c h

Diagnosis of cause and effect:

 description of what has happened

(in quantitative terms)

x x x x

 description of what has happened

(in qualitative terms)

x

 description of how and why it has hap

pened in interaction with the local context and other policies

x

 impact is measured at the right territorial

level

partly x x partly x

Indirect results of policy intervention are taken into account

x x x x

Unintended effects of the policy intervention are taken into account

x Reveals reasons why actors participate in

a policy measure

x Covers the whole territory in which measure

is applied

x x x partly

Easy to apply for evaluator x x

Extension to EU and Rural Development Programme level

The evaluation methods in the CMEF type approach and LEITAP are designed for use at EU level. All other discussed evaluation methods are designed for use at national level or for a group of countries, but the use of these methods can also be extended to other EU Member States. Some of the evaluation methods are designed for the evaluation of programmes, whereas others are designed for individual measures. It appears that the evaluation methods designed for individual policy measures cannot be applied for other rural development meas ures, as these methods tend to be measurespecific. There are a few excep tions: the method of interviews and a survey in the evaluation of the setting up of young farmers could be extended to other rural development measures if the

102

questions in the interviews and survey are adapted; the nonparametric propen sity score matching approach for evaluating agrienvironmental and LFA meas ures can also be applied to rural development measures, which try to affect input use at individual farms, such as measures under axis 1 directed at improv ing human and physical capital.

103