Copyright 2011 by Bernd Schuh, Robert Lukesch, Jerzy Michalek, Peter Kaufmann, Andrea Pufahl, Simone Schiller, Pille Koorberg, Guy Beaufoy, Gilles Pinay, Dominique Moran, Horst
METHODOLOGICAL AND EMPIRICAL CHALLENGES OF POLICY EVALUATION"
Ancona, February 17-18, 2011
Approaches for assessing the impacts of the Rural
Development Programmes in the context of multiple
intervening factors
Schuh B.1, Lukesch R.2,Michalek J., Kaufmann P., Pufahl A., Schiller S., Koorberg P., Beaufoy G., Pinay G., Moran D., Gomann H., Storti D., Rossi P. and Paracchini M.L.
1 European Evaluation Network for Rural Development, Brussels, Belgium 2 European Evaluation Network for Rural Development, Brussels, Belgium
Approaches for assessing the impacts of the Rural
Development Programmes in the context of multiple
intervening factors
Schuh B., Lukesch R.,Michalek J., Kaufmann P., Pufahl A., Schiller S., Koorberg P., Beaufoy G., Pinay G., Moran D., Gomann H., Storti D., Rossi P. and Paracchini M.L.
Abstract
The Common Monitoring and Evaluation Framework (CMEF) provides a single framework for monitoring and evaluation of all EU Rural Development Programmes (RDP) in the current programming period (2007-2013). It provides continuity from previous periods and constitutes a significant simplification as regards assessment of results and impacts, while at the same time offering greater flexibility to Member States.
The European Evaluation Network for Rural Development has published a Working Paper on Approaches for assessing the impacts of the Rural Development Programmes in the context of multiple intervening factors. The aim of the Working Paper is to inspire and to encourage programme evaluators, not to restrict or constrain them.
From a methodological perspective, the three common socio-economic impact indicators of the CMEF (economic growth, employment creation, labour productivity) are more closely related than the four common environmental impact indicators (reversing biodiversity decline, maintenance of High Nature Value faming and forestry, improvement in water quality, contribution to combating climate change).
Keywords: assessment of impacts, Rural Development Programmes, policy evaluation, EU policy
JEL classification: Enter JEL codes.
1. INTRODUCTION
For the current 2007-2013 programming period, the Common Monitoring and Evaluation Framework (CMEF) requires Member States (MS) to assess the impacts of their rural development programmes (RDPs) during two main evaluation milestones – mid-term (MTE) in 2010 and ex-post in 2015. As MS have reported difficulties in identifying the impacts attributable to specific RDP measures particularly where there are multiple intervening factors, the Evaluation Network has produced a new guidance document on how to assess such impacts, focusing to a large extent on measurement and interpretation of indicators.
The challenges in establishing the intervention logic are considerable for environmental impacts (CMEF common impact indicators 4-7), as such impacts are strongly influenced by site-specific circumstances (e.g. soil, temperature, rainfall). Significantly, both environmental and socio-economic impacts (reflected by the common impact indicators 1-3) may take a long time to emerge and may depend on other intervening factors (e.g. national/regional policies, implementation mechanisms).
2. THE GUIDANCE DOCUMENT, OUTCOME OF AN ITERATIVE PROCESS
The guidance document is the end result of a thematic working group (TWG) of the Evaluation Expert Network, which consisted of two sub-groups (socio-economic and environmental indicators). Two core team members1 of the Evaluation Helpdesk, coordinated the work of a group of 12 European associated experts2 between May and December 2009. Two workshops were organised at the Helpdesk premises in Brussels, and a further two workshops were hosted by Member States during October 2009 (in Rome for socio-economic indicators and Vienna for environmental indicators). The draft and final documents were presented to representations from Member States at the Evaluation Expert Committee meetings in Brussels.
The guidance document starts with an introductory section recalling the main requirements of the CMEF. It then sets out the core process and methodological challenges of assessing RDP impacts, and finally tackles the assessment of socio-economic and of environmental impacts in separate chapters. A comprehensive bibliography completes the document.
3. THREE STAGES AND TWO BASIC QUESTIONS
The purpose of the document is to inspire and to encourage the evaluators of RDPs, not to restrict or constrain them. There is no “standard method” to be followed mechanically. The assortment of recommended methods and pathways reflects two underlying principles: to strive for optimal evidence and to appreciate the complexity and uniqueness of rural societies, economies and places.
Assessing impacts must not be understood as merely measuring indicator values. Therefore the document proposes a three-stage process. The first stage consists in gathering the information and data from various sources to build up a body of evidence to gauge change. From this evidence, in the second stage, answers to the common evaluation questions and programme-specific additional evaluation questions can be given. These answers always address two basic questions:
• Has there been change which can be traced back to the causal influence of rural development measures?
• How and by which concatenation of circumstances did change happen?
1
Robert Lukesch for the three socio-economic indicators: Economic growth, Employment and Labour Productivity; Bernd Schuh for the four environmental indicators: Reversing Biodiversity Decline; Maintenance of HNV farming and forestry; Improvement in Water Quality; Combating Climate Change.
2
G. Beaufoy and M.L. Paracchini (HNV); H. Gomann (Water quality); P. Kaufmann and A. Pufahl (Employment creation); P. Koorberg and P. Rossi (Biodiversity); J. Michalek (Economic growth); D. Moran (Climate change); G. Pinay (Water quality); S. Schiller and D. Storti (Labour productivity).
Other information (statistics, surveys/inquiries…) Additional Programme-specific Indicators Trends of baseline indicators
Output & Results Indicators Programme-specific Evaluation Questions Evaluation Results Body of evidence Common Evaluation Questions (per Axis and horizontal)
Policy Recommendations
Common Impact Indicators
Gauging the evidence of change
Identifying the drivers of change
Understanding change and concluding on future
interventions
Reflection space
Source: European Evaluation Network for Rural Development, 2010
The answer to the first question is crucial for the control function of evaluation to explore how far public authorities have invested taxpayers’ money in meaningful and effective ways. The answer to the second question allows us to look into the “black box of rural development”. It gives insights into how we can repeat successful interventions and improve them further.
These answers are then compiled into the third stage of evaluation, which feeds directly into policy recommendations.
4. KEY CHALLENGES FOR IMPACT ASSESSMENT
Evaluation should reduce the level of uncertainty sufficiently enough to inform responsible and accountable political action. This means tackling the following key questions and challenges:
• What would have happened to the respective programme area without a given programme? This challenging question implies the requirement to assess, wherever possible, the programme impacts against their counterfactual, i.e. calculating the changes that would have occurred without the specific programme intervention.
• To disentangle the effects of single measures or the programme as a whole from effects of other intervening factors. This challenge implies the requirement to measure both the micro and the macro level effects and to meaningfully combine the results into one picture. It also implies netting out deadweight, substitution and multiplier effects;
• To ensure the availability and validity of data and information required to construct a viable body of evidence: This challenge implies the requirement to construct a data and information base which allows for the unbiased computation of the effects as stipulated above.
• To put the partial results in a meaningful relation with the overall rural development programme and the overall policy context to be able to provide pertinent answers to the evaluation questions.
5. THE SOCIO-ECONOMIC INDICATORS
From a methodological point of view, the three socio-economic indicators have more in common than the environmental indicators. For all three indicators the document recommends using propensity score matching (PSM) for constructing the control group. It also suggests calculating the difference in differences (DiD), i.e. the combined comparison between two points in time (before-after) and between beneficiaries and non-beneficiaries.
After having estimated the direct effects on programme beneficiaries, the indirect effects can be measured by appropriate methods to discount deadweight, substitution and displacement effects, and to take into account leverage and multiplier effects. The latter requires considering and cross-relating impacts at micro and macro level for which modelling or econometric methods – or combinations of both – are in use. The database is mainly founded on the FADN and complementary national databases concerning farms, whereas data on enterprises and communities will in many cases have to be drawn by specific surveys.
An overview of current practice shows that for Axis 1 measures the recommended methods can be more easily applied than for Axis 3 and 4 measures, due to low-uptake, or project-type (and therefore quite singular) measures, or site or community specific particularities which makes it difficult to establish control groups or sites. However, the potentials of the counterfactual approach are still not exploited in full, and the confirmation of impacts on the basis of mere before-after comparisons should be taken with more scepticism than hitherto.
Taking these obstacles into account, case studies (such as comparative cost-effectiveness analyses) should be integrated. In many cases, surveys based on interviews of beneficiaries, are required.
5.1. Impact Indicator 1: Economic Growth
According to the CMEF the impact of a RD programme on economic growth is to be measured in terms of the Net Additional Gross Value Added in purchasing power standard: NAGVA-PPS.
The indicator on economic growth should not be taken as a proxy for sectoral competitiveness, rural diversification or quality of life. To make conclusions of this kind, requires looking at a set of additional indicators at aggregated level (e.g. output shares) or indices (e.g. rural development index).
5.2. Impact Indicator 2: Employment Creation
The CMEF suggests measuring employment effects in Full Time Equivalent (FTE) jobs created, expressed as the number of additional jobs created directly in supported projects and indirectly in the programme area and lasting for at least 10 years.
Apart from calculating employment effect on farms, a mix of qualitative and quantitative methods should be applied to cover possible effects on employment outside of agriculture and to get insights not only on the magnitude of the effect but also on how RD policies affect individuals, communities or regions.
The guidance document also mentions some caveats for interpreting the outcomes, e.g. the time lag until an investment brings forth lasting employment, missing critical mass (especially for non-agricultural beneficiaries) or displacement effects between regions. Furthermore, employment effects should be interpreted in a common context. For instance, rising total factor productivity (labour, capital, land) may explain why jobs have been lost in the agricultural sector.
5.3. Indicator 3: Labour productivity
The CMEF defines labour productivity as the change in Gross Value Added per Full Time Equivalent (GVA/FTE). GVA is defined as value of output less the value of intermediate consumption; the definition of FTE is equal to that of indicator 2.
The indicator is intra-sectoral and does therefore not express the competitiveness of one sector against another. The indicator also does not allow side effects to be taken into consideration, for instance if funding is provided to companies whose improved performance makes no direct contribution to rural development. In order to overcome the limitations of the GVA/FTE indicator the competitiveness of the agricultural sector can be measured in alternative ways, such as Competitive Performance or Revealed Comparative Advantage (RCA).
6. ENVIRONMENTAL INDICATORS
The complexity of the environmental indicators requires focusing on various approaches, trends of baseline indicators and place-specific additional indicators.
In comparison to the assessment of socio-economic impacts, assessing the impact of RDPs on the environment poses a number of intrinsic methodological challenges, among which:
• The sub-sequential effects of rural development measures: firstly on the behaviour and management practices of farmers and forest holders, and, secondly, in terms of impact on the environment due to the changed farming/forestry practices;
• Impacts are often depending on site-specific circumstances, such as soil, temperature, rainfall etc. As a consequence, linking the results of on-site observations to overall conclusions at the level of the programme area is not a straightforward task;
• Impacts may take a long time to emerge. Therefore the assessment should preferably make use of long-time series data, where these are available;
• Due to complexity and site specific impacts of RDPs on the environment, the identification of control groups and the establishment of a situation with and without the programme in place (counterfactual situation) are particularly difficult;
• In the context of Rural Development Programmes, a broad range of measures, from different axes, may affect the environmental conditions of a given programme area;
• It is often difficult to establish cause-effects relationships for environmental impacts. However the methods for constructing control groups (PSM) as well as DiD can also potentially be applied for the environmental indicators. Nevertheless, data availability is still an important issue.
6.1. Indicator 4: Reversing Biodiversity Decline
The CMEF defines this impact indicator as change in trend (biodiversity decline) in the area targeted by the intervention. Farmland bird species population is an indicator of general biodiversity trends for which the best data exists in terms of time series and geographic distribution. The farmland bird species population trends are measured with the multi-species Farmland Bird Index (FBI). The FBI has been adopted as an EU Structural Indicator and a Sustainable Development Indicator.
With regard to their local conditions, Member States and regions may need to use an alternative composition of bird species where this is appropriate and may also choose a different reference year. The FBI can also be complemented by other existing indicators such as population trends of agriculture related butterfly species, or trends in important bird areas (IBAs) considered as threatened by agricultural intensification, under-utilisation of land or abandonment.
At present the data for the calculation of FBI originates from national monitoring of widespread birds collected and compiled by the Pan-European Common Bird Monitoring Scheme (PECBMS) in cooperation with Statistics Netherlands. To guarantee a high quality FBI at the national level it is necessary to have an appropriate monitoring scheme covering representative amounts of farmland. Lack of monitoring data and finances to carry out special studies may lead to unadequate or misleading evaluation results. In any event, a variety of sources of information will have to be taken into account in order to understand what is going on in the area-specific context.
6.2. Impact indicator 5: Maintaining of HNV farming and forestry
The CMEF defines the impact indicator as changes (UAA ha) in High Nature Value farmland and forestry. HNV farmland refers to farmland characterised by the presence of particular land cover types and patterns which indicate that this farmland is valuable for nature conservation. The presence of populations of particular wildlife species may also provide this indication. The denomination refers to both the land cover (farmland or forest) and the way it is managed for production by a particular farming system and practices.
The evaluation of the impact of RD measures examines intended and unintended influences of RD measures on farmers’ decisions, the extent of participation, the coincidence of participation with the observed changes and the distinction of programme-induced changes from those induced by other factors (climate, commodity prices, etc.).
HNV criteria (farming or forestry practices) may be combined in a points system to allocate an HNV score for a given unit of land, such as the farm holding or the municipality, on the basis of the characteristics that are present. Data may also be translated into maps to visualise the territorial distribution of HNV characteristics.
As in many Member States the baseline (number of HNV hectares) has not been sufficiently established, it is important to complement the quantified estimate with qualitative assessment, for example through multi-disciplinary studies.
Sample surveys of areas with a concentration of HNV farming and forestry systems will allow for a far more rigorous assessment of programme impacts. HNV sample surveys should aim at monitoring trends in key farming/forestry practices and the condition of land cover, species populations, as well as the socio-economic situation of HNV farming/forestry holdings.
Only an investment in appropriate data collection and monitoring schemes will ultimately allow a full evaluation of the effects of rural development programmes on HNV farming and forestry.
6.3. Impact indicator 6: Improvement in Water Quality
The CMEF defines this impact indicator as estimated changes in gross nutrient balance (GNB) attributable to the intervention. The GNB indicates potential nutrient losses to the water bodies likely to be detrimental for the quality of water. The GNB includes all residual nutrient emissions of environmentally harmful compounds from agriculture.
The farm represents the micro unit of measurement. Several methods have been developed for assessing a farm nutrient budget, either based on an aggregate of individual fields or on an analysis of the farm as a whole. The latter is more recommendable since it takes into account transfers of matter between fields and farming practices. The most appropriate method to determine the impact of RD measures on the change in GNB is calculating the difference in differences.
The macro level of analysis is the farming region, identified as the geographic entity with similar geological, pedologic, climatic and social features. Several models (e.g. CAPRI, RAUMIS) have been developed to estimate soil gross or net nutrient balance at aggregated regional levels in Europe.
Besides the farm structure survey (FSS), and IACS data land cover can be determined due to the development of remote sensing devices, digital elevation models and GIS software. All European countries have access to these tools and are currently using them.
6.4. Impact Indicator 7: Contribution to Combating Climate Change
The CMEF defines this indicator as quantitative and qualitative change in the production of renewable energy, measured in units of ktoe (kilotonnes of oil equivalent). The indicator shows the reduction of net greenhouse gas emissions (i.e. carbon dioxide) attributable to the substitution of fossil fuels by non fossil alternatives such as bioenergy crops, perennial grassland, short rotation forests on agricultural land, afforestation, residues or biowaste, wind and hydropower capacity. The data availability is relatively good: FADN provides farm scale data on land under specific crops, and there is also information on installed capacity (biomass, biogas and wind).
For fuel crop areas (and hence kTOe or CO2e), the macro picture can be developed
bottom-up (based on qualitative surveys of a cross section sample of recipients) or top-down (based on representative modelling of a range of farm types using linear or dynamic programming methods), at best combined.
The outcomes of climate change, water quality and HNV indicators need to be considered altogether to derive a net picture of combined impact. Thus, targeting nitrogen in pursuit of water quality has inevitable impacts in terms of simultaneous reductions in atmospheric emissions and vice versa. Similarly, increased biomass and biofuel cropping will have implications for water demand, biodiversity outcomes and potentially food security. Similarly, policies on ammonia reduction (principally for human health impacts) will also be relevant.
However this information does not yet fully reflect all impacts of RD programme interventions in terms of combating climate change. In order to assess impacts at the programme level, all measures (i.e. also from Axes 1 and 3) have to be considered.
7. ADDITIONAL IMPACT INDICATORS AS USED BY THE EUMEMBER STATES:
Additional/programme specific impact indicators form a crucial part of the “body of evidence” to be collected within the process of assessing impacts of RDPs. They provide important opportunity for the programme evaluators to include programme specific circumstances, including aspects of implementation and additional objectives, into the assessment of RDP impacts and they help significantly to bridge the gap between interpretation of the programme results and the judgement on the overall programme impact in the specific programming areas. The selection and application of additional impact indicators is therefore a crucial element of setting the evaluation frame of RDPs.
The MS had to develop and include programme specific indicators from the outset of programme development, and the ex-ante evaluations of RDPs had to provide a list of these indicators together with the expected target levels.
An analysis of the additional impact indicators as listed in the ex-ante evaluations of the RDPs 2007-2013 (see Synthesis of ex-ante Evaluations of EU RD Programmes – Annex 7) showed 718 additional specific impact indicators, identified by the MS. However only about a
third of the additional indicators listed (i.e. 210) were real additional impact indicators in the MS.
This fact, that from a total of more than 700 additional impact indicators listed by the MS in the ex-ante evaluations only less than a third are to be classified as real additional impact indicators according to the CMEF criteria shows the challenge for MS how to deal with this instrument for assessment of impacts.
7.1. Economic Growth
In total there were 19 additional impact indicators listed by MS in this field most of them were simply providing deviating definitions of the common impact indicator – i.e. net additional value added expressed in PPS.
In some RD programmes the role of additional indicators was interpreted as enlargement of the scope of measures contributing to a given impact: This means that the common impact indicator has been used, but for an enlarged set of measures than stipulated by the CMEF. Only Northern Ireland introduced genuine new impact indicators in the economic growth field, i.e.:
Number of participants indicating the measure had a financially positive effect on their farm business; Number of new businesses which are still in existence two years after final funding; Number of supported new businesses which are still in existence two years after final funding.
7.2. Employment Creation
In total 20 additional impact indicators have been listed by MS for employment creation. More than half of them have used deviating definitions of the impact indicator according to the CMEF – i.e. net additional full-time equivalent jobs created. In most of the cases the MS deemed a split up of job creation by economic sectors more appropriate (e.g. Emilia Romagna, Calabria). In some cases the quantification of the impact on employment creation is seen as not feasible (e.g. Hamburg), thus a qualitative assessment is suggested.
Only 2-3 additional indicators in the field have been listed. They were rather implying that instead of the creation of employment, preservation of existing employment should be seen as an impact of the RDPs: preservation of existing jobs, securing jobs for educational professionals/auxiliary forces in child care (Sachsen-Anhalt). One programme listed the additional impact indicator “human capital” (measured by: promotion of competences development) (Niedersachsen and Bremen), which seems an innovative approach in this theme.
7.3. Labour Productivity
There have been no real genuine additional indicators in the field.
10 additional impact indicators were listed by MS in this field. In the vast majority of cases the deviation of indicator definitions from the CMEF definition has lead to the creation of an additional one (e.g. Hamburg, Cataluña). The CMEF defined the assessment of labour productivity as: change in Gross Value Added per full-time equivalent (GVA/FTE). Some of
the MS added indicators depicting the labour productivity divided into economic sectors (e.g. Cataluña, Emilia Romagna, Calabria).
7.4. Reversing Biodiversity Decline
In this thematic field, 17 additional indicators have been listed by MS. In contrast to the socio-economic impact indicators, no deviating definitions of this impact indicator have been used.
The additional indicators in the field show that mostly the enlargement of the focus of biodiversity (including species other than birds) has been the main trend when designing additional indicators.
7.5. Maintenance of HNV farmland and forestry
Taking into account the comparably “new” concept of High Nature Value farming and forestry and its assessment within RDPs, it does not come as surprise that this theme generated quite a number of additional impact indicators by the MS – 32 in total.
The bulk of the indicators listed by MS are stressing that a more thorough break-down of the concept into the territorial setting is needed in order to operationalise the assessment of impacts. The various approaches by the MS, fairly equally representing both the “land-cover approach” and the farming practice approach of HNV.
In many cases HNV farmland is equated with Natura 2000 sites and other nature preservation zones. The German speaking programme areas in particular show a high affinity for HNV with the concept of cultural landscapes. This is due to the fact that a substantial body of research has been conducted in this field in these countries and thus the linking of these concepts is fairly easy.
7.6. Improvement in Water Quality
A total of 28 additional indicators have been identified by the MS for water quality. In the majority of cases the MS followed the tendency to enlarge the scope to assessment aspects of water quality by adding other pollutants. The different approaches in the MSs are ranging from adding measurement of changes in levels of phosphorus and pesticides to ammonium. The Italian programme Molise used an Italian national water quality index as an additional impact indicator. All in all the field of water quality assessment seems to be well covered and the impression is that unlike other themes (see e.g. labour productivity) the overall scope of assessment does not pose too many problems. This impact indicator seems to be a good example of MS using the potential of additional impact indicators in order to sufficiently enlarge the scope of gauging evidence for assessing impacts of RDPs.
7.7. Contribution to Combating Climate Change
Like the other environmental indicator fields the theme of climate change has triggered quite a substantial list of additional/programme specific indicators, as for many MS the focus of
the measurement as stipulated by the CMEF (i.e. increase in production of renewable energy) seems not extensive enough. There have been 28 additional impact indicators listed by the MS.
In most of the cases MS followed the strategy to enlarge the focus of the assessment by adding additional aspects of impacts arriving from the implementation of the RDPs. Mostly the sequestration of greenhouse gases (GHG) by natural sinks is included (e.g. UK-England) but also the energy efficiency as the demand side factor of climate change is taken into account. In some cases the specification of renewable energy production has been attempted (e.g. concentration on biomass).
7.8. New impact indicators not related to any common impact indicators
Aside from the additional indicators in the seven impact categories as listed in the CMEF there have been a number of additional indicators (18 in total), which are genuinely new impact indicators set up by MS. A majority of indicators in this category deal with the need to depict the overall RDP objective to improve the quality of life in rural areas. Eight indicators are set up for this purpose.
Another important objective of RDPs, which needs to be assessed in terms of impacts, is the challenge of migration and population decline. About 5 indicators deal with these aspects. Last but not least, the assessment of impacts derived from the implementation of Axis 4 (LEADER) led to the establishment of some additional indicators.
The following table provides an overview of these indicators, which are genuinely additional indicators without thematic relation.
Table 1: New impact indicators not related to any common impact indicators
Programme Additional Indicator Measurement Thematic Field
Hessen Life quality attractive life environment (life
quality)
quality of life Mecklenburg
Vorpommern
Stabilize the population number no measurement provided demography
Niedersachsen and Bremen
Lifequality and governance living milieu and quality, social
life, local identity: data collection difficult, mostly qualitative data collected (via public consultations)
quality of life
Niedersachsen and Bremen
Lifequality and governance governance - improvement of
regional competencies
quality of life Niedersachsen and
Bremen
Lifequality and governance governance - planned and
implemented plans/proposals
quality of life
Rheinland-Pfalz Attractive living environment no measurement provided quality of life
Saarland Population trends: indicator to assess
the prevention of the migration of population
no measurement provided demography
Sachsen Impacts on safety and recreation
function of the forest (total, including private forest, including forest of the public sector)
Programme Additional Indicator Measurement Thematic Field
Sachsen Implementation rate of regional
concepts
no measurement provided LEADER
Schleswig-Holstein Improvement of living quality later collected based on
consultation of concerned population
quality of life
Bayern Number of persons who benefit
directly of the flood protection
no measurement provided environment
Bayern increase of life quality the effects are to be investigated
within special case studies
quality of life
Castilla y León New LAGs no measurement provided LEADER
Pais Vasco Study of quality of life no measurement provided quality of life
Cantabria % age reduction of the farmer
holders.
no measurement provided demography
Corsica (but national priorities)
Generation renewal Number of farmers under 35 yers
old related to the number of farmers over 55
demography
Marche population dynamics resident population interested by
program
demography
Scotland improvement in community capacity no measurement provided LEADER
Source: European Commission (2008)
Another group of additional/programme specific indicators was identified in the field of environmental impacts, which are however not linked to any of the four thematic fields of the common impact indicators as listed in the CMEF. The assessment of impacts of RDPs on soil quality seems to be a prominent aspect from the MS perspectives as 9 RDPs have listed soil quality related impact indicators. Other environmentally oriented additional impact indicators cover aspects as: “Improvement of the ecological stability of forest resources”; “Restoration of
forestry production potential”; “Changes in environmental awareness of agricultural producers”; and “Improvement in animal welfare in beneficiary farms”, just to name a few.
In a minority of cases additional impact indicators have been established due to specific national legal framework conditions (e.g. labour law).
These additional impact indicators could be seen as valuable input for a review of the set of common indicators, as they represent a creative attempt by MS to grasp the impact of RDPs more comprehensively.
8. CONCLUSIVEREMARKS
The Guidance Document on “Approaches for assessing the impacts of the Rural Development Programmes in the context of multiple intervening factors” is designed in an interactive pdf file to be downloaded at http://enrd.ec.europa.eu/evaluation/
The interactive pdf file provides the following features to ensure its user-friendliness:
• Full text of the working paper
• Current practice examples are highlighted/in boxes
• Bookmarks are set at each of these sections, chapters and summary tables of all seven impact indicators
• Indices are linked with chapters, tables, figures and boxes in the document The document is supposed to support the following different reader groups:
• Readers with administrational background (Managing Authorities, Steering Group Members, etc)
• Evaluators and evaluation related readers
• Actors within RD Programme implementation and beneficiaries
A vast variety of current practices, which may be seen as inspiration and source for ongoing and future evaluation exercises are included in the document as well as a large set of additional (country specific) impact indicators, which may serve as reference for own indicator development in the MS.
ACKNOWLEDGMENT
The authors would like to express their acknowledgement to the European Evaluation Network for Rural Development and the Evaluation Unit of DG Agriculture and Rural Development for initiating and supporting this working paper and for providing constructive input and support for its dissemination. Additional information about the activities of the European Evaluation Network for Rural Development are available on the internet through the Europa server (http://enrd.ec.europa.eu/evaluation) and by contacting the Evaluation Helpdesk, 260 Chaussée St Pierre, B-1040 Brussels, Tel. +32 (0)2 736 18 90. E-mail: [email protected]
REFERENCES
EENRD, European Commission (2008): Guidance Document – The Application of the High Nature Value Impact Indicator. Available on the web at http://ec.europa.eu/agriculture/rurdev/eval/hnv/guidance_en.pdf and
http://ec.europa.eu/agriculture/rurdev/eval/hnv/guidance_annexes_en.pdf
European Commission, DG Agriculture and Rural Development (2006) CMEF, Handbook on, Guidance document (including all Guidance Notes A-O); http://ec.europa.eu/agriculture/rurdev/eval/guidance/document_en.pdf; Brussels European Commission, DG Agriculture and Rural Development (2006) CMEF, Handbook on, Guidance note B – Evaluation Guidelines; http://ec.europa.eu/agriculture/rurdev/eval/guidance/note_b_en.pdf; Brussels
European Commission, DG Agriculture and Rural Development (2006) CMEF, Handbook on, Guidance note G – Baseline Indicator Fiches; http://ec.europa.eu/agriculture/rurdev/eval/guidance/note_g_en.pdf; Brussels
European Commission, DG Agriculture and Rural Development (2006) CMEF, Handbook on, Guidance note I – Result Indicator Fiches; http://ec.europa.eu/agriculture/rurdev/eval/guidance/note_j_en.pdf; Brussels
European Commission, DG Agriculture and Rural Development (2006) CMEF, Handbook on, Guidance note J – Impact Indicator Fiches; http://ec.europa.eu/agriculture/rurdev/eval/guidance/note_j_en.pdf; Brussels
European Commission, DG Agriculture and Rural Development (2006) The Community Strategic Guidelines for
Rural Development http://ec.europa.eu/legislation_summaries/agriculture/general_framework/l60042_en.htm;
Brussels
European Commission, DG Agriculture and Rural Development (2008): Synthesis of Ex Ante Evaluations of Rural
Development Programmes 2007-2013; Evaluation Report; Brussels;
European Evaluation Network for Rural Development (2010) Working Paper on Approaches for assessing the impacts of the Rural Development Programmes in the context of multiple intervening factors. Working Paper, European Evaluation Network for Rural Development; DG Agriculture; http://enrd.ec.europa.eu/evaluation/; Brussels EVALSED (2008): The Guide; http://ec.europa.eu/regional_policy/sources/docgener/evaluation/evalsed/guide/ methods_techniques/indicators/assessing_en.html