Estimation of direct programme effects occurring at the level of SAPARD programme

Background

PSM methodology was applied170 in order to assess the results of the RD SAPARD programme in Slovakia (Measure 1: Support of investment in agricultural enterprises) on programme beneficiaries. The programme support under Measure 1 was primarily targeting the following agricultural sectors: a) beef sector, b) pig sector, c) sheep sector, d) poultry sector, e) fruits and vegetables sector. Programme support under Measure 1 had the form of a capital grant covering up to 50% of the costs to investments in the above sectors. The major beneficiaries of programme support (they received approximately 67% of funds available under this measure) were large agricultural companies located in relatively well developed regions of West Slovakia. The assessment of programme impacts on the agricultural companies was carried out on the basis of Slovak FADN database in the years 2002-2005.

Method

The following methodological steps were carried out in order to estimate the direct programmes effects on beneficiaries:

(a) SAPARD beneficiaries were identified and selected from the existing FADN databases to the panel. Data for each SAPARD beneficiary was collected in the years 2002-2003 (i.e. prior to their participation in SAPARD) and 2005 (after implementation of SAPARD).

(b) SAPARD general and specific eligibility criteria (e.g. pre-defined farm performance coefficients and farm profitability ratios; various minimum/maximum production, age, etc. thresholds, etc.) that were valid in individual years were translated into respective quantitative coefficients and applied to all non-SAPARD units included in FADN databases.

(c) Units which satisfied the above criteria in the years 2002-2005 and did not receive support from SAPARD programme were selected to the panel of a control group (eligible non-participants). (d) Respective balanced panels (i.e. embracing SAPARD beneficiaries and all non-SAPARD units

meeting SAPARD eligibility criteria in specific years) were constructed for the years 2002-2005, i.e. observations on the same units in period 2002-2005.

On the basis of available Slovak FADN database, 232 agricultural companies were selected for further analysis (balanced panel data) which was performed for the years 2003 (before SAPARD) and 2005

169

Pufahl and Weiss, 2009; Farm Structure and the Effects of Agro-Environmental Programs: Results from a Matching Analysis for European Countries, University of Kent, Cantenbury, UK, 2009

170_{Michalek J. (2012a), “Counterfactual impact evaluation of EU Rural Development Programmes - Propensity Score Matching}

methodology applied to selected EU Member States”, Volume 1 – A micro-level approach.”, European Commission, JRC Scientific and Policy Reports, pp 1-95, http://ipts.jrc.ec.europa.eu/publications/pub.cfm?id=5379

(after SAPARD). Out of the selected 232 agricultural enterprises, there were 51 agricultural farms SAPARD participants and 181 SAPARD non-participants but yet, SAPARD eligible.

The preliminary analysis showed that agricultural companies that received support from the SAPARD programme differed significantly in a number of important characteristics from eligible programme non- participants, i.e. SAPARD beneficiaries were in general much larger (ha), they employed more people and were more profitable (i.e. less unprofitable) compared to the agricultural companies that were non- supported from SAPARD. Given above, the group of SAPARD participants could not be directly compared to non-participants, as a selection bias would have been incurred.

In order to ensure comparability, the PSM method was applied using number of individual characteristics of agricultural companies as covariates (methodology applied to the selection of variables into the estimated logit function is described in the cited study). Imposition of common support region and selection of appropriate matching algorithm resulted in dropping from a further analysis the companies that were non-comparable. The applied balancing property tests (t-test) showed that the selected matching procedure (i.e. kernel epanechnikov bandwidth 0.06) considerably improved comparability of both groups of agricultural companies, making a counterfactual analysis more realistic. Indeed, there were previously significant differences in the most important farm characteristics between the group of agricultural companies supported from the SAPARD programme (D=1) and non-supported farms (D=0) dropped after matching (differences became no more statistically significant). This applies to all important variables determining both programme participation and outcomes, e.g. profit per company (prior to SAPARD programme), liabilities, value of current assets, etc.

The assessment of the micro-economic effects of a given RD programme on programme beneficiaries was carried out in both groups of farms using seven results indicators available from a standard FADN system:

 Profit per company,

 Profit per ha,

 Profit per person employed,

 Gross value added (GVA) per company,

 Employment per company,

 Labour productivity (Gross value added per employed),

 Land productivity (Gross value added per ha).

The results also showed that simple techniques applied to the estimation of programme effects can be highly misleading, whereas the application of advanced evaluation methodologies can lead to quite different yet much more reliable results.

Table 15 Estimation of the direct effect of RDP on agricultural companies (comparison of various methods) Methods/units GVA/company In SKK171 1,000 Before programme (T0) After programme (T1) DID (difference between T1 and T0) Programme participants (P=1) 17,727 18,478 751 Programme non-participants (P=0) 9,950 9,680 -270 Average Ø 11,660 11,614 -46 Difference (1-0) 7,777 8,798 1,021 Difference (1- Ø) 6,067 6,864 797

Matched programme participants (M=1) 11,082 9,610 -1,472 Matched programme non-participants

(M=0)

9,367 9,701 334

Average Treatment Effect on Treated (ATT) using Propensity Score Matching (PSM)

1,715 -90 -1,805

Where:

(P=1) A group consisting of all direct programme beneficiaries (before matching); (P=0) A group consisting of all programme non-beneficiaries (before matching).

(1-0) Programme effects calculated as a difference in outcome indicator (here: GVA/company) between direct programme beneficiaries (P=1) and non-beneficiaries (P=0).

(1- Ø) Programme effects calculated as a difference in outcome indicator (here: GVA/company) between direct programme beneficiaries (P=1) and an average Ø (country or sample).

(M=1) A matched group of direct programme beneficiaries. (M=0) A matched group of programme non-beneficiaries.

(ATT) Average Treatment on Treated calculated as a difference between (M=1) and (M=0). DID Difference in differences between outcomes observed in periods T1 and T0.

Results

Analysing the figures in the table above shows that:

 If a naïve before/after estimator was applied, the effect of the programme would be assessed as very positive (average change in GVA per company = +751 thousand SKK). Yet, this estimator is statistically biased.

 If programme participants were compared with all (unmatched) programme non-participants (before and after) and DID estimator was applied, the effect of the programme would also be assessed as very positive (average change in GVA per company = +1,021 thousand SKK). Yet, this estimator is statistically biased.

 If effects observed for programme participants were compared with a country’s average (e.g. performance standards) calculated for all farms, i.e. programme participants and non- participants (before and after) and DID estimator was applied the effect of the programme would be assessed as very positive (average change in GVA per company = +797 thousand SKK). Yet, similar as in (1) and (2) this estimator is statistically biased.

 The conclusions above have to be revised in case the programme effects are measured using statistically similar (matched) groups (participants vs. non-participants). In this case the estimated programme effect (DID in ATT) was found to be negative (!) (average change in GVA per company = -1,805 thousand SKK). The reason is a much higher growth in GVA per

company in the similar (matched) group of SAPARD non-participants (average change in GVA = +334 thousand SKK) compared with the matched SAPARD participants (average change in GVA = -1,472 thousand SKK).

The above example clearly shows that application of an incorrect (naïve) methodology to assess programme effects (e.g. a micro-economic effect of an investment support measure) may lead to a significant bias in estimated programme results making them unreliable.