can be a two-stage estimator (Heckman, 1976) that treats unobservable heterogeneity as a problem of an omitted variable, and solves this problem by including an estimate of the omitted variable as a regressor in the outcome equation along with the participation dummy and individual characteristics21.
In our study the presence of hidden bias is formally tested using the approach described below in Chapter 5.6. <sensitivity analysis>.
5.5. Combined PSM and Difference-
in-Differences estimator
(conditional DID)
As shown above, conventional DID methods fail if the impact of unobservables is not time-invariant so that a group of programme participants and a control group are on different development trajectories. The probability of different development trajectories increases if already at the beginning of the programme the observed heterogeneity of both groups (and therefore the selection bias) is large. While propensity score matching can be applied to control for selection bias on observables at the beginning of the programme, a combination of PSM with DID methods (conditional DID estimator) allows for a better controlling of selection bias in both observables and unobservables. The combined PSM and DID method is a highly applicable estimator in case the outcome data on programme participants and non-participants is available both “before” and “after” periods (t’ and t, respectively). The PSM-
21 A recent microeconomic evaluation literature focussed on constructing and estimating of models allowing for heterogeneity in response to programme participation among otherwise observationally identical units. Important outcome of these studies is the development of a new class of econometric estimators which allow for the possibility of selection to treatment (e.g. decision to participate in a programme) that is based on unobserved components of heterogeneous responses to treatment. (Heckman and Vytlacil, 2005).
DID measures the impact of the RD programme by using the differences between comparable to each other programme participants (D=1) and non-participants (D=0) in the before-after situations. In this method observed changes over time for the matched (using PSM) programme non-participants are assumed to be appropriate counterfactual for programme participants.
The simplified notation for PSM-DID calculation can be described as follows:
PSM-DID = {Σ (Yit | (D=1) – Yit | (D=0)) – Σ
(Yit’ | (D=1) – Yit’ | (D=0))}/n (26a)
Where:
(Yit | (D=1) – Yit | (D=0)) is the difference in
mean outcomes between the i participants and the i matched comparison units after the access to the RD programme and
(Yit’ | (D=1) – Yit’ | (D=0)) is the difference in
mean outcomes between the i participants and i
matched comparison units at date 0 (prior to the
RD programme).
A decisive advantage of the PSM-DID estimator (conditional DID estimator), compared to a conventional DID estimator, is that by applying this methodology, initial conditions regarding observable heterogeneity of both groups (programme participants and non-participants) that could influence subsequent changes over time are largely eliminated22. Similarly, an
application of a conditional DID estimator (PSM- DID) to the measurement of the effects of a given RD programme may greatly improve research findings compared with a situation where a standard PSM (e.g. for estimation of ATT) that uses post-intervention data only is applied.
31
C ou nt er fa ct ua l i m pa ct e va lu at io n of E U r ur al d ev el op m en t p ro gr am m es - P ro pe ns ity S co re M at ch in g m et ho do lo gy ...The following example illustrates a potential qualitative difference in results (i.e. a conditional
DID estimator (ATT-DID) vs. a standard ATT), see: Graph 4.
Graph 4. Comparison of ATT with ATT-DID estimator
In the above example (Graph 4) the use of a standard ATT estimator (based on post-intervention data only) calculated as a difference between mean values of a result indicator in the group of programme beneficiaries and matched control group (Y3-Y2) would have led policy makers to conclude incorrectly that the effect of a given RD programme was positive (the calculated post intervention ATT is higher than zero). Yet, had a ATT-DID estimator been applied, the effects of a given RD programme would have to be judged negatively, i.e. a mean value of a result indicator in the group programme beneficiaries remained unchanged (Y1) while in the matched control group
<i.e. without the programme> it increased in the examined period <before and after programme> from Y1 to Y2 (the calculated ATT-DID estimator is
negative, i.e. (Y3-Y2) – (Y3-Y1) < 0).
5.6. Sensitivity analysis
5.6.1. Rosenbaum bounding approach
Since estimation of the magnitude of selection bias with non-experimental data is
impossible one possibility to address the issue of unobservables is the bounding approach proposed by Rosenbaum, 2002. The approach allows determining how much hidden bias would need to be present to render plausible the null hypothesis of no effect (Rosenbaum, 2002) or in another words how strongly an unmeasured variable must influence the selection process in order to undermine the implications of matching analysis (Caliendo and Kopeinig, 2005).
As stated in (Becker and Caliendo, 2007) the bounding approach does not test the unconfoundedness assumption itself, because this would amount to testing that there are no (unobserved) variables that influence the selection into the programme, but instead this approach provides an evidence on the degree to which any significance results hinge on this untestable assumption.
While an extensive discussion of this approach was provided in Rosenbaum, 2002 an outline of this approach can also be found in (Caliendo and Kopeinig, 2005; Becker and Caliendo, 2007).