Sensitivity analysis

Recently checking the sensitivity of the estimated results becomes an increasingly important topic in the applied evaluation literatures (Caliendo and Kopeining, 2008).

50

Matching method is based on the conditional independence or unconfoundedness assumption, which states that evaluator, should observe all variables simultaneously influencing the participation decision and outcome variables. This assumption is intrinsically non-testable because the data are uninformative about the distribution of the untreated outcome for treated units and vice versal (Becker and Caliendo, 2007). As outlined in equation (5) that the estimation of treatment effects with matching estimators is based on the unconfoundedness or selection on observables assumption. However, if there are unobserved variables which affect assignment into treatment and the outcome variable simultaneously, a ‘hidden bias’ might arise (Rosenbaum, 2002). In other word, if treatment and outcomes are also influenced by unobservable characteristics, then CIA fails and the estimation of ATTs are biased. The size of the bias depends on the strength of the correlation between the unobservable factors, on the one hand, and treatment and outcomes, on the other.

It should be clear that matching estimators are not robust against this ‘hidden biases. Different researchers become increasingly aware that it is important to test the robustness of results to departures from the identifying assumption. Since it is not possible to estimate the magnitude of selection bias with non-experimental data, the problem can be addressed by sensitivity analysis.

Rosenbaum (2002), proposes using Rosenbaum bounding approach in order to check the sensitivity of the estimated ATT with respect to deviation from the CIA. The basic question to be answered here is whether inference about treatment effects may be altered by unobserved factors. In other words, one wants to determine how strongly an unmeasured variable must influence the selection process in order to undermine the implications of matching analysis.

The bounding approach does not test the unconfoundedness assumption itself, because this would amount to test that there are no (unobserved) variables that influence the selection into treatment. Instead, Rosenbaum bounds provide evidence on the degree to which any significance results hinge on this untestable assumption. If the results turn out to be sensitive,

51

the evaluator might have to think about the validity of his identifying assumption and consider other estimation strategies.

As noted above, it is not possible to estimate the magnitude of selection bias using observational data, instead the sensitivity analysis using the bounding approach that involves calculating upper and lower bounds, using the Wilcoxon signed rank test. This rank tests the null hypothesis of no-treatment effect for different hypothesized values of unobserved selection bias.

The central assumption of the analysis is that treatment assignment is not unconfounded given the set of covariates , i.e., that equation (5) no longer holds. In addition, it is assumed that the CIA holds given and an unobserved binary variable : In other words the probability of participation F(⋅) needs to be complemented by a vector U containing all unobservable variables and their effects on the probability of participation captured by .

PX, U  prD  1/X, U  FXβ  Uγ  e^{F?A]^} 15

Where _ is the effect of on the probability of participation in the program. Assuming that F follows logistic distribution, the odds ratio of two matched individuals (let say m and n), who are identical in observable characteristics, receiving the treatment written as:

PX, u_a

PX, u_J X1 PX, u_J

1 PX, u_a e^{?bFbA^bcb}

e^{?CFCA^CcC}  e^{+^]C9]b-} 16

Equation (16) states that two units with the same differ in their odds of receiving the treatment by a factor that involves the parameter and the difference in their unobserved covariates . As long as the there is no difference in between the two individuals or if the unobserved covariates have no influence on the probability of participation ( = 0). This happens if the probability of participation will only be determined by the vector and the selection process is random. >0 implies that two individuals with the same observed

52

characteristics have different chances of participating in the program due unobserved selection bias. In our sensitivity analysis, we examined how strong the influence of or (

U_a U_J) on the participation process needs in order to attenuate the impact of market development on potential outcomes.

Following Rosenbaum (2002), equation (16) can be rewritten as:

e1^{^} d PX, U_a1 PX, U_J

PX, U_J 1 PX, Ua d e^{^} 17

Both matched individuals have the same probability of participating only if e^f=1 provided that they are identical in . Consequently there will be no selection bias on unobservable covariates. If =2, one of the matched individuals may be twice as likely to participate as the other agent (Rosenbaum, 2002). If is close to one and changes the inference about the treatment effect, the impact of participation on potential outcomes, the estimated effect is said to be sensitive to hidden bias. In contrast, insensitive treatment effects would be obtained if a large value does not alter the inference about treatment effects. In this sense, can be interpreted as a measure of the degree of departure from a study that is free of unobservable selection bias (Rosenbaum, 2002). Several values of bounds are calculated on the significance level, and hence, the null hypothesis of no effect of treatment on potential outcomes, is then tested.

Eventually, using predicted probabilities of participation in the program (i.e. propensity score) match pairs are constructed using alternative methods of matching estimators. Then the impact estimation is the difference between simple mean of outcome variable of interest for participant and non participant households.

The difference involvement in market development by IPMS project between treatment and matched control households is then computed. The ATT is obtained by averaging these

53

differences in market development outcomes (_g) across the k matched pairs of households as follows:

ATT 1

K HiY^//∈kK Y_/^/∈kKl

m /K

18

A positive (negative) value of ATT suggests that households who have participated in market development have higher (lower) of outcome variable non-participants.