Program evaluations: a brief review

Past research has shown that income transfers have been successful in improving health indicators, increasing school attendance, and reducing current poverty and inequality indicators. CCTs focus on poverty alleviation and household investment in human capital, primarily by increasing school enrollment. As with other successful policy interventions, policymakers and researchers alike have questioned the possibility of further efficiency gains by fine tuning the design of the program.

Impact evaluations are an important tool to measure program outcomes ex-post. The

Oportunidades program was designed from the start to have a measurable impact. The program

was implemented as a randomized experiment, and as such it offers a perfect benchmark for evaluation. The results of the short-run impact evaluations in rural areas are summarized in Skoufias (2005). Another valuable source is Levy (2006) who summarizes many impact evaluations of the Mexican program including consumption, health and nutrition, education, and targeting.

The assessed impact of the program on education indicators vary by educational level and socio- economic background.24 According to Schultz (2000) the program’s impact on primary school enrolment has been small and this might be due to the fact that primary school enrolment rate was already high (around 90%) when the program was implemented. This result is corroborated

37 by Parker (2003), whose research also indicated that the program affected dropout rates negatively (keeping students enrolled). Todd et al. (2005) also confirms this result for urban areas.

The scenario is very different if considering secondary education. Estimates of the increase in secondary education levels are around 11 percent for girls and 7.5 percent for boys in rural areas after just 2 years of the program (Schultz, 2000). Another study uncovers similar results: Parker (2005) finds a 24 percent increase in secondary school enrolment between the launching of the program and the 2002-2003 enrolment rates in rural areas. She indicates that the increase was higher for girls than for boys— 28.7 percent and 15.7 percent respectively. A study by Hernandez et al. (2000) finds that the program had a bigger impact on school attendance of children of less educated mothers. In schools where the average years of education of the mothers were low (around 3 years of education) school enrolment increased by 75 percent more than in schools where the average mothers’ education was higher (around 9 years).

As described above, the evaluation results largely indicate that the program has had a stronger positive effect on secondary and high school enrolment. A study by Skoufias (2000) suggests increasing education transfers to students who finish high school and encourages establishing a relationship between the money allowance and school performance. As indicated by Levy (2006), the Oportunidades program increased the transfer to high school students (without linking it to school performance) in 2001. The results of Parker (2005), which analyses the impact of additional scholarships for high school students, indicate an increase of 85 percent in high school enrolment for rural areas and 10 percent for urban areas.

In order to design social programs or to fine-tune existing programs, researchers have increasingly turned to techniques that enable them to shed light on the potential impacts of policies. Ex-ante evaluation models include an array of possibilities and aim primarily at simulating the effects of policy interventions (before they take place) to quantify their impact. Finding a balance between the model’s structural complexity and feasibility of an empirical application is the challenge for the achievement of a successful ex-ante evaluation. As mentioned

38 by Wolpin (2007) there is no consensus of what constitutes the best model of ex-ante policy evaluation. The non-experimental approach, which is what defines an ex-ante evaluation of any policy intervention, must rely on parametric as well as behavioral assumptions.

For PROGRESA-Oportunidades, Todd and Wolpin (2006) estimate and validate a dynamic behavioral model of parental decisions about fertility and school attendance to evaluate the effects of the program. The authors carry out several counterfactual experiments of program rules and identify an alternative transfer scheme that could increase average school attendance without further increases in the program’s budget. They compare their ex-ante results with the results found in ex-post evaluations of the program. Using the same data, Attanazio et al. (2001) analyze a dynamic school participation model and discuss an alternative program design that focuses on secondary school attainment. De Janvry and Sadoulet (2006) use the 1998 PROGRESA survey data to analyze efficiency gains—by means of improving school enrollment among poor children—in the program. Their paper proposes improving targeting among poor households and changing the design of the cash transfers.

As pointed out by Heckman (2000), the forecast of policy changes before their implementation is one of the greatest challenges of empirical research. Simulation exercises as well as ex-ante evaluations techniques are frequently implemented in the context of social programs (Freije et al., 2006; Todd and Wolpin, 2006 and 2007). More than a few surveys in the literature analyze the evolution and application of these techniques in different fields. BFL propose a behavioral ex-ante microsimulation methodology to assess the potential impact of alternative transfer schemes on the household decision to send a child to school. Their paper applies the methodology to the Brazilian Bolsa Escola program using data from a household survey. This paper applies the BFL model for the Mexican program and perform simulations of alternative designs of the transfers scheme to fine-tune the 2005 program rules. Bornhorst (2004) applies the BFL model to the evaluation data base of rural PROGRESA. In the same line as this work—with the objective of validating the BFL model—his findings indicate that the effects measured by the ex-post evaluation technique are within the confidence interval obtained through the

39 microsimulations. However, his results are not representative at the national level and in the rural sample transition to secondary education is not captured well by the model.