T
his report has presented the main findings of a quantitative evaluation of a randomized community-based intervention, RPS, against its primary objectives. Where possible, we err on the side of assessing what are the short-term effects of the program in con- servative manners, for example, by presenting intent-to-treat estimates and in the treatment of possible contamination and attrition. The estimates presented represent the effect of the pro- gram as a whole, in particular combining supply- and demand-side components. They also represent only the short-term effects of the program (after 1 or 2 years), although some of the outcomes examined are themselves long-run indicators. In many instances, for example, when assessing the effects on expenditures during what turned out to be an economic downturn, the critical importance of having baseline data and a control group for the evaluation was evident —without one it would have been next to impossible to make reliable assessments of the pro- gram effects. Even with one, it was at times difficult to make certain assessments of effects for some indicators since the evaluation did not exist in a vacuum, and a variety of actors not under the control of the program continued to operate in the program areas.Overall, we found that RPS had positive (i.e., favorable) and significant double-difference estimated average effects on a broad range of indicators and outcomes. Where it did not, it was often because of similar, although smaller, improvements in the control group, which were likely to have been influenced in part by the program itself. Nearly all estimated effects were larger for the extremely poor, often reflecting their lower starting points (e.g., percentage of children enrolling before the program). Among poorer beneficiaries there was simply more potential for improvement on many of the indicators. As a result, the program has reduced inequality of these outcomes across expenditure classes.
During Phase I, RPS supplemented per capita annual total household expenditures by 18 percent, on average. For beneficiary households, this increase compensated for the large in- come loss experienced by non-beneficiaries during this period, while producing a small over- all increase in expenditures. Most of the increase in expenditures was spent on food; the pro- gram resulted in an average increase of C$640 (US$50) in per capita annual food expenditures and an improvement in the diet of beneficiary households. Expenditures on education also increased significantly, although there was no discernible effect on other types of investment expenditures. Labor market participation was apparently little changed with the program over time, although there was an indication of slightly fewer hours worked by men in the last week, relative to the control group. The economic difficulties experienced by these communities enabled RPS to operate somewhat like a traditional social safety net, aiding households dur- ing a downturn. For schooling, RPS produced a large average net increase on enrollment of 12.8 percentage points and an even larger effect of 20 percentage points on current attendance for the target population. Examining the number of children in grades 1–4 who advanced two
grades between 2000 and 2002, RPS led to an average increase of 7.3 percentage points, despite the fact that advancement past grade 4 was not a formal requirement of the pro- gram. In tandem with the increased school- ing, the percentage of children 7–13 years working declined by 5.6 points.
For child health care, RPS induced an av- erage net increase of 16.3 percentage points in the participation of children younger than age 3 in VPCD but the effect was only 8.4 percentage points (and insignificant) in 2002 since participation by control group households increased substantially. The health-care services provided by the pro- gram, as measured by process indicators in- cluding whether the child was weighed and whether his or her health card was updated, improved even more. Participation by chil- dren ages 3–5 also increased substantially. Although it is not possible to statistically demonstrate that RPS increased vaccination coverage for children ages 12–23 months in the intervention group (relative to the con- trol group), it was demonstrated that vacci- nation rates climbed 30 percentage points in the intervention and control areas at a time when they were, on average, decreasing in rural areas nationally. One would be hard pressed not to attribute this substantial im- provement in large part to RPS.
Finally, the more varied household diet and increased use of preventive health-care services for children have been accompanied by an improvement in the nutritional status of beneficiary children younger than age 5. The net effect was a 5.5 percentage point decline in the number of children who were stunted. This decline was 1.7 times faster than the rate of annual improvement seen at the national level between 1998 and 2001. Very few programs in the world have shown such a decrease in stunting in such a short time. Despite improvements in the distribu- tion of iron supplements to these same chil-
dren, however, RPS was unable to improve hemoglobin levels or lower rates of anemia. In related work, Caldés and Maluccio (2005) analyze program and non-program costs for RPS, including costs for design, planning, and execution of the program in the evaluation areas and everywhere else it was operating (see Appendix A). While they were unable to separate out program costs pertaining to the households evaluated in this report, they present a number of find- ings relevant to this study. In particular, they found that administrative costs were a little less than 20 percent in the pilot phase, after taking out fixed costs associated with de- sign, planning, and evaluation. These were higher than the costs for PROGRESA or PRAF (Caldés, Coady, and Maluccio 2004), although the difference appears to be due to RPS’s complicated design (with an in- volved supply-side component that the other two programs did not have), pilot nature, and its small size, particularly in comparison to PROGRESA. Furthermore, RPS arguably had the most impressive effects (Rawlings and Rubio 2005).
Nearly half of the costs were due to a number of special program features that are hallmarks of conditional cash transfer pro- grams, in particular targeting, monitoring and conditioning, and coordinating the health- care supply. Removing these costs would reduce the administrative costs substantially, but it is not possible to determine by how much it would reduce the effectiveness of the program—indeed their removal could rep- resent a false savings (Caldés and Maluccio 2005).
The findings presented here played an important role in the decision in late 2002 by the Government of Nicaragua and IADB to continue and expand this effective pro- gram as RPS–Mi Familia, Phase II.49 In
the expansion, the program was altered in three important ways. First, the health-care
CONCLUSIONS 57
49The program was transferred within the Nicaraguan government from FISE to the Ministry of the Family (Mi
services were expanded to benefit additional household members, in particular with pre- natal services for pregnant women, vaccina- tions for 6- to 9-year-olds, and preventive health care for adolescents and for all women age 20 and older. Second, initially designed as a 3-year program with both demand- and supply-side components lasting for 3 years, the supply-side components of the program (in particular the health-care services and the teacher transfer) were extended in Phase II to last for a total of 5 years, such that in the final 2 years participants would receive supply-side benefits only. Third, based on evidence of the effectiveness of the program and cost considerations, the transfer size was reduced by one-third.
RPS has improved a number of the indi- cators included in the Nicaraguan national poverty reduction strategy, during a time in which many of them are not on track to achieve the goals set out in the plan to 2015 (World Bank 2003). RPS demonstrates that it is possible in Nicaragua, as it has been in Mexico and other places, to combine short- term reduction in rural poverty with im- provements in human capital of children.
The evidence from the evaluation strongly suggests that if the program were expanded elsewhere in poor rural areas of Nicaragua (as it was in 2003), it would be effective, al- though due to the caveats discussed through- out this report, we stop short of saying it would necessarily be equally as effective. Nonetheless, it could prove to be an im- portant component of Nicaragua’s overall poverty reduction policy.
A crucial question that the current eval- uation was unable to answer is whether the effects documented here will persist after the program exits, and whether there are longer-term effects that have not been cap- tured in what is only a short-term, 2-year evaluation. In late 2003, RPS delivered the final demand-side transfers in the original intervention areas, although it is scheduled to continue offering health-care services and teacher transfers until the end of 2005. Con- tinued survey work in these areas will pro- vide some of the information necessary to examine the effects of that transition, and begin to understand better the sustainability of the large changes achieved by RPS.