Development after one Year of Treatment

Descriptive statistics

In Table 3.2, I present the characteristics of the children and the community mothers for the three cohorts from the monitoring dataset. This sample includes the data of 511 children. Across the three samples, we observe that the children display cognitive, psychosocial, and psychomotor levels of between 68% and 78%. Indicating that according to this UNICEF scale children in my sample fall behind. Furthermore, while the cognitive and psychomotor developmental indexes are not significantly different at baseline, the psychosocial develop- ment of the children in the third cohort is significantly lower compared to the other two cohorts. Figure 3.4 depicts the average of the three indexes at baseline and at the end of

the intervention (endline) separately for the three cohorts.22

22_{The cognitive and the psychosocial development at baseline between the three cohorts are not signif-}

icantly different (based on two tailed t-tests). The same holds true for the psychosocial index between the first and the second cohort. However, the children from the third cohort display a psychosocial develop- ment which is significantly lower compared to the other two cohorts. I depict the separate indexes in the

For the community mothers, I observe a significant difference with respect to their age and the years of experience. The community mothers in the first cohort are slightly older than the community mothers in the other two cohorts. This is also reflected in the experience of the community mothers, which refers to the years that a women worked as community mother. In cohort one they worked on average for 20 years as community mother. In cohort two the average years of experience are 11 years, and in cohort three their average experience amounts to 15 years. The income of the community mothers in the first cohort is 377’000 COP (∼ 124 USD). This is lower than the income of the community mothers in the second and the third cohort (cohort 2: 421’000 COP (∼ 139 USD) and cohort 3: 430’000 (∼ 142 USD)). In all three cohorts, the income is less than the legal minimum salary in Colombia which is 690’000 COP (∼ 240 USD). The educational levels of the community mothers are comparable across the cohorts. The variable education captures the level of education of the community mother. 2.5 indicates that on average a community mother finished secondary school (level two) but not tertiary school (level three).

As outlined in the previous section, I will account for all these observable differences in the regression analysis by including them as control variables.

0 20 40 60 80 100 Pe rce n ta g e s 2011 2012 2013 Child Development Baseline Endline

Figure 3.4: Development of the children in the three cohorts (in percentages).

Table 3.2: Descriptive statistics.

Cohort 1 (2011) Cohort 2 (2012) Cohort 3 (2013)

Mean SD Mean SD Mean SD

Children:

Age (in months) 32.39 9.21 29.38 9.66 29.56 9.52

Female (D) 0.46 0.50 0.44 0.50 0.55 0.50 Cognitive level (%) 76.23 14.38 74.74 8.52 75.74 6.13 Psychosocial level (%) 75.70 13.11 76.93 7.17 59.83 6.18 Psychomotor level (%) 68.34 15.81 68.64 14.51 74.01 7.38 Observations 136 328 164 Community Mother:

Age (in years) 53.32 7.35 46.83 8.92 48.85 8.21

Experience (in years) 19.42 6.28 11.22 5.99 15.54 8.74 Income (in 1’000 COP) 377.24 118.63 421.34 165.82 430.97 159.50

Education 2.57 0.79 2.49 0.75 2.17 0.90

Observations 20 20 20

Note.Descriptive statistics for the three cohorts. The children from the first cohorts are born between 2007 and 2009. The children from the second cohort are born between 2007 and 2010, and the children from the third cohort are born between 2008 and 2010. Both age variables depict the respective age at the beginning of the intervention. The develop- mental status of the children are from the baseline measurement. For cohort 1, baseline was elicited in January 2011, for cohort 2 the baseline was elicited in January 2012. For cohort 3 the baseline was elicited in January 2013. The income of the community mothers corresponds to the income at the beginning of the treatment.

Regression results

Table 3.3 depicts the results of the OLS estimations. The first column shows the treatment effect on cognitive development. Children in the MD control group reach on average 70% of the cognitive items. In contrast, treated children score on average 11 percentage points (p < 0.001) higher on the cognitive development index. In column (2) I regress the psychosocial development status on the treatment and the control variables. I find that the intervention affected the psychosocial status of the children considerably. Children in the MD control group reach on average 70% of the items. Whereas children from the MD treatment group attain on average 12 percentage points more on the psychosocial index (p < 0.001). I observe a similar pattern for the psychomotor development (column (3)). Children from the MD control group accomplish 60% of the items on average, and the

intervention accounts for an increase of 13 percentage points (p < 0.001).23

Interestingly, girls score on average significantly lower on the cognitive and the psy-

23_{The separate results (sample 1 and sample 2) are comparable. The corresponding tables are in the}

chosocial index. Additionally, I observe that age has predictive power for all three indexes.

Table 3.3: OLS estimates for the development of the children.

Cognitive Psychosocial Psychomotor

Treated (D) 10.957∗∗∗ _12.212∗∗∗ _13.387∗∗∗

(0.467) (0.546) (0.868)

Age (in months) 0.247∗∗∗ _0.209∗∗∗ _0.459∗∗∗

(0.075) (0.075) (0.090)

Female (D) −1.857∗∗∗ _−1.551∗∗ _−1.508

(0.663) (0.689) (0.928)

Age (in years) 0.134 0.125 0.101

(0.114) (0.116) (0.125)

Experience (in years) −0.241∗∗ _{−0.217 −0.197}

(0.117) (0.134) (0.146)

Income (in 1’000 COP) 0.003 0.004 0.003

(0.004) (0.004) (0.004) Education −0.071 0.124 0.001 (0.540) (0.466) (0.645) Cohort 2 (D) −4.206∗ _{−2.064 −4.138}∗ (2.138) (2.148) (2.278) Cohort 3 (D) −3.535 −19.649∗∗∗ _−2.790 (2.114) (2.066) (2.413) Constant 69.660∗∗∗ _69.594∗∗∗ _60.876∗∗∗ (4.289) (3.679) (4.609) F-test 116.8 280.9 62.7 Prob > F 0.000 0.000 0.000 R2 adjusted 0.387 0.692 0.423 N 511 511 511

Note. OLS estimates. The unit of all dependent variables is percentages. Depen- dent variable in column 1 is cognitive development. In column 2 the dependent variable is psychosocial development and in column 3 it is psychomotor develop- ment. Robust standard errors, clustered on community mother, in parentheses.

∗ _{p <0.1,}∗∗ _{p <0.05,}∗∗∗ _{p <0.01.}

In a further set of estimations (reported in Table B4 the appendix) I estimated the effect including all interaction terms. I observe that the intervention was less effective for girls with respect to cognitive and psychosocial development. Furthermore, it seems that for the psychosocial and the psychomotor development, the effect of the intervention is smaller the older the child.

To sum up, based on ordinary least squares regressions, I find that the program led to large skill gains for the treated children. There are two potential caveats with these results. First, the implementing NGO was also the NGO which elicited these develop-

mental measures. Therefore, these measures might be biased. With the data at hand, I cannot elaborate this caveat further. However, the second evaluation (four years after the treatment) will not be subject to this bias and hence depending on the conclusion, strengthen the results of this part. Second, it is possible that unobservable characteristics of the community mother or the child are responsible for the differences and not the inter- vention itself. Hence, in the next section, I will assess the validity of the results using two alternative estimation methods.

Robustness checks

Given that the assignment to treatment was not random, I conduct two robustness checks. First, I estimate the effects of the program using inverse probability weighted regression adjustment (IPWRA) as suggested by Imbens and Wooldridge (2009) and in line with the analysis conducted by Bernal (2015).

The estimation procedure is as follows: first, I estimate for each child the probability to belong to the MD treatment group using a probit regression. For that, I include all observable characteristics of the child and the community mother.

I present the results in Table 3.4. This estimation serves as the basis for the subsequent regression adjustment. They indicate that the income and the experience of the community mother predict treatment participation. In the second step, I estimate the propensity score. Figure 3.5 shows the propensity scores to check the common support condition. For the subsequent estimation, I include all observations which are within the common support. It is not surprising that all observations fall into the common support, given the small number of control variables.

Table 3.4: Probit of participation in treatment.

Prob. treated

Age (in months) 0.009

(0.006)

Female (D) −0.074

(0.114)

Age (in years) 0.001

(0.010) Experience (in years) 0.017

(0.011) Income (in 1’000 COP) −0.001∗

(0.000) Education 0.285∗∗∗ (0.076) Constant −0.970∗∗ (0.483) χ2_{-test 24.2} p 0.000 Pseudo R2 0.034 N 511

Note. Probit estimates for the probability to belong to the treated group. Standard errors in parentheses. ∗ _{p < 0.1,} ∗∗ _{p < 0.05,} ∗∗∗

0 1 2 3 4 D e n si ty .2 .4 .6 .8

Estimated propensity score

Control Treated

Figure 3.5: Common support.

Finally, to assess the treatment effect, I estimate a linear regression where I weight the observations of the treatment by the inverse probability of the propensity score and the observations of the control group by one minus the inverse probability of the propensity score (Imbens & Wooldridge, 2009). Table 3.5 summarizes the average treatment effects for the three dependent variables estimated with IPWRA. The point estimates of the treatment effect are very similar to the estimates of the OLS regressions and support the results reported in the previous section.

Table 3.5: Treatment effects estimated by IPWRA.

Outcome variable Control group Treatment effect Standard error N

Mean SD

Cognitive development 76.461 7.163 12.574 0.819∗∗∗ ₅₁₁

Psychosocial development 69.528 11.038 21.823 0.953∗∗∗ ₅₁₁

Psychomotor development 73.757 11.284 14.582 0.937∗∗∗ ₅₁₁

Note. Treatment effects estimated using inverse probability weighted regression adjustment.

Second, I try to estimate the magnitude of the omitted variable bias using the method proposed by Altonji et al. (2005) and by Oster (2014). This methodology bases on the assumption that selection on unobservables is equal to selection on observables. It allows

to calculate the relative degree of selection (δ). The authors propose δ = 1 to be an

Table 3.6: Selection on unobservables.

δ N

Cognitive 1.24 511

Psychosocial 1.83 511

Psychomotor 1.33 511

Note. Amount of selection on unob- servables relative to selection on ob- servables required to attribute the en- tire high-quality preschool education effect to selection bias. Calculation based on Oster (2014).

I find that all values for δ reach the critical threshold. Hence, the observed effects are likely to hold if we assume that selection on unobservables is equal to selection on observables.

To sum up, the analysis of the monitoring data provides evidence that the program had a considerable positive effect in the short run on all three outcome variables. Thanks to the intervention, the children are able to catch up with the national averages on these three

indexes.24 Furthermore, the results are in line with an evaluation by Bernal (2015). She

assessed a similar program from Bogotá in 200725and observes that children’s psychosocial,

psychomotor and cognitive skills increased by up to 0.3 standard deviations compared to children who visited a standard community mother.

These results are very promising, but the most important question remains, namely whether these differences persist in the longer run, which is the scope of the next section.