Chapter 8 Empirical strategy
8.4. Outcome variable analysis
Having tested the validity of the SRD design, the estimation of the SRD needs to define the outcome of interest, and the model specification. These definitions are critical for answering the research questions previously defined, see Table 8.1. Then the proposed robustness checks are described.
Outcome of interest
In general, y(ijt+z) is the outcome at year t + l, where l is the number of years after the application year, i.e., t + 1 is both one year after the application and the year of receiving the treatment. As explained in Chapter 5, in 2007 and 2008, AEP only included the financial incentive; thus, estimating the program’s effect for those years is equivalent to isolating the effect of the financial component, while using the data for the rest of the years provides information for the effects of the Full program. The outcome definition varies with the research question being explored in the following way:
• To answer Question #1 Does AEP affect teacher’s performance? (Full program effect),
yijt+l is student i’s SIMCE test score for applicant teacher j in 𝑡 + 𝑙 for application years t 2003 to 2006, and 2009 to 2011; l ϵ (1, 2, 3).
106
• To answer Question #5, Does the effect of AEP fade-out? (Fade-out of the full program
effect), yijt+l is student i’s SIMCE test score for applicant teacher j in 𝑡 + 𝑙 where l ϵ (1, 2 or 3) for application years 2003 to 2006, and 2009 to 2011.
• To answer Question #2 Does each component of AEP affect teachers’ performance?
(Unbundled program effect), yijt+l is student i’s SIMCE test score for the applicant teacher j in 𝑡 + 𝑙 for application years t 2007 to 2008; l ϵ (1, 2, 3).
• To answer Question #3 Does AEP affect teachers’ behavior? (Underlying mechanism
of the full program), yjt+l is teacher j’s self-confidence (self-report on how prepared the teacher
feels to teach the curriculum evaluated by SIMCE) in t + l for application years t 2003 to 2006, and 2009 to 2011. Additionally, yjt+1 is teacher j’s effort level (proportion of the curriculum that
teacher covers during the academic year and number of hours spent in class preparation) in 𝑡 + 𝑙 for application years t 2003 to 2006, and 2009 to 2011; l ϵ (1, 2, 3).
• To answer Question #4, Does each component of AEP affect teachers’ behavior?
(Underlying mechanism of each program component), yjt+l is teacher j’s self-confidence or the effort level in 𝑡 + 𝑙 for application years t 2008 to 2009; l ϵ (1, 2, 3).
Regression discontinuity estimates
The SRD estimates are obtained using two supplemental methods: one parametric and another nonparametric. Using these two methods allows us, on the one hand, to estimate the effect of the full and the unbundled program, and, on the other hand, to make a specification robustness check of the results found. For a better understanding of the estimation methods, they are explained
107
below; complementarily, Table 8.1. shows the relationship between the outcome and parameter of interest in Equations 8.6 and 8.7, with the corresponding research question for each method.
A nonparametric local linear polynomial estimation method: the robust specification
In practice, after defining the outcome of interest, I determine the regression to estimate the SRD treatment effect SRD. With that, the robust bias-corrected local linear regression that I use is:
𝑌¡|šQ& = 𝛼-+ 𝛼&1œ𝑆|š− 𝑆r|š ≥ 0• + 𝑎3—𝑆|š− 𝑆rš˜ + 𝛼ž—𝑆|š− 𝑆rš˜ × 1œ𝑆|š− 𝑆rš ≥ 0• + 𝜔š + 𝜐|š (8.6)
Yijt+l is outcome of interest within three years after application year–i,e, l = 1, 2 or 3. The
outcome depends on the research question being addressed, 𝑆|šis the score of teacher j in year t, 𝑆r| is the cutoff in year t, and 1[𝑆|š− 𝑆r|]is 1 if the teacher j was certified in 𝑡 + 1. Application
year, subject and grade dummies (𝜔|).
This specification has important features and implications. First, it is estimated twice. Once to estimate the full program effect with data for years t 2003 to 2006 and 2009 to 2011, and another to estimate the financial component effect with data for years t 2007 and 2008. In both cases, 𝛼& is the parameter of interest that represents the local average treatment effect. Second, the specification aims to locally approximate the regression functions by using a polynomial of grade 1, as recommended by Gelman and Imbens (2014), to improve the estimation of causal effects. Third, it allowsthe approximation of the two regression functions by differing on either side of the threshold. Fourth, this specification allows to use the rdrobust STATA command to perform the local linear polynomial method to approximate the regression functions within the MSE-optimally bandwidth as suggested by Skovron and Titiunik (2015). This nonparametric
108
approximation uses a triangular kernel to weight observations and to fit a weighted least squares regression of Equation 8.6. The standard errors are clustered at school level.
A parametric estimation method: the alternative specification
In order to show the unbundled program effect on teacher’s behavior and performance, I estimate the following alternative parametric specification:
𝑌¡|šQ& = 𝛼-+ 𝛼&1œ𝑆|š− 𝑆r|š ≥ 0• + 𝑎3—𝑆|š− 𝑆rš˜ + 𝛼ž—𝑆|š− 𝑆rš˜ × 1œ𝑆|š− 𝑆rš ≥ 0• +
𝛽&œ𝑆|š− 𝑆rš≥ 0•𝑃𝑖𝑛š+ 𝛽3—𝑆|š− 𝑆rš˜𝑃𝑖𝑛š+ 𝛽ž𝑃𝑖𝑛š—𝑆|š− 𝑆rš˜ × 1œ𝑆|š− 𝑆rš ≥ 0• + 𝜔š+ 𝜐|š (8.7)
Yijt+1 is the outcome of interest depending on the research question being addressed, Sjt is the score of teacher j in year t, 𝑆r| is the cutoff in year t and Pint = 1 if in 𝑡 + 1 there was a ceremony to give a pin to each teacher who applied in 𝑡 and was certified in 𝑡 + 𝑙. Year, subject and grade dummies (𝜔|) are included.
In this alternative specification, 1 is the effect of the program when only financial incentives
are provided; and 𝛼&+ 𝛽& provides the effect of the program when financial incentives and non-
financial incentives (ceremony and pin) were given. Thus 𝛽& is the additional effect of the non-
financial reward.
It is important to notice that this alternative specification cannot be estimated using STATA command rdrobust because it includes interaction terms needed to estimate the unbundled effect of the program. This implies that the polynomial estimation within the MSE-optimally defined bandwidth cannot be calculated. Instead, a least-squares regression is used with a triangular kernel to weight observations within a certain data-driven neighborhood. In practical terms, this means running Equation 8.7 by using the observations within the same two bandwidths that were
109
automatically calculated when estimating Equation 8.6 by using the CCT method. Therefore, I can recover the bandwidths estimated by Equation 8.6 for the full program, to estimate the full program’s and the pin’s effect by running Equation 8.7. In the same way, I use the bandwidth estimated for the financial component. The standard errors are clustered at school level.
Parametric versus nonparametric method
The alternative specification has important features that make it both similar and different to the robust specification defined by Equation 8.6. They are similar because both aim to locally approximate the two regression functions independently by using a polynomial of grade 1. Despite these similarities, the alternative specification uses pooled data for years 2003 to 2011 in order to test for differences in the program components simultaneously. Another difference has to do with the fact that the alternative specification imposes a parametric form on the unknown regression functions, while the robust specification leaves these functions unspecified and employs nonparametric local polynomial methods for estimation and inference.
Parametric and nonparametric approaches are different but complementary. The nonparametric local linear polynomial approach has three distinctive features:
(i) the bandwidth is chosen in a data-driven way based on nonparametric approximations, (ii) the RD point estimator is asymptotically MSE-optimal, and (iii) inference procedures explicitly incorporate the effects of local parametric misspecification (i.e., nonparametric smoothing bias) (Cattaneo et al., 2017, p. 654).
While the alternative specification—Equation 8.7—represents a parametric method that does not account for misspecification bias in estimation and inference procedures. Nevertheless, and
110
despite the positive features, the robust specification does not allow the interaction needed to estimate the marginal effect of the pin over the financial component of the program. This is why the parametric estimation method is helpful as a complement to identify the unbundled effect.
Robustness Checks
In order to show that the results in the next chapter are not driven by the choices made to estimate the program effect, two strategies are followed. The first one is related to the
specification of the model, while, the second one is related to the sensitivity of the results to the bandwidth that is automatically calculated by the CCT Method.
First, I estimate the robust and alternative specifications with covariates following Equations 8.8 and 8.9 and clustering the standard errors at school level. These are similar to Equations 8.6 and 8.7 but include a vector of covariates (𝑋|š), which might bring efficiency gains (Calonico et al., 2016b). Due to this, the set of covariates are defined by taking into consideration the balance check on teachers’ characteristics as described in Section 8.2. This set of covariates might comprise teacher j’s characteristics such as experience, and gender. In doing so, I apply the same estimation method but with covariates in all estimations except to those balance checks. The results of this exercise are presented in Appendix I.
𝑌¡|šQª = 𝛼-+ 𝛼&1œ𝑆|š − 𝑆r|š ≥ 0• + 𝑎3—𝑆|š− 𝑆rš˜ + 𝛼ž—𝑆|š− 𝑆rš˜ × 1œ𝑆|š − 𝑆rš ≥ 0• + 𝛼«𝑋|šQª+
𝜔š+ 𝜐|šQª (8.8) 𝑌¡|šQª = 𝛼-+ 𝛼&1œ𝑆|š − 𝑆r|š ≥ 0• + 𝑎3—𝑆|š− 𝑆rš˜ + 𝛼ž—𝑆|š− 𝑆rš˜ × 1œ𝑆|š − 𝑆rš ≥ 0• +
𝛽&œ𝑆|š − 𝑆rš≥ 0•𝑃𝑖𝑛š+ 𝛽3—𝑆|š− 𝑆rš˜𝑃𝑖𝑛š+ 𝛽ž𝑃𝑖𝑛š—𝑆|š− 𝑆rš˜ × 1œ𝑆|š − 𝑆rš ≥ 0• +𝛽4𝑋𝑗𝑡+𝑙+
111
The second strategy is a sensitivity analysis of the results to the window length. Given that the bandwidth for each estimation varies, the sample of teachers used to estimate the effect of the program on final and intermediate outcomes also varies. This is because the CCT Method
automatically calculates the data-driven MSE-optimal bandwidth for each estimation done with the different outcomes of interest. Thus, using the CCT Method does not follow the more
traditional way of presenting the RD results, which is using a fixed bandwidth for all estimations. In this context, the sensitivity analysis is done in order to see if the results vary as a function of the bandwidth size in a way that affects its statistical significance. The analysis shows the p- values calculated when testing the null hypothesis on the parameters of interest for a range of pre-determined bandwidths.
More precisely, this sensitivity exercise is constructed in the following way. First, I define a list of values for the bandwidth. Then, for each window I estimate the treatment effect using the robust specification with Equation 8.6. Lastly, I recover the robust p-value of testing the
hypothesis of null treatment effect. Each p-value is plotted against the bandwidth size and presented at the end of the results’ section in Chapter 10. The corresponding plots also highlight the p-values shown in main results helping make a comparison between this p-value found using a data-driven bandwidth and those found when the program effect is estimated using fixed windows for the running variable.
112
Table 8.1. Research questions, parameters, and outcomes.
Research Question Outcome of Interest in Equation 8.6
Parameter of Interest in Eq. 8.6 Outcome of Interest in Eq. 8.7 Parameter of Interest in Eq. 8.7 Question #1, Does
AEP affect teacher’s performance? (Full program effect)
𝑦¡|šQª is student i’s SIMCE Test
Score of AEP teacher j in 𝑡 + 𝑙 for years t 2003 to 2006 and 2009 to 2011, with 𝑙 = 1,2,3.
𝛼& 𝑦¡|šQª is student i’s SIMCE Test
Score of AEP teacher j in 𝑡 + 𝑙 for years t 2003 to 2011, with 𝑙 = 1,2,3. 𝛼&+ 𝛽& Question #3, Does each component of AEP affect teachers’ performance? (Unbundled program effect)
𝑦¡|šQª is student i’s SIMCE Test
Score of AEP teacher j in 𝑡 + 𝑙 for years t 2007 and 2008, with
𝑙 = 1,2,3.
𝛼′& for the
financial component
𝛼& for the
financial component; 𝛽& for
the non-financial component. Question #4.1, Does
AEP affect teachers’ self-confidence? (Underlying mechanism of the full program)
𝑦|šQª is teacher j’s self-
confidence (self-report on how much teacher feels prepared to teach the curriculum that were evaluated by SIMCE) in 𝑡 + 𝑙 for years t 2003 to 2006 and 2009 to 2011, with 𝑙 = 1,2,3. 𝛼& 𝑦|šQª is teacher j’s self- confidence (self-report on how much teacher feels prepared to teach the curriculum that were evaluated by SIMCE) in 𝑡 + 𝑙 for years t 2003 to 2011, with 𝑙 = 1,2,3. 𝛼&+ 𝛽& Question #5.1, Does each component of AEP affect teachers’ self-confidence? (Underlying mechanism of each component of the program) 𝑦|šQª is teacher j’s self-
confidence (self-report on how much teacher feels prepared to teach the curriculum that were evaluated by SIMCE) in 𝑡 + 𝑙 for years t 2007 to 2008, with
𝑙 = 1,2,3.
𝛼′& for the
financial component
𝛼& for the
financial component; 𝛽& for
the non-financial component. Question #4.2, Does
AEP affect teachers’ effort (Underlying mechanism of each component of the program)
𝑦|šQª is the teacher j’s effort
level (proportion of the curriculum that teacher covers
during the academic year, or class preparation time) in 𝑡 + 𝑙
for years t 2002 to 2006 and 2009 to 2011, with 𝑙 = 1,2,3. 𝛼& 𝑦|šQª is the teacher j’s effort level (proportion of the curriculum that teacher covers during the academic year and class preparation) in 𝑡 + 𝑙 for years t 2004 to 2011. 𝛼&+ 𝛽& Question #5.2, Does each component affect teachers’ effort (Underlying mechanism of each component of the program)
𝑦|šQª is the teacher j ’s effort
level (proportion of the curriculum that teacher covers
during the academic year, or class preparation time) in 𝑡 + 𝑙 for years t 2007 and 2008, with
𝑙 = 1,2,3.
𝛼′& for the
financial component
𝛼& for the
financial component; 𝛽& for
the non-financial component. Question #2, Does
the effect of AEP fade-out? (Fade-out of the full program effect)
𝑦¡|šQª is student i’s SIMCE test
score for applicant teacher j in 𝑡 + 𝑙 where l can take the value
1, 2 or three. This is for application years t 2002 to
2006, and 2009 to 2011.
113