Difference-in-Differences and Causal Inference

Taken together, the results from the early literacy activities analysis could be used to make the argument for justifying programs that encourage parents to engage both girls and boys (and especially boys) more in early literacy activities. The analyses from Phase 1-5 triangulate evidence that early literacy activities are positively related to student reading achievement at the fourth grade across cross-sectional and longitudinal approaches and across different levels of aggregation. The evidence from Phase 7 suggests that parents engage boys less frequently in such activities, and this could contribute to differences in reading achievement at the fourth grade. Likewise, the evidence in Phase 6 suggests that when boys and girls are engaged in early literacy activities their engagement tends to have a similar relationship with reading

152 Nevertheless, causal claims should be tempered because the models do not control for time-varying omitted variables. Classical difference-in-differences allows for causal inference when the common trend assumption is met, because showing common trends provides strong evidence that other time-varying factors are not influencing the outcome variable. From Figure 2.8 from Chapter 2, it can be seen that the common trend assumption is not supported across countries when examining PIRLS data since the country trend lines do not appear to be parallel.

Without fulfilling the common trend assumption, it becomes difficult to dismiss the influence of time-varying covariates as innumerable factors could lead to increases or decreases in student achievement across countries. One step in the right direction would be to include a number of measured covariates in the analysis. An advantage of the subpopulation approach over the country-level approach in this regard is that the subpopulation approach offers a larger

sample size and therefore more degrees of freedom for including such time-varying covariates. Nevertheless, for analysis across countries, it remains nearly impossible using either the country or subpopulation groupings to control for all plausible influences by including measured covariates in the model. One possible extension on the subpopulation approach would be to implement sensitivity analysis (Montgomery, Richards, & Braun, 1986; Rosenbaum & Rubin, 1983). In sensitivity analysis, a hypothetical omitted variable is generated that is related to both the explanatory variable of interest for the analysis and the outcome variable. By generating this theoretical variable, which has varying correlations with both the explanatory and outcome variables, and inserting this hypothetical variable into the analysis model as a covariate, researchers are able to approximate the relationships the omitted variable must have with the explanatory and outcome variables to change the direction, magnitude, and/or significance of the coefficient estimate of interest.

153 Another possibility for strengthening causal inference through these longitudinal

approaches is to focus analysis on countries that tend to have parallel-trend lines. Similar to economists assuming that the economies of adjacent states have common trends, it may be possible to assume that common trends are expected for countries that are from the same geographic region and have shown common trend in the past. Referring back to Figure 2.8, Singapore and Hong Kong have a very similar trend line from PIRLS 2001 through PIRLS 2011. If Singapore were to have introduced a new education policy in 2012, one could assume common trend with Hong Kong and examine the efficacy of the policy by comparing the deviations in the trend line between PIRLS 2011 and PIRLS 2016.

It should be kept in mind that although the Phase 7 mediation analysis provides additional perspectives on the relationships in the data, it provides a similar level of causal evidence (or lack their of) to the country and subpopulation difference-in-differences approaches in Phase 3 and Phase 4. The advantage of the mediation modeling approach in Phase 7 is that it provides an explanation for girls’ advantage on the PIRLS reading assessment. Because there was no random assignment, however, this explanation is still primarily dependent on the theory that more

engagement in early literacy activities caused higher reading achievement. To draw causal conclusions it would still be necessary to dismiss other possible explanations for the relationship between increased participation in early literacy activities and increased reading achievement.

From a causal perspective, it should be noted that more could be done in the Phase 7 model to strengthen causal inferences by controlling for omitted variables, such as adding time- varying covariates. More demographic characteristics could also be controlled for in the random effects approach. Because the current analysis used the sandwich estimator in Mplus to estimate cluster-robust standard errors, the number of parameters estimated could not exceed the number

154 of clusters—equal to the 21 countries in the analysis—and therefore there was a limit to the number of covariates that could be added to the analysis. In more recent cycles of TIMSS and PIRLS, participation has reached around 50 countries at the fourth grade, opening up the possibility for creating even more complex models through the subpopulation approach and controlling for additional covariates.