Final Report on the Performance-Based Scholarship Demonstration
contextual features at the colleges themselves — varied as well, and cannot be untangled from the effects of services. It may be that the additional services did not improve student outcomes, but it is also possible that the services did help students to do better than they would have otherwise. When program designers included these services, moreover, they did so because they believed the services would help students overcome barriers that inhibited academic progress. Appendix Table A.6 shows the full results of the analyses.
In California, the evaluation was designed to test whether scholarships that varied by duration and amount produced different impacts. Five different performance-based scholarship types were tested against a control condition in which students did not receive a scholarship and against a $1,000 scholarship in one semester that was not performance-based. The performance-based scholarships ranged from a potential maximum award of $1,000 to $4,000 and lasted between one semester and four semesters. Analyses of these scholarship types also provide little evidence that the different scholarships produced different impacts, but the analyses had limitations. For one, students could take the scholarship to any college, but for a large portion of the sample, transcript data were not available. Consequently, evidence of different impacts on credit accumulation — the outcome where PBS programs have shown the most consistent evidence of effects — could not be examined for the full sample. The California study was also designed to detect differences between scholarship types that are larger than the pooled impact estimates for enrollment and graduation reported above. To detect differences due to duration, for example, scholarships that last for two years would need to produce a gain of at least 6 percentage points over scholarships that last for one year or less. Increasing the sample size to detect smaller impacts, however, was cost-prohibitive. Consequently, it is possible that differences in duration and amount could produce meaningful differences in impacts.44
Appendix Tables A.7 (enrollment) and A.8 (degrees earned) show impact estimates for each scholar- ship type and the p-values for the F-tests for each year.
•
The impacts do not appear to differ across different subgroups of students.In each year of follow-up, for each of the three outcomes, statistical tests examined whether impacts were different for student subgroups defined by gender, whether the students were Latino, whether the students were parents, whether the students were younger than 20 years old, whether they were the first in their family to attend college, and whether they were employed at the beginning of the 44. For more information about the California program and evaluation, see Richburg-Hayes et al. (2015).
“I PROBABLY WOULDN’T HAVE KNOWN ABOUT THE MATH LAB IF NOT FOR MAPS [HILLSBOROUGH’S PBS PROGRAM]. . . . BEFORE THIS PROGRAM, I TOOK . . . THREE YEARS OFF, AND I TRIED AN ONLINE COURSE AND I WAS ON MY OWN WITHOUT THE MATH LAB.”
—A STUDENT IN FLORIDA
program. These analyses provide little evidence of variation in impacts and suggest that the pro- grams were similarly effective across different groups of students.45 Notably, the estimated impacts
on credits earned are positive and statistically significant for all student subgroups in each of the first two years. Appendix Tables A.9 (enrollment), A.10 (credits earned), and A.11 (degrees earned) provide more detail.
In many cases, particular groups of students were expressly targeted by the programs. Consequently, these subgroups are correlated with the programs themselves, so the relationships between subgroups and outcomes are difficult to untangle from other factors that vary, such as context and program characteristics.
Summary
These findings suggest that the performance-based scholarship programs generally worked as de- signed. The analyses also detected little variation in the impact estimates, although the evaluation was not designed to determine whether some program configurations were more effective than others. Instead, program designers had flexibility to create scholarship programs to meet the needs of their student populations. It is possible that programs that included services may have been more effective because of those services. The implementation data discussed previously, for example, suggest that students used additional services when they were included as program components.
The programs’ scholarships covered a relatively small proportion of students’ full cost of college during the program, and even less when considering the full follow-up period during which stu- dents continued to attend college. However, the analyses presented here provide evidence that the performance-based scholarships helped students make greater academic progress toward their de- grees and modestly improved degree receipt. This impact appears to have occurred because the PBS programs encouraged students to take additional classes, but also because students passed more of the classes they took. The scholarships provided financial support, and in some cases the programs required students to use additional services that were intended to provide academic support, suc- cessfully increasing students’ participation in those services. In three cases, the programs reduced students’ debt, a growing issue in national conversations about the costs of college. Notably, the analyses described here also suggest that the performance-based scholarship programs were similarly effective across the states and for different student subgroups.