During their initial year of exploration and planning, Achieving the Dream colleges are expected to select intervention strategies based on their analyses of student outcomes data. The ideal is that colleges adopt strategies that show promise of addressing priority problems and narrowing achievement gaps, and that they draw upon available research to develop well- informed plans for implementation. Furthermore, once these strategies have been piloted and evaluated (see Chapter 3 for a discussion of strategy evaluation), colleges are expected to make informed decisions about their continuation, refining and scaling up those strategies that yield promising outcomes while improving or eliminating those that do not. The following discussion reveals that the Round 1 colleges made good strides in being more deliberative in their strategy development, but most still have work ahead of them to fully utilize data to make decisions about these interventions.
Data and Strategy Development
MDRC’s baseline research in spring 2006, during the Round 1 colleges’ first year of implementation following their planning year, revealed that colleges had mixed success using data to develop strategies. On the one hand, they tended to honor the spirit of Achieving the Dream by reflecting broadly on priority problems in student performance. About half of the Round 1 colleges conducted some form of review of their longitudinal student data to identify priority areas for reform, and a majority drew from qualitative focus group data to home in on students’ perceived barriers to success.21
Furthermore, as illustrated by the high proportion of strategies that were targeted intentionally toward developmental education students and first- year students, colleges frequently chose broadly defined focus areas for their strategies that related in some way to the problems they identified.
On the other hand, only about one-third of the colleges were able to draw direct ties be- tween their analysis of outcomes data and their selection of specific strategies at baseline. This finding is not surprising in light of the fact that most of the Round 1 colleges were still very early in their development of a culture of evidence when they began their planning years in fall 2004.22 They generally did not have the data capacity to undertake quantitative analyses that
21
Brock et al. (2007).
22
As cited in Chapter 3, only 8 of the original 27 colleges had the data capacity needed to develop a culture of evidence when they began in Achieving the Dream.
would be sufficiently sophisticated and specialized to yield meaningful information for strategy selection. Instead, a few colleges conducted literature reviews during their planning year, but they were confronted with a paucity of experimental research documenting effective interven- tion strategies at community colleges.23 The result was that much of the Round 1 colleges’ early strategy selection was based on qualitative knowledge of “best practices” gleaned from Achiev- ing the Dream conferences, as well as ideas for intervention strategies that predated colleges’ participation in the initiative.24
In addition, as presented in the discussions of conference attendance and professional development in Chapter 4 and this chapter, many colleges relied upon the faculty and staff who were charged with leading their interventions to gather know- ledge about how to design and implement them. This knowledge generally came from the Strategy Institutes, site visits to other colleges, and strategy-specific conferences and trainings.
Data and Strategy Refinement
By spring 2009, over one-third of the Round 1 colleges were making effective use of student outcomes data to refine strategies. These colleges tended to be above average in their institutional research capacities and in the strength of their strategy evaluations, both of which facilitated their efforts to gather reliable, strategy-specific data that could be used to improve their interventions. For example, one college developed a formal system in which task forces continuously reviewed data and made improvements to select strategies. Another achieved a similar process of continuous, data-based improvements through its evaluation of a new student orientation program. After distributing surveys during the pilot year of this strategy, college personnel responded to student feedback by integrating registration activities into the orientation sessions. They subsequently expanded and mandated this program based on promising patterns of persistence among orientation attendees. A third example of applied strategy evaluation occurred when a supplemental instruction program was refined and expanded based on promis- ing findings from a simple comparison evaluation.
Though some colleges had notable accomplishments in strategy evaluation and refine- ment ʊ which they attributed in large part to their participation in Achieving the Dream ʊ most still demonstrated significant room for improvement. Half of the Round 1 colleges made some strategic use of data but fell short in the consistency and rigor of their data usage. For example, few reported making decisions about strategy scale-up that were explicitly guided by their evaluations. Furthermore, a small handful of colleges showed little or no evidence of
23
Scrivener and Weiss (2009), p. ix. It was this need for high-quality evidence about strategy effectiveness that motivated MDRC’s four random assignment studies under Achieving the Dream. Unfortunately, though, the results of these studies were not available to the Round 1 colleges during their early years of participation in the initiative.
24
applying their evaluation results to strategy refinements, which is not surprising given that all of them continued to struggle with the evaluation process itself (as discussed at greater length in Chapter 3).
Summary
Colleges can expect real changes in student outcomes only when they extend meaning- fully improved programs and services to significant numbers of students. In other words, a strategy will make an observable impact on institutional performance only if it (1) raises academic outcomes among the students who are served, and (2) reaches enough students to “move the needle” on collegewide measures.
The analysis in this chapter reveals that colleges implemented a large number of strate- gies representing an enormous investment of time and work by college personnel — an average of seven strategies at each college and a total of more than 200 strategies at all 26 colleges. They also made noteworthy progress toward scale-up, as nearly all succeeded in expanding at least one direct strategy to reach at least 25 percent of its intended target population. However, a majority of strategies across the Round 1 colleges remained small in scale, particularly when they involved the kind of intensive contact that might be expected to meaningfully influence students’ performance. The result is that the benefits of promising interventions were frequently extended only to small pockets of students.
The difficulty that colleges experienced in scaling up strategies raises two important questions for the initiative. First, would Achieving the Dream institutions be better served if they aimed to implement fewer strategies? The Achieving the Dream model encourages colleges to pilot new ideas before evaluating and selecting those that prove to be most effective. However, some colleges may have overextended themselves in their planning stages, making it difficult for them to improve and expand even their most promising interventions. In the words of one Round 1 college president, “Part of the reason we have not accomplished as much as we could is that from the beginning, we set out to do too many things.” The college in question made important progress in implementing over a dozen direct strategies for policy change and professional development, but even those strategies represented only a fraction of the vast number of interventions it had envisioned under Achieving the Dream. Overreaching was a common experience among the Round 1 colleges, with many that were unable to continue with all of their planned strategies throughout their participation in the initiative. They found that both staffing and funding were stretched too thin across the multitude of interventions they wanted to pursue.
The second question is focused not on colleges’ approach, but rather on the initiative’s model itself: how can the Achieving the Dream partners help colleges to identify effective
strategies that can be scaled up at low cost, or else to locate supplemental funding for scale-up? The level of funding provided by the initiative was instrumental in jump-starting colleges’ strategic work, but it was rarely sufficient to bring their ideas to scale. It will be important for the initiative partners and colleges to work together to identify practical approaches and funding streams to support this work.
In conclusion, the colleges’ challenges with strategy scale-up underline the manifold demands of Achieving the Dream’s theory of improvement. While the Round 1 colleges have made important advances in strategy development and implementation, their struggles with scale-up have important implications for their ability to make headway on the institutional performance measures presented in Chapter 8.
Chapter 6