realistic expectations for measuring impact, more attention to real-time learn- ing, greater use of geo-coded data, and new approaches to evaluating policy and systems change.
TREND 1: SHARED EVALUATION FRAMEWORKS
Since about 2000, most community change initiatives have developed an evalua- tion framework (often created through a theory of change or logic modeling pro-
cess) through which they articulate goals and strategies and specify measurable interim and long-term outcomes. Evaluators increasingly play a role early on in helping to sharpen the thinking behind these frameworks and refine the success measures based on research-based evidence and examples of what has worked in other communities.
The development of an evaluation framework can engage key stakeholders, discipline strategic thinking, specify accountabilities, and establish shared ex- pectations and realistic timelines (Brown & Fiester, 2007). Indeed, it is often in the process of defining both short- and longer-term outcomes and identifying how they will be measured that stakeholders ultimately agree upon the nature of the work and their own roles in it. Evolving community initiatives with multiple stakeholders and complex, multilevel goals can also use the framework to keep everyone aligned around mutually agreed-upon priorities and accountabilities and to manage expectations over time (Walker & Arbreton, 2004).
A particularly promising development is that foundations increasingly rec- ognize their own critical role in determining the outcomes of community change efforts and therefore include their own performance in the evaluation frame- work (Brown, Colombo & Hughes 2009).
TREND 2: MORE REALISTIC ExPECTATIONS FOR MEASURING IMPACT
Funders and their partners have developed a better understanding of the “at- tribution problem” and the difficulty of drawing a straight causal line between investments in community change and specific outcomes. This more nuanced appreciation of the complex and dynamic nature of community change has led evaluators to focus more on trying to understand how such investments add val- ue and capacity, serve a catalytic role, build on or accelerate existing momentum, help shape relevant resource and policy decisions, leverage new resources and partnerships, and/or contribute to achievement of specific outcomes (Kramer et al., 2007; W.K. Kellogg Foundation, 2007; Grantmakers for Effective Organiza- tions, 2009).
As evaluators have grown more aware of the multiple causal factors at play within the complex ecology of community change—and as their clients have grown more interested in learning how to create change, not just proving that it has occurred (Behrens & Kelly, 2008; Westley et al., 2006)—they have begun to use multiple methods and sources of data to “make a compelling case” that links the change effort with intended outcomes or lack thereof.
Multiple Levels of Focus Capture Results
Vibrant Communities, a Pan-Canadian initiative in 12 communities using multi- sectoral approaches to poverty reduction, developed an extensive learning and
evaluation process to help partners track progress, make adjustments, and build a knowledge base (Leviten-Reid, 2008). One component of the evaluation organizes results according to three levels of action (community capacity for poverty reduc- tion, individual and household assets, and policy and systems change), each with four or five key indicators. For example, the community’s ability to solve problems and take advantage of opportunities to address poverty is assessed periodically in terms of convening capacity, multisectoral leadership, collaboration, and com- munity awareness.
TREND 3: MORE ATTENTION TO REAL-TIME LEARNING
Learning is “a continuous process, a culture, and a commitment to support the capacity of people to reflect on their work in ways that help them see the paths that can lead to ever-improving performance,” as one interviewee said. Evalu- ation in community change work has been increasingly viewed as a means to enhance real-time learning and decision-making, refine strategy, and institute midcourse corrections (Bailey, Jordan, & Fiester, 2006; Walker, 2007). Solicit- ing the opinions, concerns, and priorities of multiple and diverse stakeholders in developing key evaluation questions cultivates ownership of the learning process and increases the likelihood that the results will be useful, relevant, and credible for potential users (Preskill & Jones, 2009).
The iterative process of learning and doing helps to position evaluation as a tool for improving practices and nurturing change at every level of the work. No longer an outsourced function, evaluation becomes the collective responsibility of all stakeholders (Patrizi, 2006). In order to support this process, funders and evaluators must work hard to provide sufficient resources and structures to sup- port learning, and to create a culture that values candid dialogue and analysis and embraces the idea of learning while doing (David, 2006).
Data Improve Effectiveness
Making Connections’ evaluators developed a tool for assessing a community’s ca- pacity to develop and implement powerful change strategies. With site input, they created a developmental assessment scale along various dimensions like “scope,” “scale,” “resident leadership,” and “accountability and use of data.” Evaluators used substantiating evidence to rate each site and then facilitated sessions with site teams to elicit their perspectives, gather more evidence to refine the ratings, and ensure that all players were working toward the same goals. These periodic sessions became safe places to celebrate successes, engage in constructive criticism, and reinforce the role of empirical data from independent sources in helping local teams improve their effectiveness.
TREND 4: GREATER USE OF GEO-CODED DATA
The past decade of evaluation has increasingly recognized the power of geo-coded data to facilitate learning about neighborhood context, assist in community plan- ning, foster informed public discourse, and gain support for the change effort by educating the public about economic and racial disparities. Evaluators can map neighborhood change over time, compare the trajectories of target neigh- borhoods to others, and combine the results with other sources and types of data to explore the impact of community change efforts. Significant investments have been made in building local data intermediaries that democratize access to data for all stakeholders. Some of these data intermediaries are freestanding, while others are associated with city government or a university; many are linked to- gether in a network sponsored by the Urban Institute to “further the develop- ment and use of neighborhood-level information systems in local policymaking and community building” (see www.nnip.org).
Geo-Coded Data Reveal Neighborhood Trajectories
One component of MDRC’s evaluation of the New Communities Program (NCP) is a neighborhood trajectories analysis to measure long-term trends in the quality of life of its 16 target communities. Evaluators examine demographic changes in each neighborhood as well as the nature, extent, and pace of change in such neighbor- hood indicators as crime rates, housing market activity, and commercial vitality. The analysis can describe trends starting some years before NCP and extending a number of years afterward. It can show how trajectories vary across NCP commu- nities and how they compare to changes observed in selected non-NCP neighbor- hoods and for the city of Chicago as a whole.
TREND 5: NEW APPROACHES TO EVALUATING POLICY AND SYSTEMS CHANGE Community change efforts have paid increasing attention to the ways in which place-based change can be connected to policy and systems change in order to achieve large-scale benefits both within targeted communities and more broadly. Evaluators, too, have devoted significant effort to finding more effective ways to assess advocacy and the impact of policy and systems change (Guthrie et al., 2005, 2006; Coffman, 2007; Harvard Family Research Project, 2007; Innovation Network, 2009). New work has even produced an online resource, the Advocacy Progress Planner (www.planning.continuousprogress.org), and a tool that evalu- ators can use to design evaluations of advocacy efforts that are useful, manage- able, and resource efficient (Harvard Family Research Project, 2009).
These new approaches are particularly important for emerging community