Measure Nonprofit Performance

(1)

Building a Common Outcome Framework

to Measure Nonprofit Performance

December 2006

The Center for What Works The Center for What Works 3074 West Palmer Boulevard 3074 West Palmer Boulevard Chicago, Illinois 60647 Chicago, Illinois 60647 (773) 398-8858

(773) 398-8858 The Urban Institute

The Urban Institute 2100 M Street, NW 2100 M Street, NW Washington, Washington, D.C. D.C. 2003720037 (202) 833-7200 (202) 833-7200

(2)

Preface

It seems clear that nonprofit organizations need to regularly collect information on the It seems clear that nonprofit organizations need to regularly collect information on the effect of their services, if they want to continue to attract funds from foundations, government, effect of their services, if they want to continue to attract funds from foundations, government, and individual donors.

and individual donors. And even more important, these And even more important, these data can help them data can help them manage theirmanage their resources to maximize the services they provide and continuously improve their offerings. resources to maximize the services they provide and continuously improve their offerings.

With funders increasing pressure to set up measurement systems, sometimes the worse With funders increasing pressure to set up measurement systems, sometimes the worse case scenario has emerged—nonprofits with multiple projects and multiple funders have to deal case scenario has emerged—nonprofits with multiple projects and multiple funders have to deal with different requirements for tracking

with different requirements for tracking outcomes for similar outcomes for similar programs. programs. If agreement on If agreement on aa common core set of outcome indicators can be reached, then outcome reporting can be efficient common core set of outcome indicators can be reached, then outcome reporting can be efficient and focused.

and focused. Even more Even more important, successful practices could be important, successful practices could be identified across similaridentified across similar programs and organizations and then shared so that outcomes could be improved.

programs and organizations and then shared so that outcomes could be improved.

The work described in this report first provides suggested core indicators for 14 The work described in this report first provides suggested core indicators for 14

categories of nonprofit organizations and then expands the notion of common core indicators to a categories of nonprofit organizations and then expands the notion of common core indicators to a much wider variety of programs by suggesting a common framework of outcome indicators for much wider variety of programs by suggesting a common framework of outcome indicators for all nonprofit programs.

all nonprofit programs. This can provide guidance to This can provide guidance to nonprofits as they figure out what nonprofits as they figure out what toto measure and how to do it and will work to ease the looming reporting nightmare that will occur measure and how to do it and will work to ease the looming reporting nightmare that will occur unless a common framework for

unless a common framework for outcome measurement emerges. outcome measurement emerges. Further research is needed toFurther research is needed to further test and revise the existing core indicators for the selected programs, add core indicators further test and revise the existing core indicators for the selected programs, add core indicators for more program areas, and expand and revise the common framework for more general

for more program areas, and expand and revise the common framework for more general guidance.

guidance.

We hope the initial material presented here will be helpful and act as a catalyst for further We hope the initial material presented here will be helpful and act as a catalyst for further work in this crucial area.

work in this crucial area.

Elizabeth T. Boris, Director Elizabeth T. Boris, Director

Center for Nonprofits and Philanthropy Center for Nonprofits and Philanthropy The Urban Institute

(3)

Building a Common Outcome Framework

To Measure Nonprofit Performance

Introduction

For most stakeholders in the n

For most stakeholders in the nonprofit sector, measuring performonprofit sector, measuring performance is elusive. ance is elusive. NonprofitNonprofit managers and staff, funders, board members, potential clients, and members of the public seeking managers and staff, funders, board members, potential clients, and members of the public seeking information are often frustrated by lengthy academic evaluations and complex, meaningless information are often frustrated by lengthy academic evaluations and complex, meaningless statistical analysis.

statistical analysis. At the same time, there is increasing At the same time, there is increasing pressure on nonprofits to accpressure on nonprofits to account for andount for and improve results.

improve results. Although classic program evaluation Although classic program evaluation is one response, practitioners and is one response, practitioners and funders alsofunders also need the tools, capacity, and standards to track and measure their own performance.

need the tools, capacity, and standards to track and measure their own performance.

With little actual information, practitioners base decisions primarily on narrative annual With little actual information, practitioners base decisions primarily on narrative annual reports, anecdotes, related social science research and journal articles, IRS Forms 990, and reports, anecdotes, related social science research and journal articles, IRS Forms 990, and administrative metrics (such as the percentage of budget spent on administration or fundraising). administrative metrics (such as the percentage of budget spent on administration or fundraising). Often, information from these sources is not timely, offers little analytical or predictive value and is Often, information from these sources is not timely, offers little analytical or predictive value and is hard to aggregate or synthesize to

hard to aggregate or synthesize to help improve services. help improve services. It is, therefore, of limited value to the staIt is, therefore, of limited value to the staff ff members actually delivering services.

members actually delivering services.

While the concept of measuring performance is not new,

While the concept of measuring performance is not new,11 the development of practical ways tothe development of practical ways to imple

implement ment actual actual measumeasures isres is. . Progress in understanding Progress in understanding how to think about how to think about performance hasperformance has been made.

been made. For example, there are mFor example, there are many handbooks on outcome any handbooks on outcome measurement, logic models,measurement, logic models, rating services, and assessment tools, but how much performance data have actually been rating services, and assessment tools, but how much performance data have actually been coll

collecteected and used? d and used? Citing the Citing the diversity of diversity of nonprofit nonprofit work, some work, some ssccholars have even concluded thatholars have even concluded that systemically measuring impact in the nonprofit sector is impossible.

systemically measuring impact in the nonprofit sector is impossible. 22 A convergence of forces,A convergence of forces, however, including increased government oversight, the call for greater accountability from various however, including increased government oversight, the call for greater accountability from various stakeholders, more professional nonprofit management, and competition for funding is accelerating stakeholders, more professional nonprofit management, and competition for funding is accelerating the need

the need to overcoto overcome bame barriers to rriers to measurement. measurement. In addition, In addition, advanadvances in compuces in computer techter technolognology nowy now permit performance data to more easily be collected and processed.

permit performance data to more easily be collected and processed.33

Some of the impetus for enhancing accountability for nonprofits and their performance Some of the impetus for enhancing accountability for nonprofits and their performance comes as a response to recommendations to the Senate Finance Committee by the Panel on the comes as a response to recommendations to the Senate Finance Committee by the Panel on the Nonprofit Sector (established by Independent Sector)

Nonprofit Sector (established by Independent Sector) in May in May 2005. 2005. The Panel The Panel recommendedrecommended that, as a best practice, charitable organizations establish procedures for measuring and

that, as a best practice, charitable organizations establish procedures for measuring and evaluating their program accomplishments based

evaluating their program accomplishments based on specific goals on specific goals and objectives. and objectives. In addition,In addition, the Panel recommended a sector-wide effort to provide information and training focused on the Panel recommended a sector-wide effort to provide information and training focused on appropriate methods for program evaluation.

appropriate methods for program evaluation.

1 1

See, for example,

See, for example, Measuring the Impact of the Nonprofit Measuring the Impact of the Nonprofit Sector Sector , Patrice Flynn and Virginia , Patrice Flynn and Virginia Hodgkinson, eds, Kluwer AcademicHodgkinson, eds, Kluwer Academic Press, 2001; “Measuring Program Outcomes: A Practical Approach, United Way of America, 1996; and

Press, 2001; “Measuring Program Outcomes: A Practical Approach, United Way of America, 1996; and “Why Measure“Why Measure Performance?” by Robert D. Behn, Harvard University, Kennedy School of Government, January 22, 2002.

Performance?” by Robert D. Behn, Harvard University, Kennedy School of Government, January 22, 2002.

2 2

Paul DiMaggio, “Measuring the Impact of the

Paul DiMaggio, “Measuring the Impact of the Nonprofit Sector on Society is Probably Impossible but Possibly Useful” inNonprofit Sector on Society is Probably Impossible but Possibly Useful” in Measuring Measuring the Impact of the Nonprofit

the Impact of the Nonprofit Sector Sector

3 3

“Performance Measurement: Getting Results,” interview with Harry Hatry,

“Performance Measurement: Getting Results,” interview with Harry Hatry, the Urban Institute atthe Urban Institute at http://www.urban.org/pubs/pm/author.html

(4)

While it appears unlikely that there will be detailed federal legislation that calls for While it appears unlikely that there will be detailed federal legislation that calls for

performance reporting, stakeholders are paying attention to assessing effectiveness and the need performance reporting, stakeholders are paying attention to assessing effectiveness and the need for improved measurement

for improved measurement and tracking of and tracking of nonprofit outcomes. nonprofit outcomes. Having a standard Having a standard framework framework for developing outcomes and indicators can help create important tools for the sector to better for developing outcomes and indicators can help create important tools for the sector to better communicate the value of its services.

communicate the value of its services.

About the Common Outcome Framework Project

The Urban Institute and its project partner, The Center for What Works, collaborated The Urban Institute and its project partner, The Center for What Works, collaborated from June 2004 through May 2006 to identify a set of common outcomes an

from June 2004 through May 2006 to identify a set of common outcomes an dd outcome indicatorsoutcome indicators or “common framework” in the measurement of performance for nonprofits.

or “common framework” in the measurement of performance for nonprofits.44 The work beganThe work began based on a recognition that nonprofit organizations often have limited capacity or resources for based on a recognition that nonprofit organizations often have limited capacity or resources for collecting, analyzing, and using data

collecting, analyzing, and using data to inform to inform practice. practice. However, funders are increasinglyHowever, funders are increasingly demanding such practice.

demanding such practice. This project has This project has attempted to identify attempted to identify a more a more standardized approachstandardized approach for nonprofits, themselves, as well as the organizations that choose to fund their efforts.

for nonprofits, themselves, as well as the organizations that choose to fund their efforts.

To meet this need, the research team selected, and then examined, 14 separate program To meet this need, the research team selected, and then examined, 14 separate program areas as to their missions, the outcomes they sought, and potential outcome indicators for areas as to their missions, the outcomes they sought, and potential outcome indicators for tracking progress towards

tracking progress towards these programs’ mthese programs’ missions. issions. The programs The programs of nonprofit of nonprofit organizationsorganizations almost always have multiple outcomes and require a number of outcome indicators—both those almost always have multiple outcomes and require a number of outcome indicators—both those that measure “intermediate” (usually early) outcomes and those that measure “end” outcomes. that measure “intermediate” (usually early) outcomes and those that measure “end” outcomes. The team developed sample “outcome sequence charts” for each of the 14 programs to portray The team developed sample “outcome sequence charts” for each of the 14 programs to portray the sequence of these outcomes.

the sequence of these outcomes.

The 14 programs included in this project represent only a small proportion of the great The 14 programs included in this project represent only a small proportion of the great variety of programs that exist.

variety of programs that exist. Therefore, as a final task, we Therefore, as a final task, we developed a common developed a common framework forframework for outcomes, one that might provide other programs with a starting point for identifying outcomes outcomes, one that might provide other programs with a starting point for identifying outcomes and outcome indicators for themselves.

and outcome indicators for themselves.

We hope that this guidance can help nonprofit organizations reduce their time and cost We hope that this guidance can help nonprofit organizations reduce their time and cost of implementing an outcome measurement process and improve its quality.

of implementing an outcome measurement process and improve its quality.

With improved and more consistent reporting from grantees, funders, too, would be With improved and more consistent reporting from grantees, funders, too, would be better able to assess and compare the results of their grants.

better able to assess and compare the results of their grants.

An outcome sequence chart for the project is shown as Exhibit 1. An outcome sequence chart for the project is shown as Exhibit 1.

4

4_{Project support was provided primarily from the Hewlett Foundation with}_{Project support was provided primarily from the Hewlett Foundation with additional support from the Cisco and}_{additional support from the Cisco and}

Kellogg Foundations. Kellogg Foundations.

(5)

Outcome Sequence Chart:

Outcome Sequence Chart: Creating a Common Framework Creating a Common Framework for Measuring Performancefor Measuring Performance

Identify, classify Identify, classify

and

and dissemidisseminatenate relevant, useful relevant, useful performance performance indicators for indicators for nonprofit work nonprofit work Better performance Better performance indicators and indicators and outcome data outcome data Identificati Identification on of of common program common program and organizational and organizational indicators across indicators across program areas program areas Better Better benchmarks benchmarks and and comparisons comparisons across across programs and programs and organizations organizations Better data Better data on what on what works works Smarter decisions Smarter decisions about allocation about allocation of resources, of resources, stronger stronger management, management, improved improved program design program design through through identificati identification oon o f f effective practices effective practices Better Better organizational organizational efficiency efficiency Organizations Organizations fulfill their fulfill their missions missions Clients are Clients are better off better off More effective More effective and impactful and impactful programs programs Better Better resources for resources for grant makers grant makers

About This Report

This project description has been prepared so that the current results can be used as a This project description has been prepared so that the current results can be used as a resource for nonprofit organizations and

resource for nonprofit organizations and their funders. their funders. Although the materials presented Although the materials presented are notare not complete—without the necessary next steps of testing, refining, and expanding to more program complete—without the necessary next steps of testing, refining, and expanding to more program areas—we feel they can offer guidance and help nonprofits and grantmaking organizations in areas—we feel they can offer guidance and help nonprofits and grantmaking organizations in developing their outcome measurement programs.

developing their outcome measurement programs. The information is presented in four parts:

The information is presented in four parts: Part 1: Project Approach

Part 1: Project Approach

Part 2: Candidate Outcomes, Outcome Indicators, and Outcome Sequence charts for Part 2: Candidate Outcomes, Outcome Indicators, and Outcome Sequence charts for

Specific Programs Specific Programs

Part 3: Draft Common Outcome Framework Part 3: Draft Common Outcome Framework

Part 4: Tips on Using the Common Framework Project Materials Part 4: Tips on Using the Common Framework Project Materials

(6)

Part 1

Part 1 Project Approach

Project Approach

While there is no shortage of outcomes and their indicators in some program areas, there is While there is no shortage of outcomes and their indicators in some program areas, there is no centralized grouping of them or assessment of their quality that could serve as a resource for no centralized grouping of them or assessment of their quality that could serve as a resource for organizations that wish to develop outcome m

organizations that wish to develop outcome measurement systems. easurement systems. And because of the And because of the vastvast range of programs in the social sector, major gaps exist in the outcome indicators that have been range of programs in the social sector, major gaps exist in the outcome indicators that have been developed.

developed. This project took a This project took a first step in attempting to provide first step in attempting to provide a resource for quality indicatorsa resource for quality indicators and also provide guidance for nonprofits on the development of good indicators, if indicators for and also provide guidance for nonprofits on the development of good indicators, if indicators for their specific programs are not yet available.

their specific programs are not yet available.

First, we chose a number of specific program areas and identified program outcomes and First, we chose a number of specific program areas and identified program outcomes and indicators already in use and/or recommended.

indicators already in use and/or recommended. Outcomes are defined as Outcomes are defined as the results of a the results of a programprogram or service that is

or service that is of direct interest and concern to of direct interest and concern to customers of the program. customers of the program. Outcomes areOutcomes are distinguished from program outputs, which while important to the program, are primarily of distinguished from program outputs, which while important to the program, are primarily of internal use and not of direct concern to customers (such as the number of training sessions internal use and not of direct concern to customers (such as the number of training sessions provided to staff).

provided to staff).

It is often difficult to measure outcomes directly, so many indicators are proxies. For It is often difficult to measure outcomes directly, so many indicators are proxies. For

example, while tracking the avoidance of a certain kind of behavior can be difficult, a client can example, while tracking the avoidance of a certain kind of behavior can be difficult, a client can be tested about a

be tested about a level of knowledge about why level of knowledge about why someone should avoid that behavior. someone should avoid that behavior. However,However, evidence that the degree to which increased knowledge leads to the desired change in behavior evidence that the degree to which increased knowledge leads to the desired change in behavior must be strong before this increased knowledge is deemed a “good” indicator of the desired must be strong before this increased knowledge is deemed a “good” indicator of the desired change in behavior.

change in behavior.

Information was collected from a wide range of sources, from national nonprofit umbrella Information was collected from a wide range of sources, from national nonprofit umbrella groups in the US, national accreditation agencies in specific fields, and from national nonprofits groups in the US, national accreditation agencies in specific fields, and from national nonprofits with local affiliates.

with local affiliates. Outcomes and outcome Outcomes and outcome indicators were assessed as indicators were assessed as to which ones to which ones werewere useful, relevant, and feasible.

useful, relevant, and feasible. It is important to consider outcome It is important to consider outcome information that is not usuallyinformation that is not usually being currently collected but

being currently collected but should be. should be. A highly useful A highly useful basis for developing basis for developing quality indicatorsquality indicators we found to be outcome sequence chart (based on the logic model format) for the program—the we found to be outcome sequence chart (based on the logic model format) for the program—the sequence of a program’s outputs, intermediate (earlier) outcomes, and the ultimate desired end sequence of a program’s outputs, intermediate (earlier) outcomes, and the ultimate desired end outcomes?

outcomes? Once the desired outcomes are Once the desired outcomes are identified, with the help of the identified, with the help of the outcome sequenceoutcome sequence charts, appropriate outcome indicators for measuring progress toward those outcomes can then charts, appropriate outcome indicators for measuring progress toward those outcomes can then be identified.

be identified.

Basic criteria for quality indicators included ones that were:

Basic criteria for quality indicators included ones that were: specificspecific(unique, unambiguous);(unique, unambiguous);

observable

observable (practical, cost effective to collect, measurable);(practical, cost effective to collect, measurable); understandable (understandable (comprehensible);comprehensible);

relevant (

relevant (measured important dimensions, appropriate, related to program, of significance,measured important dimensions, appropriate, related to program, of significance, predictive, timely);

predictive, timely); time bound (covered a specified period of time)time bound (covered a specified period of time); and; and valid valid (provided reliable,(provided reliable, accurate, unbiased, consistent, and verifiable data)

accurate, unbiased, consistent, and verifiable data)

The characteristics of a successful taxonomy or common framework were also reviewed. The characteristics of a successful taxonomy or common framework were also reviewed. The most useful tend to reflect the manner in which the sector organizes, collects, and reports the The most useful tend to reflect the manner in which the sector organizes, collects, and reports the

(7)

information.

information. Although essential principles Although essential principles of of comprehensiveness, mutual exclusivity comprehensiveness, mutual exclusivity of of elements, and logical consistency must be followed, there must be a grounding in what is elements, and logical consistency must be followed, there must be a grounding in what is actually in use by

actually in use by practitioners and what has worked for practitioners and what has worked for the specific program areas. the specific program areas. Thus,Thus, testing by stakeholders (including nonprofit staff; funders, both public and private; clients, testing by stakeholders (including nonprofit staff; funders, both public and private; clients, participants, and service users; and even the public, where appropriate) is vital.

participants, and service users; and even the public, where appropriate) is vital.

Outcomes and indicators were collected for fourteen different program areas to help inform Outcomes and indicators were collected for fourteen different program areas to help inform the development of the

the development of the common framework. common framework. Lists of quality outcomes and Lists of quality outcomes and their indicators weretheir indicators were selected for program areas ranging from emergency shelter to youth mentoring to health risk selected for program areas ranging from emergency shelter to youth mentoring to health risk reduction programs.

reduction programs. The The 14 14 program program areas areas are are listed in listed in Part Part 2. 2. The The outcomes outcomes identified foridentified for these 14 programs were then reviewed for common elements, which then became the basis of the these 14 programs were then reviewed for common elements, which then became the basis of the draft of the common framework, described in Part 3.

draft of the common framework, described in Part 3.

The project efforts and products to date represent the completion of our first phase of work. The project efforts and products to date represent the completion of our first phase of work. More work is highly desirable to further refine, test, and expand the outcomes framework to More work is highly desirable to further refine, test, and expand the outcomes framework to increase its relevance and

increase its relevance and maximize its potential utility for maximize its potential utility for the sector. the sector. The following tasksThe following tasks represent next steps:

represent next steps:

 Development of an interactive website tool, including references to sample dataDevelopment of an interactive website tool, including references to sample data

collection instruments and protocols, “build-your-own” outcome sequence charts, etc. collection instruments and protocols, “build-your-own” outcome sequence charts, etc. Linking the outcome indicators to actual existing data collection instruments, such as Linking the outcome indicators to actual existing data collection instruments, such as questionnaires or interview protocols would considerably increase the helpfulness of this questionnaires or interview protocols would considerably increase the helpfulness of this material.

material.

 Expanding the number of program areas for which candidate outcomes, outcomeExpanding the number of program areas for which candidate outcomes, outcome indicators, and outcome sequence charts are available on the Internet.

indicators, and outcome sequence charts are available on the Internet.  Refinement of the outcomes framework by adding common indicators.Refinement of the outcomes framework by adding common indicators.

 Developing program performance outcomes and indicators for internal organizationalDeveloping program performance outcomes and indicators for internal organizational strategy, including:

strategy, including: management management effectiveness, financial sustainability, and effectiveness, financial sustainability, and communitycommunity engagement.

(8)

Part 2

Part 2 Candidate Outcomes, Outcome Indicators, and Outcome Sequence

Candidate Outcomes, Outcome Indicators, and Outcome Sequence

Charts for Specific Programs

The 14 nonprofit program areas selected for detailed analysis emphasized health and human The 14 nonprofit program areas selected for detailed analysis emphasized health and human services but also included some programs that extend beyond the typical client-centered services services but also included some programs that extend beyond the typical client-centered services to broader community

to broader community outcomes and interests. outcomes and interests. They include the They include the following:following:

 Adult Education and Family LiteracyAdult Education and Family Literacy

 AdvocacyAdvocacy

 Affordable HousingAffordable Housing

 Assisted LivingAssisted Living

 Business AssistanceBusiness Assistance

 Community OrganizingCommunity Organizing

 Emergency ShelterEmergency Shelter

 Employment TrainingEmployment Training

 Health Risk ReductionHealth Risk Reduction

 Performing ArtsPerforming Arts

 Prisoner Re-entryPrisoner Re-entry

 Transitional HousingTransitional Housing

 Youth MentoringYouth Mentoring

 Youth TutoringYouth Tutoring

For each, we developed a program description, an outcome sequence chart, and detailed For each, we developed a program description, an outcome sequence chart, and detailed spreadsheets with associated program outcomes and outcome indicators, described in more detail spreadsheets with associated program outcomes and outcome indicators, described in more detail below.

below. This material for each This material for each of the above of the above 14 program areas is 14 program areas is provided atprovided at http://

http:// wwwwww.urban.org/center/cnp/commonindicators.cfm.urban.org/center/cnp/commonindicators.cfm

Program description:

Program description: A short paragraph A short paragraph illustrating the types of illustrating the types of programs included.programs included.

Includes the scope or coverage of activities included or excluded from consideration. Includes the scope or coverage of activities included or excluded from consideration. Appears on both the outcome sequence chart and the indicators spreadsheet for each Appears on both the outcome sequence chart and the indicators spreadsheet for each program area.

program area.

Outcome sequence chart:

Outcome sequence chart: A visual depiction of the A visual depiction of the order in which program order in which program outcomes areoutcomes are expected to occur.

expected to occur.

Connects outcomes with directional lines and arrows to indicate the expected sequence of Connects outcomes with directional lines and arrows to indicate the expected sequence of key results. Intended to provide a quick one-page snapshot for each program area,

key results. Intended to provide a quick one-page snapshot for each program area, whereas the outcome indicator tables (described below) offer a more comprehensive whereas the outcome indicator tables (described below) offer a more comprehensive review and detail.

review and detail. The outcome sequence chart The outcome sequence chart for the project, provided above, is for the project, provided above, is anan example.

example.

8 8

(9)

Comments, caveats, and suggestions about the program outcome sequence charts include the Comments, caveats, and suggestions about the program outcome sequence charts include the following:

following:



 The focus is The focus is exclusively on program results—intermediate or exclusively on program results—intermediate or end outcomes. end outcomes. SomeSome

diagrams cover program inputs, activities, and outputs that occur internally and at the diagrams cover program inputs, activities, and outputs that occur internally and at the earlier stages of

earlier stages of a program. a program. The focus here The focus here on outcomes is on outcomes is deliberate. deliberate. We hope We hope thatthat nonprofit organizations will concentrate more of their (often limited) data collection nonprofit organizations will concentrate more of their (often limited) data collection efforts on measuring and reporting program results, rather than merely counting internal efforts on measuring and reporting program results, rather than merely counting internal activities (for example, number of staff training sessions held) or outputs (number of activities (for example, number of staff training sessions held) or outputs (number of pamphlets produced or distributed) that are increasingly of less interest to stakeholders pamphlets produced or distributed) that are increasingly of less interest to stakeholders such as foundations or the public.

such as foundations or the public.55



 The charts organize outcomes from The charts organize outcomes from left (first to occur) to left (first to occur) to right (later to occur). right (later to occur). Boxes atBoxes at

the left are usually intermediate outcomes, which tend to be realized sooner than end or the left are usually intermediate outcomes, which tend to be realized sooner than end or final outcomes, on the right.

final outcomes, on the right. In some cases, the In some cases, the end outcomes may end outcomes may take so long take so long toto achieve that they may appear to be beyond the scope of the program to track or imagine achieve that they may appear to be beyond the scope of the program to track or imagine claiming responsibility. While these are valid concerns, we chose to include such claiming responsibility. While these are valid concerns, we chose to include such longer-term, end outcomes to illustrate the ideal or ultimate goal for program participants or term, end outcomes to illustrate the ideal or ultimate goal for program participants or other recipients of services, such as the broader community.

other recipients of services, such as the broader community.



 Participant satisfaction is a very important, but sometimes overlooked element of Participant satisfaction is a very important, but sometimes overlooked element of

program performance.

program performance. While some debate While some debate about terminology (for about terminology (for example, are factorsexample, are factors such as timeliness or ease of service an outcome or better grouped as separate indicators such as timeliness or ease of service an outcome or better grouped as separate indicators of quality?), program satisfaction is of interest and should be measured by most

of quality?), program satisfaction is of interest and should be measured by most providers.

providers. Satisfaction indicators tend to be Satisfaction indicators tend to be similar across program similar across program areas, so they areas, so they areare included in a box below the chart.

included in a box below the chart. This is a reminder to include one or This is a reminder to include one or several suchseveral such measures in the overall measurement framework.

measures in the overall measurement framework.



 The outcome sequence charts are limited in their ability to fully illustrate the dynamicThe outcome sequence charts are limited in their ability to fully illustrate the dynamic

and sometimes circular

and sometimes circular nature of many nature of many programs. programs. Because they are Because they are identifying keyidentifying key outcomes for a

outcomes for a program area, they program area, they are intentionally somewhat generic. are intentionally somewhat generic. Most chartsMost charts illustrate a series of outcomes on a continuum that proceeds from left to right and are illustrate a series of outcomes on a continuum that proceeds from left to right and are connected by a series of

connected by a series of forward arrows. forward arrows. In some cases, however, we In some cases, however, we attempted toattempted to illustrate exceptions or the more circular nature of results by using dotted lines, arrows illustrate exceptions or the more circular nature of results by using dotted lines, arrows pointing in both directions, or stacked boxes (intended to reflect a certain equality among pointing in both directions, or stacked boxes (intended to reflect a certain equality among outcomes, rather than a rank-order or intended sequence.)

outcomes, rather than a rank-order or intended sequence.)



 We consulted many sources in the production of these charts and subsequent sets of We consulted many sources in the production of these charts and subsequent sets of

indicators.

indicators. The key written sources of The key written sources of materials are noted at the materials are noted at the bottom of outcomebottom of outcome

5 5

We make a limited number of exceptions to this rule by including a lead box to

We make a limited number of exceptions to this rule by including a lead box to illustrate the position and relativeillustrate the position and relative nature of outputs in relation to recommended program outcome

nature of outputs in relation to recommended program outcomes. s. See for example, affordable housing, advocacy, orSee for example, affordable housing, advocacy, or performing arts.

performing arts. The inclusion of these references were made at the recommendation of program reviewers whoThe inclusion of these references were made at the recommendation of program reviewers who argued that it was

argued that it was important to at least acknowledge that without such important to at least acknowledge that without such outputs being produced by the outputs being produced by the program thereprogram there would never be cause

(10)

sequence charts and also at

sequence charts and also at the end of the end of the tables of outcome the tables of outcome indicators. indicators. We encourageWe encourage users to locate and consult these references for additional examples of indicators, and for users to locate and consult these references for additional examples of indicators, and for information on data

information on data collection strategies, program context, etc. collection strategies, program context, etc. In addition to In addition to reviewingreviewing numerous written sources for each program area, we consulted with project advisors and numerous written sources for each program area, we consulted with project advisors and content experts for each

content experts for each program area. program area. These individuals provided essential feedback These individuals provided essential feedback and helped us to refine the charts as they are currently presented.

and helped us to refine the charts as they are currently presented.



 Finally, these outcome sequence charts are intended to be a starting point forFinally, these outcome sequence charts are intended to be a starting point for

organizations establishing

organizations establishing outcome outcome measurement measurement processes. processes. These These charts charts are are notnot intended to be comprehensive, but rather to identify outcomes and associated indicators intended to be comprehensive, but rather to identify outcomes and associated indicators that meet important selection criteria and have

that meet important selection criteria and have been vetted by been vetted by experts in the field. experts in the field. TheyThey almost always should be modified by a nonprofit organization so as to better meet the almost always should be modified by a nonprofit organization so as to better meet the needs of the organization.

needs of the organization.

Table of Outcomes

Table of Outcomes and Outcome Indicators: and Outcome Indicators: Provides detailed information for Provides detailed information for each outcomeeach outcome and outcome indicator identified for a particular program area.

and outcome indicator identified for a particular program area.

Each indicator is accompanied by a suggested data collection strategy, explanatory notes Each indicator is accompanied by a suggested data collection strategy, explanatory notes (where appropriate), as well as a suggested classification as an intermediate or end

(where appropriate), as well as a suggested classification as an intermediate or end outcome indicator.

outcome indicator.

Comments, caveats, and suggestions about the outcome indicator tables include the following: Comments, caveats, and suggestions about the outcome indicator tables include the following:



 The outcomes and outcome indicators are expected to be key result areas of interest forThe outcomes and outcome indicators are expected to be key result areas of interest for

many if not all nonprofits for this particular program area. many if not all nonprofits for this particular program area.



 One or more candidate specific outcome indicators are included for each programOne or more candidate specific outcome indicators are included for each program

outcome.

outcome. Outcome indicators are expressed in a Outcome indicators are expressed in a measurable format (such as a measurable format (such as a numbernumber and/or percent) and attempt to capture and report measures of the program outcome. and/or percent) and attempt to capture and report measures of the program outcome.



 A suggested data collection procedure for obtaining data for each outcome indicator isA suggested data collection procedure for obtaining data for each outcome indicator is

included.

included. Having a Having a sound practical data collection sound practical data collection procedure is vital procedure is vital to obtaining theto obtaining the outcome data.

outcome data. More than simply More than simply offering a framework for offering a framework for consideration and discussion,consideration and discussion, we hope these materials can readily be incorporated into planned or on-going

we hope these materials can readily be incorporated into planned or on-going management and data reporting efforts.

management and data reporting efforts.



 Notes providing additional details or caveats related to specific outcome indicators areNotes providing additional details or caveats related to specific outcome indicators are

included on the

included on the spreadsheets. spreadsheets. Often they provide Often they provide suggestions for important client suggestions for important client groupsgroups that might be considered individually at the stage of data analysis and reporting.

that might be considered individually at the stage of data analysis and reporting.

As noted earlier, the table of outcome indicators and outcome sequence charts for the 14 program As noted earlier, the table of outcome indicators and outcome sequence charts for the 14 program areas examined during this project can be found at

areas examined during this project can be found at

http://author.urban.org/center/cnp/commonindicators.cfm and http://www.whatworks.org. http://author.urban.org/center/cnp/commonindicators.cfm and http://www.whatworks.org.

(11)

Part 3

Part 3 Draft Common Outcome Framework

Draft Common Outcome Framework

A common outcomes framework provides an organized, generalized, set of outcomes and A common outcomes framework provides an organized, generalized, set of outcomes and outcome indicators that nonprofit programs can use to help them determine the outcomes and outcome indicators that nonprofit programs can use to help them determine the outcomes and outcome indicators appropriate for their service programs.

outcome indicators appropriate for their service programs.

This draft was developed using basic classification principles and the extensive This draft was developed using basic classification principles and the extensive

information gathered during the development of the outcomes and indicators for the 14 specific information gathered during the development of the outcomes and indicators for the 14 specific program areas described in Part

program areas described in Part 2. 2. To develop the common To develop the common outcomes framework, we reviewedoutcomes framework, we reviewed the outcomes and outcome indicators for these specific program areas to identify those that the outcomes and outcome indicators for these specific program areas to identify those that appeared to be applicable across multiple program areas.

appeared to be applicable across multiple program areas. The framework has these major purposes:

The framework has these major purposes:

•

• It provides a starting point for programs to begin developing their own outcomeIt provides a starting point for programs to begin developing their own outcome

measurement process. measurement process.

•

• For programs that already have some form of outcome measurement process, itFor programs that already have some form of outcome measurement process, it

provides a checklist for reviewing their coverage to determine whether other provides a checklist for reviewing their coverage to determine whether other outcomes and/or outcome indicators should be included in their outcome outcomes and/or outcome indicators should be included in their outcome measurement process.

measurement process.

•

• To the extent that nonprofit organizations use such common outcome indicators,To the extent that nonprofit organizations use such common outcome indicators,

this will provide an opportunity for across-program comparisons, enabling each this will provide an opportunity for across-program comparisons, enabling each nonprofit organization to benchmark itself against other organizations that are nonprofit organization to benchmark itself against other organizations that are providing similar services.

providing similar services. Programs often have similar goals.

Programs often have similar goals. For example, many For example, many different types of programs seek different types of programs seek to change

to change knowledge, attitudes, behavior, and knowledge, attitudes, behavior, and status or status or condition of condition of clients/particlients/participants. cipants. ManyMany different types of programs seek

different types of programs seek to achieve the same to achieve the same quality-of-serquality-of-service elements. vice elements. If the types of If the types of outcome information collected across a wide number of targeted program areas are collected, outcome information collected across a wide number of targeted program areas are collected, reviewed for quality, and grouped by program area, the results are likely to be useful to those and reviewed for quality, and grouped by program area, the results are likely to be useful to those and other nonprofits providing similar services.

other nonprofits providing similar services.

Such an arrangement of outcomes with associated indicators can become a standard Such an arrangement of outcomes with associated indicators can become a standard framework that provides guidance and context, helping users learn

framework that provides guidance and context, helping users learn what they need to what they need to know. know. ForFor example, although much information on program outcomes is available from a web-based key example, although much information on program outcomes is available from a web-based key word search, the results are likely to be undifferentiated—overwhelming in volume and time word search, the results are likely to be undifferentiated—overwhelming in volume and time consuming to assess

consuming to assess for relevance. for relevance. And the And the search results might search results might vary significantly if differentvary significantly if different key terms were chosen for the search.

key terms were chosen for the search.

The development and refinement of the common framework should continue to be an The development and refinement of the common framework should continue to be an iterative process, as outcomes and indicators are

(12)

from the

from the common framework is common framework is presented below. presented below. It includes program-centered outcomes (reach,It includes program-centered outcomes (reach, participation, satisfaction); participant- centered outcomes (knowledge/learning/attitude,

participation, satisfaction); participant- centered outcomes (knowledge/learning/attitude, behavior, condition/status); community-centered outcomes (policy, public health/safety, civic behavior, condition/status); community-centered outcomes (policy, public health/safety, civic participation, economic, environmental, social); and organization-centered outcomes (financial, participation, economic, environmental, social); and organization-centered outcomes (financial, management, governance).

management, governance). Little work has Little work has been completed been completed on the on the organization-ceorganization-centeredntered outcomes.

outcomes. The full version of The full version of the current draft framework can be the current draft framework can be found atfound at

http://author.urban.org/center/cnp/commonindicators.cfm and http://www.whatworks.org. http://author.urban.org/center/cnp/commonindicators.cfm and http://www.whatworks.org.

(13)

Common Framework of Outcomes

Excerpt: Participant-Centered Outcomes

1) Knowledge/Learning/Attitude 1) Knowledge/Learning/Attitude

a)

a) Skills Skills (knowledge, (knowledge, learning)learning) Common Indicators:

Common Indicators: Percent increase in scores after Percent increase in scores after attendingattending

Percent that believe skills were increased after attending Percent that believe skills were increased after attending Percent increase in

Percent increase in knowledge (before/after program)knowledge (before/after program) b) Attitude

b) Attitude

Common Indicators:

Common Indicators: Percent improvement as reported by parent, teacher, co-worker,Percent improvement as reported by parent, teacher, co-worker, other

other

Percent improvement as reported by

Percent improvement as reported by participantparticipant c)

c) Readiness Readiness (qualification)(qualification) Common Indicators:

Common Indicators: Percent feeling well-prepared for a particular task/undertakingPercent feeling well-prepared for a particular task/undertaking Percent meeting minimum qualifications for next

Percent meeting minimum qualifications for next level/undertaking

level/undertaking 2) Behavior

2) Behavior a)

a) Incidence Incidence of of bad bad behaviorbehavior Common Indicators:

Common Indicators: Incidence rateIncidence rate

Relapse/recidivism rate Relapse/recidivism rate

Percent reduction in reported behavior frequency Percent reduction in reported behavior frequency b)

b) Incidence of Incidence of desirable actidesirable activityvity Common Indicators:

Common Indicators: Success rateSuccess rate

Percent that achieve goal Percent that achieve goal Rate of

Rate of improvementimprovement c)

c) Maintenance Maintenance of of new new behaviorbehavior Common Indicators:

Common Indicators: Number weeks/months/years continuedNumber weeks/months/years continued Percent change over time

Percent change over time Percent moving to

Percent moving to next level/condition/statusnext level/condition/status Percent that do not reenter the

Percent that do not reenter the program/systemprogram/system 3) Condition/Status

3) Condition/Status a)

a) Participant Participant social social statusstatus Common Indicators:

Common Indicators: Percent with Percent with improved relationshipsimproved relationships Percent who graduate

Percent who graduate Percent who move to

Percent who move to next level/condition/statunext level/condition/statuss Percent who

Percent who maintain current level/condition/statusmaintain current level/condition/status Percent who avoid undesirable course

(14)

b)

b) Participant Participant economic economic conditioncondition Common Indicators:

Common Indicators: Percent who Percent who establish career/employmentestablish career/employment Percent who move to long term housing Percent who move to long term housing

Percent who maintain safe and permanent housing Percent who maintain safe and permanent housing Percent enrolled in education programs

Percent enrolled in education programs Percent who retain

Percent who retain employmentemployment Percent with increased earnings Percent with increased earnings c)

c) Participant Participant health health conditioncondition Common Indicators:

Common Indicators: Percent with reduced incidence of health problemPercent with reduced incidence of health problem Percent with immediate positive response

Percent with immediate positive response

Percent that report positive response post-90 days Percent that report positive response post-90 days

(15)

Part 4

Part 4 Tips on Using the Common Framework Project Materials

Tips on Using the Common Framework Project Materials



 Outcome information seldom, if ever, tellsOutcome information seldom, if ever, tells whywhy the outcomes have occurredthe outcomes have occurred.. YourYour

program will seldom

program will seldom be 100 be 100 percent responsible for those percent responsible for those outcomes. outcomes. Inevitably, otherInevitably, other factors, both external

factors, both external and internal, will and internal, will affect outcomes. affect outcomes. However, outcome informationHowever, outcome information is vital for indicating what needs to

is vital for indicating what needs to be done to be done to improve future outcomes. improve future outcomes. Your choice of Your choice of outcome indicators to track should not be determined by the extent of your influence over outcome indicators to track should not be determined by the extent of your influence over the outcome but the importance of the outcome for your clients.

the outcome but the importance of the outcome for your clients.



 Outcome data should be used to identify where results are going well and where not soOutcome data should be used to identify where results are going well and where not so

well. When not going well, the program

well. When not going well, the program needs to attempt to needs to attempt to find out why. find out why. This process isThis process is what leads to continuous program learning and program improvement.

what leads to continuous program learning and program improvement.



 Outcome information is much more useful if the measures are tabulated for variousOutcome information is much more useful if the measures are tabulated for various

categories of customers/clients, for example, by gender, age group, and race/ethnicity, categories of customers/clients, for example, by gender, age group, and race/ethnicity, income level, etc.

income level, etc.



 It may be wise to start tracking only a very small number of the indicators, especially if It may be wise to start tracking only a very small number of the indicators, especially if

you have had only very little experience with such data collection and have very limited you have had only very little experience with such data collection and have very limited resources. Not all outcomes or indicators listed will be relevant to every organization. resources. Not all outcomes or indicators listed will be relevant to every organization. Once your organization becomes more comfortable with outcome measurement, then Once your organization becomes more comfortable with outcome measurement, then more outcomes and indicators can be added to the system.

more outcomes and indicators can be added to the system.



 Review the list of outcome indicators for the program that most closely matches, but alsoReview the list of outcome indicators for the program that most closely matches, but also

check out the common framework to see if the more general set suggests other relevant check out the common framework to see if the more general set suggests other relevant indicators.

indicators.



 Selecting which outcomes and Selecting which outcomes and indicators to monitor is indicators to monitor is crucial. crucial. Sessions with staff andSessions with staff and

board members, and perhaps clients, to discuss what outcomes and outcome indicators board members, and perhaps clients, to discuss what outcomes and outcome indicators your program should monitor will be important and will keep all aware of the outcome your program should monitor will be important and will keep all aware of the outcome measurement efforts.

measurement efforts. The staff and board The staff and board members will be the members will be the persons most able to persons most able to useuse the findings to improve services.

the findings to improve services.



 Some of the most important client outcomes and outcome indicators will require newSome of the most important client outcomes and outcome indicators will require new

data collection procedures (such as determining the extent to which improved client data collection procedures (such as determining the extent to which improved client conditions have been sustained for at least, say 6 or 12 months, after service to the client conditions have been sustained for at least, say 6 or 12 months, after service to the client has been completed).

has been completed). Nonprofit organizations should not give Nonprofit organizations should not give up too up too quickly onquickly on implementing such

implementing such data collection data collection procedures. procedures. Often, surprisingly inexpensiveOften, surprisingly inexpensive procedures can be used, especially if the program has any type of aftercare process. procedures can be used, especially if the program has any type of aftercare process.

(16)

Additional resources: Additional resources: •

• Urban Institute Series on Outcome Management for Nonprofit Organizations:Urban Institute Series on Outcome Management for Nonprofit Organizations: −

− “Key Steps in Outcome Management”“Key Steps in Outcome Management” by Harry P. Hatry and Linda M. Lampkinby Harry P. Hatry and Linda M. Lampkin

(http://www.urban.org/publications/310776.html) (http://www.urban.org/publications/310776.html)

−

− “Finding Out What Happened to Former Clients” by Ritu Nayyar-Stone and Harry P.“Finding Out What Happened to Former Clients” by Ritu Nayyar-Stone and Harry P.

Hatry (http://www.urban.org/publications/310815.html) Hatry (http://www.urban.org/publications/310815.html)

−

− “Developing Community-wide Outcome Indicators for Specific Services” by Harry P.“Developing Community-wide Outcome Indicators for Specific Services” by Harry P.

Hatry, Jake Cowan, Ken Weiner and Linda M. Lampkin Hatry, Jake Cowan, Ken Weiner and Linda M. Lampkin (http://www.urban.org/publications/310813.html)

(http://www.urban.org/publications/310813.html)

−

− “Surveying Clients about Outcomes” by Martin D. Abravanel“Surveying Clients about Outcomes” by Martin D. Abravanel

−

− “Analyzing Outcome Information” by Harry P. Hatry, Jake Cowan and Michael“Analyzing Outcome Information” by Harry P. Hatry, Jake Cowan and Michael

Hendricks (

Hendricks (http://www.urban.org/publications/310973.htmlhttp://www.urban.org/publications/310973.html ))

−

− “Using Outcome Information” by Elaine Morley and Linda M. Lampkin“Using Outcome Information” by Elaine Morley and Linda M. Lampkin

•

• The Center for What Works Performance Measurement Toolkit and other tips, tools,The Center for What Works Performance Measurement Toolkit and other tips, tools,

resources and training resources and training

(http://www.whatworks.org/displaycommon.cfm?an=1&subarticlenbr=13%20) (http://www.whatworks.org/displaycommon.cfm?an=1&subarticlenbr=13%20)

•

• 2006 Performance Measurement: Getting Results, forthcoming, 22006 Performance Measurement: Getting Results, forthcoming, 2ndndedition by Harry Hatryedition by Harry Hatry

(http://www.urban.org/books/pm/chapter1.cfm); (http://www.urban.org/books/pm/chapter1.cfm);

•

• “Measuring Program Outcomes: A Practical Approach” by the United Way of America“Measuring Program Outcomes: A Practical Approach” by the United Way of America

(http://national.unitedway.org/outcomes/index.cfm) (http://national.unitedway.org/outcomes/index.cfm)

•

• Ten Steps to a Results-Based Monitoring and Evaluation System,” 2004 by World Bank Ten Steps to a Results-Based Monitoring and Evaluation System,” 2004 by World Bank

•

• “Guidebook for Performance Measurement, 1999 by Turning Point”“Guidebook for Performance Measurement, 1999 by Turning Point”

•

• Boys & Girls Club of America, “Youth Development Outcome Measurement Tool KitBoys & Girls Club of America, “Youth Development Outcome Measurement Tool Kit

(http://www.bgca.org/) (http://www.bgca.org/)

•

• Benchmarking for Nonprofits: Benchmarking for Nonprofits: How to How to Measure, Manage, and IMeasure, Manage, and Improve Performancemprove Performancebyby

Jason Saul (http://www.fieldstonealliance.org/productdetails.cfm?PC=52) Jason Saul (http://www.fieldstonealliance.org/productdetails.cfm?PC=52)

(17)

Appendix

Common Measures Advisory Committee Members

Audrey R. Alvarado, Executive Director Audrey R. Alvarado, Executive Director National Council of Nonprofit Associations National Council of Nonprofit Associations

Mark Moore, Director Mark Moore, Director

The Hauser Center for Nonprofit Organizations The Hauser Center for Nonprofit Organizations Patrick Corvington, former Executive Director

Patrick Corvington, former Executive Director Innovation Network, Inc.

Innovation Network, Inc.

Margaret C. Plantz Margaret C. Plantz United Way of America United Way of America Kathleen Enright, Executive Director

Kathleen Enright, Executive Director Grantmakers for Effective Organizations Grantmakers for Effective Organizations

James Saunders James Saunders

The Evaluation Center The Evaluation Center Kathleen Guinan, Executive Director

Kathleen Guinan, Executive Director Crossway Community

Crossway Community

Laura Skaff, Director of Evaluation Laura Skaff, Director of Evaluation Volunteers of America

Volunteers of America Susan Herr, Managing Director

Susan Herr, Managing Director Community Foundations of America Community Foundations of America

Ken Voytek Ken Voytek

Goodwill Industries International, Inc. Goodwill Industries International, Inc. Amy Coates Madsen, Program Director

Amy Coates Madsen, Program Director Standards for Excellence

Standards for Excellence

Dennis R. Young Dennis R. Young

Andrew Young School, Georgia State Andrew Young School, Georgia State Ricardo Millet, former President

Ricardo Millet, former President The Woods Fund of Chicago The Woods Fund of Chicago

Common Measures Project Staff

The Urban Institute The Urban Institute

Linda Lampkin Linda Lampkin Mary Winkler Mary Winkler Janelle Kerlin Janelle Kerlin Harry Hatry Harry Hatry

The Center for What Works The Center for What Works

Debra Natenshon Debra Natenshon Jason Saul Jason Saul Julia Melkers Julia Melkers Anna Seshadri Anna Seshadri