BEN-GURION UNIVERSITY OF THE NEGEV GUILFORD GLAZER FACULTY OF BUSINESS AND MANAGEMENT DECEMBER 2014

Organizational Assessment & Program Evaluation in Nonprofit Organizations - 1

BEN-GURION UNIVERSITY OF THE NEGEV

GUILFORD GLAZER FACULTY OF BUSINESS AND MANAGEMENT

DECEMBER 2014

ORGANIZATIONAL ASSESSMENT AND PROGRAM EVALUATION

IN NONPROFIT ORGANIZATIONS

LECTURER:

Robert L. Fischer, Ph.D.

Research Associate Professor, Mandel School of Applied Social Sciences, Case Western

Reserve University

Co-Director, Center on Urban Poverty and Community Development

Faculty Director, Master of Nonprofit Organizations (MNO) Degree Program

BRIEF COURSE DESCRIPTION:

The course is designed to introduce students to the approaches to organizational

assessment and evaluation of nonprofit programs and interventions. The class will explore

a variety of ways of assessing organizations, review the dimensions essential to nonprofit

organizations, and explore some processes useful to enable change. In addition, the course

will focus on the process of creating and measuring program outcomes.

SOURCES:

1.

Rossi, P. H., Lipsey, M. W., & Freeman, H. E. (2004). Evaluation: A Systematic

Approach. Seventh Edition. Thousand Oaks, CA: Sage Publications, Inc. [Referred to

as Rossi et al. in course outline]

2.

Additional outside readings as assigned in the syllabus (below).

DETAILED COURSE DESCRIPTION:

This course is designed to introduce students to the design, implementation, management,

analysis, and utilization of evaluation research. The major rationales for this course are:

1)

The strong desire for accountability in nonprofit service delivery

2)

The importance of informing agency and program decision making to promote

program improvement, and

3)

The need to demonstrate the effectiveness and efficiency of nonprofit programs, and

contribute to the knowledge base on “what works” in specific program areas.

Students will develop their understanding of and use of skills in the interpretation and

conduct of different types of evaluation research, including needs assessment,

Organizational Assessment & Program Evaluation in Nonprofit Organizations - 2

This course is especially relevant for students interested in community or program

research in nonprofit agency settings. This course is particularly appropriate for students

who anticipate the need to develop practical skills in evaluating program performance in

future planning or management roles.

The course will explore a variety of ways of viewing organizations and programs, assessing

their effectiveness and success in achieving desired outcomes. A major focus of the course

will be on understanding the use of outcome measurement techniques (e.g., United Way of

America) and evaluation research methods to assess the condition of programs and inform

program improvement processes.

COURSE GOALS:

1.

Understand the current state of nonprofit management as it relates to the evaluation

of program and agency outcomes;

2.

Learn to assess evaluation studies and use prior research in enhancing the quality of

program interventions;

3.

Formulate program evaluation questions appropriate for specific nonprofit settings;

4.

Apply theories, research principles, and methods to evaluation in specific settings;

5.

Learn to use appropriate research design, measurement, sampling, data collection,

and data analysis in conducting program evaluation;

6.

Understand the effects of organizational environment and cultural context in

designing, managing, implementing, and utilizing program evaluation;

7.

Perceive ethical and political issues in program evaluation; and

8.

Appreciate the role of class, race, gender, sexual orientation, and culture in

evaluation research design and the interpretation of study results.

COURSE REQUIREMENTS:

The course grade will be based on three dimensions –

Final Exam 50%

Presentation

35%

Participation 15%

Class Participation (15%) -- Each student is expected to participate fully in the class as

demonstrated through completion of the assigned readings, active engagement in class

activities, and attendance during all scheduled class periods.

In-Class Presentation (35%)

Each student should prepare a short presentation of a potential evaluation research topic

of their choosing. Presentation will be graded based on thoroughness and credibility of the

approach proposed.

Final Exam (50%) Each student will take a written exam on the material covered in the

course.

Organizational Assessment & Program Evaluation in Nonprofit Organizations - 3

COURSE PLAN

12/18/14 THURSDAY

Unit 1 Introduction to the Course

Overview of course, schedule, and assignments Read:

[1] Tampa Bay Times. (2013). America’s Worst Charities.

Interactive data available at http://www.tampabay.com/americas-worst-charities/

[2] Forti, M. & Yazbak, K. (2012). Data’s not just for donors—it can improve a nonprofit’s work. The Chronicle of Philanthropy. April 1.

[3] Cunnigham, K., & Ricks, M. (2004). Why measure? Nonprofits use metrics to show that they are efficient. But what if donors don’t care? Stanford Social Innovation Review. Summer.

[4] Snibbe, A. C. (2006). Drowning in data. Stanford Social Innovation Review. Fall. 39-45.

[5] Newcomer, K.E. (1997). Using performance measurement to improve programs.

New Directions for Evaluation, 75, 5-14. Skim:

Grantmakers for Effective Philanthropy. (2009). Evaluation in Philanthropy: Perspectives from the Field.

Unit 2 The Field of Evaluation

The interrelationship between organizational assessment and program evaluation Read:

[4] Program, Policies and Evaluation (Rossi et al., Ch. 1, 3-30) [5] Tailoring Evaluations (Rossi et al., Ch. 2, 31-66)

[6] Patton, M. Q. (1994). Developmental evaluation. Evaluation Practice, 15(3), 311-319.

Unit 3 Ethical Issues & Protection of Human Subjects

What are the role of ethics and human subject protection in evaluation research? Read:

[7] Oakes, J. M. (2002). Risks and wrongs in social research: An evaluator’s guide to the IRB. Evaluation Review, 26(5), 443-479.

[8] Twersky, F., Buchanan, P., & Threlfall, V. (2013). Listening to those who matter most, the beneficiaries. Stanford Social Innovation Review. Spring.

Organizational Assessment & Program Evaluation in Nonprofit Organizations - 4

What are social indicators, how are they accessed, and how are they used? How can other data on the status of communities be integrated to understand specific needs? Read:

[9] Assessing the Need for a Program (Rossi et al., Ch. 4, 101-132)

[10] Beaulieu, L. J. (2002). Mapping the assets of your community: A key component for building local capacity. Southern Rural Development Center. 1-13.

Skim:

McKnight, J.L., & Kretzmann, J. P. (1996) Mapping Community Capacity. Institute for Policy Research. Evanston, IL: Northwestern University.

12/19/14 FRIDAY

Unit 5 Assessing Agency Performance

What are the analytic and approaches available to management for assessing the organization?

Read:

[11] Kaplan, R. S., & Norton, D. P. (1996). Using the Balanced Scorecard as a strategic management system. Harvard Business Review, Jan-Feb, 75-85.

[12] Putnam, K. (2004). Measuring Foundation Performance: Examples from the

Field. Oakland, CA: California Healthcare Foundation.

[13] Schalock, R. L., & Bonham, G. S. (2003). Measuring outcomes and managing for results. Evaluation and Program Planning, 26, 229-235.

Skim:

Measuring Efficiency (Rossi et al., Ch. 11, 331-368)

La Piana Associates, Inc. (2003). Tools for Assessing Startup Organizations: A Due

Diligence Supplement for Grantmakers. Washington: Grantmakers for Effective Organizations.

Drucker, P. F. (n.d.). Strategic Planning using the Drucker Foundation Self-Assessment Tool

United Way of Allegheny County (2003). Organizational Assessment Tool (OAT) for Full Drucker Self-Assessment Process. Pittsburgh: Bayer Center for Nonprofit Management.

Unit 6 Introduction to Process and Outcome Evaluation Methods

How can organizations begin to explore and adopt methods for measuring their performance? What is the starting place?

Organizational Assessment & Program Evaluation in Nonprofit Organizations - 5

[14] Identifying Issues and Formulating Questions (Rossi et al., Ch. 3, 67-100) [15] Jacobs, F. H. (2003). Child and family program evaluation: Learning to enjoy complexity. Applied Developmental Science, 7(2), 62-75.

Skim:

Preskill, H., & Jones, N. (2009). A Practical Guide for Engaging Stakeholders in

Developing Evaluation Questions. Princeton, NJ: Robert Wood Johnson Foundation Hatry, H., Lampkin, L., Morley, E., & Cowan, J. (eds). (2002). How and Why Nonprofits

Use Outcome Information. Washington: The Urban Institute. 12/21/14 SUNDAY

Unit 7 Articulating Program Theory & Structure

What are the primary elements of the intervention, project or program? How can a logic model be used to frame the evaluation of a program?

Read:

[16] Expressing and Assessing Program Theory (Rossi et al., Ch. 5, 133-168) [17] Plantz, M. C., Greenway, M. T., & Hendricks, M. (1997). Outcome measurement: Showing results in the nonprofit sector. New Directions for Evaluation, 75, 15-30. [18] Fischer, R. (2001). The sea change in nonprofit human services: A critical assessment of outcomes measurement. Families in Society, 82(6), 561-568. Skim:

Hatry, H. & Lampkin, L. (eds). (2003). Key Steps in Outcome Measurement. Washington: The Urban Institute.

Unit 8 Monitoring Approaches and Measures

Specifying evaluation questions and selecting data measures and approaches Use of surveys, focus groups, administrative data, key informants, and other tools Read:

[19] Assessing & Monitoring Program Process (Rossi et al., Ch. 6, 169-202)

[20] Measuring & Monitoring Program Outcomes (Rossi et al, Ch 7, 203-232)

[21] Czaja, C. & Blair, J. (2005) Questionnaire design: Writing the questions.

Designing Surveys: A Guide to Decisions and Procedures. Second Edition. Thousand Oaks: Sage. 59-83.

Unit 9 Designing Systematic Evaluation Plans I

Organizational Assessment & Program Evaluation in Nonprofit Organizations - 6

Read:

[22] Assessing Program Impact: Randomized Field Experiments (Rossi et al., Ch. 8, 233-264)

[23] Myers, D., & Dynarski, M. (2003). Random Assignment in Program Evaluation

and Intervention Research: Questions and Answers. Washington: U.S. Department of Education.

[24] Boruch, R. (2002). The virtues of randomness. Education Next. Fall. 37-41. [25] Cook, T. D. (2001). Sciencephobia: Why education researchers reject randomized experiments. Education Next. Fall. 63-68.

12/25/14 THURSDAY

Unit 10 Designing Systematic Evaluation Plans II

Choosing the most rigorous evaluation design given the realities of applied settings Read:

[26] Assessing Program Impact: Alternative Designs (Rossi et al., Ch. 9, 265-300) [27] Detecting, Interpreting, & Analyzing Program Effects (Rossi et al., Ch.10, 301-330)

[28] Gueron, J. M. (2004). Throwing good money after bad: A common error misleads foundations and policymakers. Stanford Social Innovation Review. 69-71. Unit 11 Real World Evaluation

Deciding what to measure, how and when. Accuracy and feasibility of measures. Data management and the use of statistical software for analysis (e.g., SPSS) Practical aspects of evaluation - logistics, staffing, budget, etc.

Read:

[29] The Social Context of Evaluation (Rossi et al., Ch. 12, 369-421)

[30] Bamberger, M., Rugh, J., Church, M., & Fort, L. (2004). Shoestring evaluation: Designing impact evaluations under budget, time, and data constraints. American

Journal of Evaluation, 25(1), 5-37.

[31] Stufflebeam, D. L., & Shinkfield, A. J. (2007). Psuedoevaluations. Evaluation

Theories, Models, & Applications. Chapter 6 (pp. 145-158). San Francisco: Jossey-Bass.

Organizational Assessment & Program Evaluation in Nonprofit Organizations - 7

Unit 12 Evaluation Findings and Lesson Learned

What comparisons will you make? How compelling or valid will your conclusions be? What are the limitations on your findings? What is the significance of your findings? How will you disseminate your findings & how will they be used? Read:

[33] Henry, G. T. (2003). Influential evaluations. American Journal of Evaluation,

24(4), 515-524.

[34] Weiss, C., Murphy-Graham, E., and Birkeland, S. (2005). An Alternate Route to Policy Influence: Evidence from a Study of the Drug Abuse Resistance Education (D.A.R.E.) Program. American Journal of Evaluation , 26, 12-31.

Organizational Assessment & Program Evaluation in Nonprofit Organizations - 8

A. Evaluation-focused Journals American Journal of Evaluation

Educational Evaluation and Policy Analysis Evaluation

Evaluation & the Health Professions Evaluation and Program Planning Evaluation Review

New Directions for Evaluation

Nonprofit and Voluntary Sector Quarterly Research on Social Work Practice

B. Web Sites (all begin http://)

American Evaluation Association www.eval.org

Assessment & Evaluation (ERIC) ericae.net/intass.htm Evaluation Center - W. MI State U. www.wmich.edu/evalctr/

Harvard Family Research Project hugse1.harvard.edu/~hfrp Human Services Research Institute www.hsri.org/eval/eval.html

Innovation Network

Nat’l Neighborhood Indicators Project

www.inetwork.org/ www.urban.org/nnip/

Nat’l Science Foundation handbook www.ehr.nsf.gov/EHR/REC/pubs/NSF97-153/start.htm

Ohio Program Evaluators’ Group www.opeg.org

Search Institute www.search-institute.org/

U.S. Government Accountability Office www.gao.gov

U.S. DHHS - Planning & Evaluation aspe.os.dhhs.gov/

United Way Resource Network www.unitedway.org/outcomes

C. Selected Bibliography on Program Evaluation

Bernstein, D. J. (2001). Local government measurement use to focus on performance and results. Evaluation and Program Planning, 24, 95-101.

Bickman, L., & Rog, D. J. (Eds). (1998). Handbook of Applied Social Research Methods. Thousand Oaks, CA: Sage.

Bocialetti, G., & Kaplan, R.E. (1986). “Self-Study” for human service agencies: Managing a three-sided relationship. Evaluation and Program Planning, 9(1), 1-11.

Chambers, D., Wedel, E., & Rodwell, M. (1992). Evaluating Social Programs. Beverly Hills, CA: Sage Publications.

Chelimsky, E. (1987). The politics of program evaluation. In D. S. Cordray, H. S. Bloom, & R. J. Light (Eds.), Evaluation Practice in Review (pp. 5-21). New Directions for Program Evaluation, No. 34. San Francisco: Jossey-Bass.

Chelimsky, E., & Shadish, W. R. (Eds). (1997). Evaluation for the 21st Century. Thousand Oaks, CA: Sage Publications.

Chen, H. T. (1990). Theory-Driven Evaluations. Beverly Hills, CA: Sage Publications.

Cousins, J.B. (1998) Framing participatory evaluations. New Directions for Evaluation (80). 5-23. Kennedy, M. M. (1983). The role of the in-house evaluator. Evaluation Review, 7(4), 519-541.

Organizational Assessment & Program Evaluation in Nonprofit Organizations - 9

Lawrence, J. E. (1982). Designing useful evaluations: The stakeholder survey. Evaluation and Program Planning, 5, 327-355.

Mark, M. M., Henry, G. T., & Julnes, G. (2000). Evaluation: An Integrated Framework for Understanding, Guiding, and Improving Public and Nonprofit Policies and Programs. San Francisco: Jossey-Bass. Miller, A., Carnevale, J. & Simeone, R. (2001). Logic models: A systems tool for performance management.

Evaluation and Program Planning, 24, 73-81.

Neuber, K. (1980). Needs Assessment: A Model for Community Planning. Beverly Hills, CA: Sage Publications. Palumbo, D. J., & Nachmias, D. (1983). The Preconditions for successful evaluation: Is there an ideal

paradigm? Policy Sciences, 16, 67-79.

Patton, M. Q. (1990). Qualitative Evaluation and Research Methods. Second Edition. Newbury Park, CA: Sage Publications.

Pietrzak, B. J., Ramler, M., Renner, T., Ford, L., & Gilbert, N. (1990). Practical Program Evaluation: Examples from Child Abuse Prevention. Beverly Hills, CA: Sage Publications.

Posavac, E. J., & Carey, R. G. (1992). Program Evaluation: Methods and Case Studies. Fifth Edition. Upper Saddle River, NJ: Prentice-Hall.

Rossi, P. H., Lipsey, M. W., & Freeman, H. E. (2004). Evaluation: A Systematic Approach. Seventh Edition. Thousand Oaks, CA: Sage Publications, Inc

Rutman, L., & Mowbray, G. (1983). Understanding Program Evaluation. Newbury Park, CA: Sage Publications. Scheirer, M. A. (1987). Program theory and implementation theory: Implications for evaluators. In L.

Bickman (Ed.), Using Program Theory in Evaluation (pp. 59-76). San Francisco: Jossey-Bass.

Scheirer, M. A., & Rezmovic, E. L. (1983). Measuring the degree of program implementation: A methodological review. Evaluation Review, 7(5), 599-633.

Schalock, R. L., & Thornton, C. V. D. (1988). Program Evaluation: A Field Guide for Administrators. New York: Plenum Press.

Scriven, M. (1991). Evaluation Thesaurus. Fourth Edition. Newbury Park, CA: Sage Publications.

Shadish, W. R., Cook, T. D., & Leviton, L. C. (1991). Foundations of Program Evaluation: Theories of Practice. Thousand Oaks, CA: Sage Publications.

Smith, M. (1990). Program Evaluation in Human Services. New York: Springer.

Thayer, C. & Fine, A. (2001). Evaluation and outcome measurement in the non-profit sector: Stakeholder participation. Evaluation and Program Planning, 24, 103-108.

Trochim, W. M. K. (2001). An introduction to concept mapping for planning and evaluation. http://trochim.human.cornell.edu/research/epp1/epp1.htm

United Way (2001). Outcome measurement resource network. United Way of America, Alexandria Va. http://national.unitedway.org/outcomes/ndpaper.htm.

Viadro, C. Altpeter, M. & Earp, A. (1997). Designing a process evaluation for a comprehensive breast cancer screening intervention: Challenges and opportunities. Evaluation and Program Planning, 20 (3), 237-249.

Yin, R. K. (1981). The case study crisis: Some answers. Administrative Science Quarterly, 26(1), 58-65. Weiss, C. H. (1998). Evaluation. Second Edition. Upper Saddle River, NJ: Prentice Hall.

Wholey, J. S., Hatry, H. P., & Newcomer, K. E. (Eds). (1994). Handbook of Practical Program Evaluation. San Francisco: Jossey-Bass Publishers.

Organizational Assessment & Program Evaluation in Nonprofit Organizations - 10

INITIAL QUESTIONNAIRE

(For completion and return at first class meeting)

Name:

Undergraduate discipline:

Your responses will assist in orienting the course toward class interests and needs. 1. What, if any, previous experience do you have in evaluation or applied research?

2. Are there any specific skills or knowledge that you hope to acquire in this course?