Module 2: Evaluation Essentials

(1)

Module 2:

Evaluation Essentials

Do, Measure, Improve

Module 2: Evaluation Essentials

• Types of research

o

Basic new knowledge or evaluation?

(2)

NatGuild2014 Kamella Tate MFA, EdD [email protected] 3

Types of Research

Different

purposes

typically lead to different ways

of conceptualizing

problems

, different

designs

,

different types of

data gathering

, and different

ways of

publicizing and disseminating

findings.

(Patton, 2002, p. 222)

Types of Research

(variation on Patton, 2002, pp. 213-225)

1. Basic/Applied New Knowledge

2. Evaluation

(3)

Types of Research

(Patton, 2002, pp. 213-225)

Basic New Knowledge Research

To contribute to

fundamental knowledge

and theoretical understanding about

basic

human and other natural processes

.

“We have reached a milestone in our understanding of

nature . . . Likely to shed light on other mysteries of

our universe.”

Types of Research

(Patton, 2002, pp. 213-225)

Applied New Knowledge Research

To produce

(

generalizable

)

new knowledge

focused on answering

practical, real-world

questions

and provide

relatively immediate

solutions

.

(4)

Types of Research

(Patton, 2002, pp. 213-225)

Formative Evaluation

Ongoing

improvement

of products,

processes, strategies, change

interventions, institutions, personnel,

policies, etc.;

to form or shape

the

“thing being studied”

(Patton, 2002, p. 220).

Types of Research

(Patton, 2002, pp. 213-225)

Summative Evaluation

To

determine the effectiveness

of products,

processes, strategies, change interventions,

institutions, personnel, policies, etc.; to

render judgments

that inform decisions.

(5)

Distinguishing Between

Basic/Applied and Evaluation

Research

Framing Evaluation Research

• Assessment and evaluation basics

(6)

What we mean by . . .

Assessment

Assessment is the

organized and ongoing

process

of

collecting and analyzing

data

and information so as to

describe

activities, practices, progress, and other

dimensions of performance.

What’s happening?

What we mean by . . .

Evaluation

Evaluations are

systematic investigations

that

involve synthesizing and integrating assessment

data and then using this information to make

inferences

and

judgments

about:

• the

merit

(i.e., quality, excellence)

• the

worth

(i.e., value, cost-effectiveness)

• and/or the

significance

(i.e., importance, impact)

of a project, program, or organization.

Is it working/not? Why is it working/not?

(7)

So what?

• Documentation

: What’s

happening

? What’s

our

story

?

• Planning and improvement:

Evidence-based

quality management

.

• Effectiveness

: What’s

working

? What’s

not

working

?

• Advocacy

: Find the

case

. Make the

case

.

Evaluation is accountability-in-action.

13 NatGuild2014 Kamella Tate MFA, EdD [email protected]

A Note On Advocacy

In the same way that an (effective) grant

proposal or application is an

artifact

of a

(effective) program/project design process . . .

(Effective) advocacy is an

artifact

of

(effective) assessment and evaluation

practices.

(8)

A Planning & Evaluation Flowchart

NatGuild2014 Kamella Tate MFA, EdD [email protected] 16 Learn More

Basically Speaking Part 1: http://goo.gl/47uwra Basically Speaking Part 2: http://goo.gl/5ZIQSC

(9)

An Evaluation Continuum

May

be more involved with:

• Day-to-day operations

• Improvisational,

un-structured

• Action-oriented,

problem-solving

• Staff, clients, participants

May

be more involved with:

• Systems, policies, models

• Structured activities,

workplans

• Organizational development

• Multiple stakeholder groups

within the community

“INFORMAL” EVALUATION

“FORMAL” EVALUATION

Questions and Strategies

• What's happening? How are things

going?

• Did you get X done? What do we

need to do to get X done?

• How did they (patrons, students,

visitors) like it?

• Where are we on the budget?

• How are the reviews?

• How many people came?

• Developing scaled surveys and

doing statistical analyses

• Seeking input from a variety of

stakeholders

• Using an outside evaluator

• Conducting interviews and

holding focus groups

• Measuring impact; cause-effect

• Publishing white papers

(10)

Characteristics of Evaluations

Evaluation

Purpose

Improve practice (formative)

Determine value or worth (summative)

Audience &

Setting

Users and stakeholders

Primarily local (program, organization,

community)

Methods & Data

Quantitative

Qualitative

Multi/Mixed

Generalizability

Unimportant

Utility

Critical

BASIC NEW KNOWLEDGE RESEARCH

Evaluation Purposes

Formative = Improvement

• What are a program’s

strengths

and

weaknesses

?

What are some of the

threats

and

opportunities

facing an institution?

What’s happening? How

are things going?

o

Terms used:

Recommendations, promising, growth,

quality, identify needs, enhancement, suggestions,

learning organization, meet challenges, overcome

barriers, advance, potential, etc.

(11)

Evaluation Purposes

Summative = Effectiveness

• Did the program

accomplish

its

purpose

? Did the

efforts taken ensure the institution

achieved

its

goals?

Did it work?

o

Terms used:

Conclusions, impact, influence, effect/

effectiveness, results, changes, outcomes, outputs,

consequences, benefits, end products, achievements,

bottom line, etc.

Quick Quiz

Which evaluation question is formative?

1. As indicated by scores on a skills test, how

effective was ScienceGrrrl’s “From Needs

to Outcomes Logic Modeling" seminar?

2. How can ScienceGrrrl enhance its “From

Needs to Outcomes Logic Modeling"

seminar?

(12)

An Audience for Evaluation

Intended Users

Program Participants &

Personnel

[Clients, Customers,

Staff, Managers]

Organizational Leaders

[Directors, Boards,

Decision-makers, etc.

Funders

Stakeholders

Community-at-Large

Funders, Philanthropists

Field, Profession, Sector

Civic and Political Leaders

Neighbors

Researchers

Society

Etc.

Methods & Data

I

t was a great step in science

when men became convinced

that in order to understand the nature of things,

they must begin by asking not whether a thing

is

good

or

bad

,

noxious

or

beneficial

,

but of

what kind

it is?

And

how much

is there of it?

Quality

and

Quantity

were

then first recognized as the primary features to be observed

in

scientific inquiry

.

— James Clerk Maxwell

Address to the Mathematical and Physical Sections of the British Association, Liverpool, 15 Sep 1870 The Scientific Papers of James Clerk Maxwell(1890 edition, reprint 2003), Vol. 2, 217

(13)

What kind?

Qualitative (data, methods, designs)

Qualitative data is evidence largely in

the form of

words

and

text

– quotes,

narratives, and stories, as well as

descriptions of images, environments,

behaviors, etc.

25 NatGuild2014 Kamella Tate MFA, EdD [email protected]

How much?

Quantitative (data, methods, designs)

Quantitative data is evidence in the form of

numbers

– numerical indices that allow for

counting

,

ranking

, or

scaled

measurement.

• N = 404; 48.4%; 3.21

•

1 st

_{, 2}

nd

_{, 3}

rd

(14)

Quick Quiz

QUANT or QUAL?

1. You count how many times people ask

questions.

2. Vivid, well-written descriptions are

what you find most useful.

3. Zip code 91607

Collecting Two Types of Data

Primary methods are in

boldface italics

Oral

Interviews

Observations

Written or Oral

Surveys

Collecting

Quantitative

Data

Diagnostic

classifications

and ratings

Records of

quantities

Frequency counts

and rates

Ratings, opinion polls,

surveys, tests,

performance

metrics, scaled

questionnaires, etc.

Collecting

Qualitative

Data

Un- or

Semi-structured

interviews

Focus groups

Observational

fieldwork

Extant documents

and artifacts

Open-ended written

or oral

questionnaires

(15)

Myth #1

Quantitative methods are

more scientific

and/or more rigorous

than qualitative

methods.

Myth #2

Qualitative methods are

easier

than

quantitative methods.

Where there are

numbers

, we also

need

words

.

Where there are

words

, we also

need

numbers

.