Over 14 million views within a week of
posting the video to YouTube.
Impressions are useful measure for
exposure, but the objective of a safety
campaign is behaviour change, not
impressions or click-throughs or
Facebook shares.
Leah Waymark, the General Manager of
Corporate Relations at Metro Trains,
summarizes the evaluation criteria as
“…if we can save one life or avoid
serious injury, then that’s how we’ll
measure the success of this
campaign”
Workshop 2:
Selecting and setting
Behavioural Goals, and
Social Marketing planning
“
Social Marketing is about
creating something that is
valued
by your audience”
What is Sociale Marketing?
FREE HEALTHY
breakfast club
Music Club
TV, music & celebrity magazines
+ a healthy breakfast
‘Eating
healthy,
before
entering
Social Marketing
•
Integrated Intervention Mix
•
Co-creation through Social Markets
•
Competition analysis and Action
•
Systematic planning & evaluation
•
Insight driven Segmentation
Social
Marketing
Techniques
•
Behavioral
Influence
•
Citizen/ Orientation Focus)
•
Social Offerings (Idea,
product, service experience
•
Relationship building
Social Marketing
Concepts
•
Social Value
Creation through
The Social
The big frustrating questions for
Donors and Governments
What is the impact
of the funds we
invest?
Lord Kelvin
26 June 1824 – 17 December
1907
To measure is to
know
Knowledge is
power
Francis Bacon
22 January 1561 – 9 April
1626
If you can’t measure
it you can’t manage
it.
What gets
measured gets
done
Social Marketing Planning
Social Marketing planning models need to be applied alongside the concepts and principles set out above. The four step STELa model is one such planning process. http://www.stelamodel.com/ The Rationale Situation Analysis Marketing Intervention Mix Strategies
Time Frame and Key Mile stones
Reporting Dissemination
Review and Plan to
Test
Learn
&
Act
Scope
Test
Enact
Learn Act
The 6 Must Do Tasks
Step 1 Setting aims and objectives
AIM
:
a broad strategic purpose
of a project, AIMS can be long
term, medium term or short term.
Objective
:
a specific
measurable goal whose
Why is setting behaviour
objectives so important?
Why so important?
1.
Set clear behavioural goals, everyone (including
yourself) knows what you want to achieve and are
asking of your target audiance
2.
You can messure it
3.
You can define what the barriers and benefits are
(because it’s tangible)……
4.
It ‘forces’ you to start with the target audience and
Focus on behaviour
Set a behavioural goal:
2 % less obesity is not a behavioural
goal,
taking the stairs is
………
Better awareness of environment isn’t,
recycling and so really throwing away your
glass away is….
Example of desired
S
pecific:
Not open to different interpretations
M
easurable:
Can observe & collect objective measures
A
chievable:
Not attempting too much
R
ealistic:
Given human & financial resources
T
ime bound:
Time scales / stages clearly identified
Objectives can be divided into:
Affective
objectives
, focused
of feelings / emotion
Cognitive
objectives
, focused
on learning / knowing
Psychomotor
objectives
,
focused on doing/ behaviour
OLD KAB MODEL ?
Knowledge
Attitude
As a 2 write an
As a 2 write an
Objectives can be divided into:
Affective
objectives
, focused
of feelings / emotion
Cognitive
objectives
, focused
on learning / knowing
Psychomotor
objectives
,
focused on doing/ behaviour
How to use behavioural
objectives to evaluate social
marketing programmes
It happens every day
Colleagues,
I recently ran a multi-channel media communications
campaign for promoting colon cancer screening
(CCS). I would have preferred an all-inclusive social
marketing approach, but time, budget and the small ‘p’
political considerations happened.
One significant component left out of the mix was
evaluation. At the time of our initial planning the team
had neither funding nor an epidemiologist/evaluator. As
the end of the fiscal year draws near, additional funds
have been identified for evaluation.
Evaluation
•
Judgement based on careful
assessment and critical
appraisal (WHO 1981)
•
Making a judgement about the
value of something by looking
critically (Ewles and Simnett
Evaluation can be defined as:
•
A process
which sets out to determine
systematically
and
objectively
the,
effectiveness, efficiency and
relevance of activities based on
specified objectives. Evaluation is a
process for improving current
activities, future programmes.
Evaluation can also aid
decision-making about resource allocation.
In 3’s discuss
Why do people not
evaluate?
Invisible Evaluation
In Victoria AUSTRALIA , during 2001, the
Government allocated $ 6 MILLION to a new
advertising and communications campaign to
reduce problem gambling
.
IN 2002 Calls to the problem gambling helpline
increased by 70% and face-to face clients
increased by 118%.
The Department’s 2002-2003 Annual
Report stated that an evaluation of the
campaign had been funded and was in
progress,
however, the results of
Cause and effect
In 2’s why do people
evaluate?
1. To assess the extent to which projects are
achieving their aims and objectives
2. To inform the developments of materials,
methods and other interventions
3. To ensure ethical practice
9 reasons for evaluation
5. To inform programme planning
6. To inform practice and contribute to
the evidence base
7. Provide funders with evidence of
success
4 step Evaluation
1.
Setting Aims and Objectives
2.
Identifying Evaluation Indicators
3.
Choosing data collection methods
What is a metric?
A set of numbers that
indicate that the objective
you have set is being
What kinds of measures
or metrics can we use to
measure
social marketing
programmes?
What Makes a good metric?
1. Rational
2. Relevant
3. Valid
4. Comparable
5. Timely
6. Scalable
7. Easily explained and communicated
8. Not induce perverse incentives
or unintended consequences
You have identified a perfect metric when:
1. Nobody complains about it!
2. Users say it helps them choose services & builds confidence
3. Colleagues say it empowers and encourages them
4. Managers use it to improve efficiency
Evaluation
Formative Evaluation:
Indicators e.g.: Did we have the
right data, did we use the right theory and evidence, did we set the
right objectives.
Process Evaluation:
Indicators
: e.g.: How many people took
part in or took advantage of a service that was offered, how many people
were contacted.
Impact Evaluation
:
Indicators
: e.g.: Data on immediate impact
of the work. For example: reported increase in knowledge, reported
behaviour or an immediate increase in the take up of a service.
Cost Per Lead
CPL
= TPB
TLG
CPL
= Cost Per Lead
TPB = Total
Programme Budget
Cost Per Change
CPC = TPB
NODBC
CPC
= Cost Per Change
TPB = Total Programme Budget
£100,00
ROI
(Return On Investment)
The ratio of £ returned for every £ spent, once
the activity has paid for itself e.g.
£100,000
Spent
£500,000 Gained
£400,000
Net
Example estimates of ROI on a proposal to extend smoking cessation
clinic opening times
Numbers of current attendance annually
1,000
Percentage increase projected by increasing opening
times
10%
Number of new clients
100
Percentage of new clients expected to quit
25%
Number of additional quitters
25
Average cost saving per quitter
£3,000
Annual gross cost saving (25 x £3,000)
£75,000
Annual costs of providing extended opening
£20,000
Net cost savings
£55,000
Insight driven
Evidence driven
Consistent
Sustained
THINK! CAMPAIGN SUCCESS
ROI as social ‘value’
Saved over 1000 lives and counting
Prevented 90,000 injuries and counting
Represents 3,494 people (and counting) who
are alive and uninjured today who wouldn’t
have been without THINK!
The monetary value to society was
Educational, enforcement and
engineering to improve road
safety.
Pre Think! total casualties grew
by 3%, and total accidents by
2%. Since Think! total casualties
and accidents declined by 12%
and 11% respectively.
For more information see:
3 more ways to assess
efficiency
CBA
VFM
C
ost
B
enefit
A
nalysis
The cost of the problem V the cost
savings from the intervention
Payback
Generating economic value
e.g.
recruitment of the best teachers improves education and economic output
Generating revenue
e.g.
a campaign to encourage people to inform on tax avoiders
Saving public money directly
e.g.
encouraging more people to complete tax returns online
Saving public money indirectly
Value For Money' (
VFM
)
A
ssesses if an organisation has obtained the
maximum benefit from the resources it controls.
Some elements may be subjective, difficult to
measure, and intangible. Judgements are required
.
T
akes account of : quality, cost, resources
used, fitness for purpose, timeliness, and
convenience to judge whether or not,
ROI and VFM
100% Effective
100% Efficient
100% Quality
Compliant
0% Effective
0% Efficient
0% Quality
Compliant
Range
of
realistic
impact
Range
of
realistic
impact
Learning and
continuous
improvement is the
major reason for
identifying and
using performance
metrics
When to use SMART Objectives and
when to use more general Goals
SMART
OBJECTIVES AND
PRESCRIBED
SYSTEMS AUDIT
MORE
GENERAL
GOALS AND
EVALUATION
When to use tight ‘
Objectives
’
When to use more general
‘Goals’
When there is:
Strong evidence & consensus, a high degree of
specification of objectives
and methodology is most
effective.
When there is :
Why is ownership of evaluation and
behaviour goals important for
organisations ?
1. Self assessment creates ownership
and can be a powerful tool for
improvement
2. Without ownership target fixing and
sabotage is possible
Involving recipients in evaluation
1.
Helps to ensure that
the project is meeting
needs
2.
Strengthens
community ownership
of the project
3.
Helps to build
community capacity
4.
Can contribute to the
sustainability of the
project
Measurement Culture
Performance
Culture
Metrics must
align with
Strategic Goals
and be
Simple enough that
each person sees that
their work contributes
to achieving the
selected metrics
.
SM a good buy ?
YES
An Ounce of Prevention – what are the returns? (2
nd
Edn 1999) CDC - USA
ftp://ftp.cdc.gov/pub/Publications/mmwr/other/ozprev.pdf
Returns on Investment in Public Health: An
epidemiological and economic analysis (2003) Access
Economics - Australia
http://www.health.gov.au/internet/wcms/publishing.nsf/Content/health-pubhlth-publicat-document-roi_eea-cnt.htm
Securing good health for the whole population: Final
report - February 2004
http://www.dh.gov.uk/en/PublicationsAndStatistics/Publications/PublicationsPolicyAndGuidance/DH_4074426
CDC Community Programme
Evaluation of
programme
objectives is
about legacy
Many