The following are some discussion topics foundations may find useful.
• Clarity around the roles and purposes of your foundation and its grants
• Clarity and agreement regarding terms/language.
• Horses for courses: measures and purposes
• Encourage and reward honesty
• Re-consider the foundation’s role in dissemination
• Beware the tail wagging the dog
• More data are little help unless organised and used
• Beware ‘outcome creep’
• Puzzles and mysteries
Each of these topics is discussed in greater depth in the following pages.
Clarity around the roles and purposes of your foundation and its grants
A strategy for measurement requires clarification of the roles and purposes of your foundation. Is your purpose to change the world, or to bring a family together, or to involve employees, or to develop your business? Do you see yourselves as existing to fund what works? Or what doesn’t – yet? Not all grant-making is outcome driven: ‘The small grants program you might describe as being one more grounded in compassion than a very structured sense of trying to achieve outcomes other than what I call EBOs - evidence of the bleedin’ obvious. If a family hasn’t got a fridge you don’t need much of an outcome report when you give them one’. (The measurement purist would undoubtedly argue that if you want to measure the outcome then you need to measure whether the fridge improved the family’s diet and long term well-being, and the children’s performance in school etc. But barring the possibility that the fridge gets filled with soft drinks, chocolate and cream cakes, it seems reasonable
42 to suppose that in Australia having a fridge allows a healthier and more balanced diet than not having one).
Clarity and agreement regarding terms/language.
One of the most striking, but hardly surprising, findings was the ways in which the same terms are used with very different meanings (for a very brief review of the variety found in the literature see Appendix One). We suggest that it may be useful to distinguish between the following:
Monitoring – often described as acquittal reporting
This may be straight fiscal monitoring/accounting; and/or monitoring against previously agreed objectives and key performance indicators
Evaluation – understanding and assessing what worked, or did not, how and why.
This may be formative – ‘a peek under the bonnet’; or summative – an ‘end of term’ assessment. It may be formal - undertaken by an independent specialist; or informal story telling by the grantee (purists might put story telling into another box)
Rationalist/technical output or outcome measures:
Using financial or other metrics, and often control groups, to measure outputs (immediate effects) or outcomes (wider/medium term effects) e.g. SROI; measures used in Social Impact Bond
calculations etc. See, for example, www.gov.uk/social-impact-bonds.
Impact measurement
Measuring the long term outcomes that may be attributed to the grant (and any associated activities) adjusted for what would have happened anyway, the actions of others, drop-off and displacement (see for example, Hehenberger et al., 2013)
43 The key difference between evaluation and much outcome measurement is that evaluation explores how and why something worked whereas outcome measurement is primarily interested in whether it worked. Impact measurement takes outcome measurement and evaluation a stage further and sets a very high bar. In this strict sense, we did not encounter any foundation attempting it.
As a general rule the cost and time required increase as you move up the scale from monitoring to impact measurement. Some would also argue that the reliability of the evidence also decreases as you move down the scale – but reliability may be different from utility.
When we asked respondents about the purposes of measuring very broadly they gave the following answers:
• Accountability of grantees
• Accountability of the foundation managers
• Course correction
• Risk assessment
• Learning for future grant-making
• Learning for replication
• Enhancing reputation and legitimacy (e.g. for raising funds/business)
• Engaging family, employees, community etc.
Horses for courses: measures and purposes
So the message is – decide what you want measurement for and then choose the least costly and most timely method for your purposes. For example, do not engage in costly long-term impact measurement if your primary goal is to ensure basic accountability of grantees and foundation managers, and/or to engage family members and enhance immediate reputation.
For a number of our respondents monitoring plus stories fulfilled their primary purposes of basic grantee and foundation manager accountability and family or employee engagement. It is difficult to learn very much from one grant – there are too many other factors at play – but looking at a whole program or a collection of similar grants in different places at different times may yield richer results.
A few foundations were using this approach to identify recurring issues worth investigating further.
44 Encourage and reward honesty
Respondents emphasised how much they wanted grantees to feel able to be honest with them, and to tell them when things were going wrong so that the problem could be resolved if at all possible. Do you give your grantees ‘permission’ (e.g. explicitly asking what went wrong as though that were a natural expectation) and opportunities to talk about things that are not going so well on a regular basis?
Re-consider the foundation’s role in dissemination
Relatively few foundations saw it as their responsibility to play a role in disseminating results of projects that went well (clearly disseminating poor results is a more complicated issue). And some gave considered reasons why they did not play this role. It may be useful for every foundation to think through its position on dissemination, and whether this may sometimes be a way in which they could
‘supercharge’ the value of their grants.
Beware the tail wagging the dog
Are you ever attracted to program areas or projects because they are easy to measure and/or produce tangible results?
‘They do a lot of employment generation type work trying to get unemployed people into paid employment ... I don’t know that any of the family is particularly passionate about that, with the exception of perhaps the Chairman, but because it is so easy to measure and it’s easy to see results, it’s considered incredibly successful’.
More data are little help unless organised and used
Do you in fact use the data you already have? ‘We’re on a learning journey and as we develop we’ll be better at reporting processes that feed the information in ways that we want. The fact is we’ve got masses of data here but it’s not very well organised’.
Beware ‘outcome creep’
One of the often suggested risks of measurement in grant-making is that it will encourage funders to go for success and/or the ‘low hanging fruit’. Interestingly the majority of our respondents did not perceive this as a risk; indeed, many suggested that measurement/monitoring made them more willing to take risks because they were aware of the challenges ahead and could course-correct if necessary.
45 This raises the interesting possibility that the main value of measuring is not what comes out at the end but the thinking and analysis that goes in at the beginning; an idea summed up by DiMaggio (2002) in the suggestion that outcome measurement in the social world is probably impossible but possibly useful.
But outcome measurement in particular may carry another danger – the focus on outcome reduces the means to a merely secondary consideration. There are plenty of social problems that would be easy to ‘solve’ if there were no limitations on the means used to do so.
Puzzles and mysteries
Many of the issues that foundations address are more like mysteries than puzzles; in other words, there is no single solution and it is not known. This has a number of implications. One is that hard measures need to be kept in perspective. The second is that we need to leave room for serendipity.
As one respondent noted: ‘Before Christopher Columbus went to the New World he had no idea where he was going. When he got to the New World he had no idea of where he was. And when he came back from the New World he had no idea of where he’d been’.