Key Principles and Best Practice for Evaluation of
RD&E programs and projects
Research outputs need to be monitored and evaluated to determine whether research outcomes are being met. Feedback through this process is essential to the R,D&E system. Evaluations are conducted to provide information to be used to assist with overall management and to provide objective evidence about stated achievements.
Evaluation is undertaken to:
• Help understand and improve performance; and
• Identify, calculate and communicate costs and benefits.
The following principles generally apply at program, project or organisational level.
Early design of evaluation strategy
The evaluation strategy should be worked out at the outset, as part of the design phase for R,D&E program and projects. Performance measurement should be factored into the design of the program and decisions made about what information to collect during this phase.
• It is often difficult and costly to collect information and data later, particularly if systems have not been established to do so.
• Take care deciding what information and data you need to collect – do not impose onerous collection requirements – be selective and use innovative measures.
• Articulating a clear ‘program logic’ or ‘theory of change’ will make it much easier to plan an appropriate evaluation strategy and identify supporting data requirements.
The evaluation process and evaluation results should facilitate learning and program improvement. It should therefore be seen as an integral and helpful component of a program, rather than something to be ‘added on’ towards the end.
Regular review and monitoring
Regular review and monitoring should be part of an integrated approach to program management and is an essential precursor to an effective program evaluation. Organisations need to be monitoring a program’s progress and be able to assess where it is at any point in time. This will only be possible if the evaluation has been ‘built in’ to the design of the program.
Page 2 of 8 PISC RD&E Subcommittee
Constant questioning is good, for example: • Is it working?
• Is it delivering the expected benefits?
• Is it still relevant for now? Has something changed?
• Is increased resourcing or a change to program design needed to get it back on track?
• Should it continue to be funded or is there some better alternative? o Program managers need to be prepared to make the hard
decisions and cut something that is not working.
Transparency and independence in evaluations
• The evaluation process should be as transparent as possible.
o It will not always be possible or practicable to have an external or independent body conduct major evaluations.
o Participatory evaluation may be more appropriate in some cases
• Regardless of the options chosen, it is crucial to ensure that governance arrangements are sound. Involve outsiders in the evaluation, or use an independent reference group to oversight the evaluation.
Method for evaluation
• A diversity of approaches may be used to evaluate RD&E programs – no one method is right or suits all programs.
o The method to be used depends on the questions being asked and the nature of the program being evaluated.
• Whatever method is used, the evaluation needs to be systematic, robust and evidence-based.
A list of documents and guidelines that can assist with determining the appropriate method is at Appendix A.
Utilisation of findings
When planning evaluation it is important to have an end point in mind and incorporate a complete feedback loop. The findings need to be utilised to improve future investments and decision making as well as improving the management of a program or project from an organisational perspective. The learning needs to be disseminated. This applies to both funders and investors in terms of assessing whether goals and outcomes have been met.
Communication of evaluation processes and results
• The needs of different stakeholders (e.g. program clients, decision makers etc) should be considered in developing a communication strategy. This is necessary to facilitate engagement with the evaluation process and to ensure results are taken on board.
• It is important for departments and agencies to disseminate learnings from evaluations. In particular, ‘mistakes’ are more likely to be repeated if poor results are delayed or not released, or sanitised.
Performance measures
Ensure you have both efficiency and effectiveness measures:
• Collect only what is relevant to monitoring and evaluating the program’s performance.
• A clear program logic and understanding of the results the program is intended to achieve will help identify appropriate performance measures. Cost-benefit analysis
Evaluation serves to help our understanding to make better investment decisions in the future and manage the programs and projects that underpin these investments. Cost-benefit analysis will strengthen the evaluation process and where possible and appropriate, both qualitative and quantitative analysis should be undertaken.
There are some common agreed standards of rigour that cost-benefit analysis, regardless of the methodology used, should address. This will help to facilitate comparison of results derived using different evaluation methodologies. These are contained at Appendix B.
Page 4 of 8 PISC RD&E Subcommittee
Appendix A – Evaluation Reference Documents
The following is a list of evaluation documents that you may find useful in developing your own evaluation processes procedure and techniques. The Australasian Evaluation Society also provides links to many overseas documents, conference and regional networks.
Agency Title Year
PISC RD&E Subcommittee member documents
The Department of Agriculture and Food WA
The Department of Agriculture and Food Evaluation Policy
2010
Department of Primary Industries Victoria
DPI Agriculture and Fisheries Investment Performance Measurement: Evaluation, Monitoring and Reporting Guideline
2009
CRRDC Guidelines for Evaluation 2009
CRRDC Instructions for the Preparation of Standardised CBAs
Australian Government National Resource Management
Caring for our Country: monitoring, evaluation, reporting and improvement strategy
2010
Public Administration Today (publication)
Measuring Contributions to Sustainable Development – A Practical Approach
2004
Bureau of Rural Sciences Signposts for Australian Agriculture
Australian Government documents
ANAO Expenditure Review Principles 2009
PM&C Implementation of Programme and Policy
Initiatives
2006
DIISR Framework of Principles for Innovation
Initiatives
2009
DIISR Innovation Metrics Framework 2009
Not-For-Profit Journal (publication)
Techniques of Social Evaluation
Documents from overseas agencies
BIS (UK) Guidance for using Additionality Benchmarks in
Appraisal
2009
OECD Enhancing public research performance
through Evaluation, Impact Assessment and Priority Setting
Agency Title Year
Documents from overseas agencies
HM Treasury (UK) The Green Book – Appraisal and Evaluation in
Central Government
2003
Cabinet Office (UK) The Magenta Book – guidance notes on Policy
Evaluation
2008
European Commission Smart Innovation: A Practical Guide to
Evaluating Innovation Programs
2006
UNFPA The Programme Manager’s Planning,
Monitoring and Evaluation Toolkit
CDC (US) Framework for Program Evaluation in Public
Health
1999
NESTA Evaluating the Enterprising Further Education
Pilot
2010
Research Councils UK Evaluation: Practical Guidelines 2005
Canadian Evaluation Society Program Evaluation Standards 2008
Others
Federation of Australian Scientific and Technological Societies
Giving Preparedness a Central Role in Science and Innovation Policy
Page 6 of 8 PISC 20: Item 6.4
Appendix B – Standards for Cost-benefit Analysis
There are some common agreed standards of rigour that cost-benefit analysis, regardless of the methodology used, should address.
Counterfactual
• Ensure you try to take into account the counterfactual situation in a robust way. What would have happened in the absence of this particular program/project? For example:
o Take into account that someone might have done this piece of research over a similar timeframe anyway.
o Consider whether the work brought forward research and
outcomes that would have taken place anyway but at a later date. • Using the appropriate counterfactual helps ensure that the true cost of
funding the particular program/activity is taken into account in the
assessment of its benefits. It also makes it easier to compare the results of different evaluations.
Cost of implementation
• Ensure all costs of implementation, for example, by business, industry, governments or consumers are taken into account when calculating the net benefits arising from the program.
Opportunity cost
• It is essential to take into account the opportunity cost of directing funds to a particular program/project, as the funds could have been used for another purpose by all parties involved.
o At a minimum, use the long term Government bond rate. This takes into account the fact that investors are foregoing the
opportunity to generate a risk free return on their investment equal to the long term Government bond rate.
Additionality and crowding out
• Don’t count benefits resulting from program activities or outputs that are not additional, for example, benefits resulting from research that would have been undertaken in any event. See ‘counterfactual’ section above. • Take into account any crowding out. This occurs where expenditure that
would have been undertaken by the private sector is displaced by
government expenditure that is financed by borrowing, which may lead to increases in interest rates, which curb individual expenditure.
o Crowding out is more likely to be an issue in relation to individual publicly funded projects, where the benefits are likely to be used by relatively narrow groups.
Benefits from a program/project/activity
• Identify economic, social and environmental benefits wherever possible. o Social and environmental benefits can be more difficult to put a
value on, but can be very relevant to science and innovation programmes.
• Ensure you have external verification of claimed benefits.
• Look at preparedness as an outcome/benefit of science and innovation: o Preparedness is becoming more relevant as economies and
societies become more connected in a global sense, particularly with regard to terrorism and pandemic diseases, for example. o The concept of preparedness as an outcome class also relates to
“forewarning” markets and the general community of future risks, in so doing bringing forward responses and mitigating future (higher) costs.
o Preparedness as an outcome class is also a useful feature of investments in capability-building.
• For economic benefits:
o Changes in consumption are more meaningful than changes in Gross Domestic Product.
o Take into account time lags an discount future economic benefits to net present value terms;
o the extent to which they can be attributed to the
project/programme should be taken into account [see above]. o When quantifying future economic benefits: involve the expected
end-users/beneficiaries; take account of previous observed impacts from similar programs in that particular sector; and take into account historically achieved productivity growth rates in that sector.
o Factor risks into the impact assessment process – one way to do this is to use expected value analysis, where probability estimates are assigned to alternative outcomes and expected outcomes generated.
Spillovers
• Identify any spillovers that are likely to occur. These are benefits that cannot be appropriated fully by one party or group (including businesses, individuals).
o The merits of public funding where there are no or few spillovers are questionable.
o Programs or projects which produce substantial spillovers are more appropriately funded by government.
o The existence of positive spillovers is not a sufficient reason to justify public expenditure.
Page 8 of 8 PISC 20: Item 6.4 Attribution
• In many cases, the particular program may be only one of a number of important contributors to an outcome.
o Where this is the case, it can be difficult to decide what proportion of the outcome can reasonably be attributed to the existence of the particular program.
o Clearly state any assumptions made about attribution rates. It is highly unlikely that one intervention will be the only contributor – particularly for longer term programs and environmental and social change.
