Discussion and conclusions
3. What is valuable about engagement processes and how can we evaluate them?
A key contribution of the research is building on the limited knowledge base on what decision-makers value about science engagement as opposed to what scientists value. Most prominent was better access to credible knowledge to give decisions legitimacy. This is reasonably well established for scientific knowledge, but elicitation of decision-maker knowledge was also highly valued.
Engagement as opposed to information provision offered a number of benefits related to: the social aspect which can initiate and consolidate professional collaborations and provide space for reflexivity; a prompt for cognitive processing of information which may not occur in passive communication modes; the opportunity to make and question claims which can lead to deeper understanding.
However, a key finding was that the benefits of discrete engagement activities manifest in individual participants (e.g. learning) may have limited longevity with high staff churn evident in the NRM domain.
155
An evaluation framework and principles were presented incorporating these findings. These differ from existing PES frameworks because they necessarily have a different normative basis that reflects the institutional setting of decision-makers. That is, engagement is about accessing, questioning and understanding relevant knowledge sources rather than becoming empowered through access to knowledge. There is also more emphasis on outcomes in acknowledgement that engagement will likely not happen without the promise of tangible outcomes.
Strengths and limitations of the research
One of the big questions raised by this research is how applicable the findings are to decision- makers in other policy domains and the private sector. The high degree of education and trust in science seen in NRM decision-makers may not be universal, for example.
There are also limitations associated with making broad generalisations and comparisons based on only two case studies. Within the cases I also did not observe dialogue and deliberation first hand. Apart from limitations in recall, identification of important aspects of the engagement by interviewees may have been limited.
On the other hand, seeking out the decision-maker perspective about science engagement highlighted how much these processes have been framed by academics and allowed some common assumptions to be tested. The focus on knowledge co-production as a communicative process rather than a decision-making process was also useful to open up ideas about what constitutes value. The opportunity to explore these values five years after project completion was also rare and a strength.
Future directions
Beyond addressing the limitations mentioned above, the value of the current work would be increased by testing the evaluation framework and operationalising it i.e. identifying appropriate measures.
With my evaluative focus, process and outcomes were key areas of analysis. However, a broader study of decision-maker engagement might also look at: how individual traits influence these processes and how different cultural and political contexts impact processes and outcomes.
Other interesting points to explore in greater depth would be around loss of expertise and the implications for the legacy of engagement as well as public administration (should greater
156
value be placed on retaining experienced staff?); how individual change is linked to institutional change; and how cross institutional narratives develop and the nature of their impact.
157
References
Abelson, J., Forest, P.-G., Eyles, J., Smith, P., Martin, E., & Gauvin, F.-P. (2003). Deliberations about deliberative methods: Issues in the design and evaluation of public participation processes.
Social Science & Medicine, 57(2), 239-251.
Allum, N., Sturgis, P., Tabourazi, D., & Brunton-Smith, I. (2008). Science knowledge and attitudes across cultures: A meta-analysis. Public Understanding of Science, 17(1), 35-54. doi: 10.1177/0963662506070159
Argyris, C., & Schön, D. A. (1996). Organisational learning II: Theory, method and practice. Reading MA Addison-Wesley.
Armitage, D., Berkes, F., Dale, A., Kocho-Schellenberg, E., & Patton, E. (2011). Co-management and the co-production of knowledge: learning to adapt in Canada's Arctic. Global Environmental Change, 21(3), 995-1004. doi: 10.1016/j.gloenvcha.2011.04.006
Arnstein, S. R. (1969). A ladder of citizen participation. Journal of the American Institute of Planners, 35(4), 216-224.
Bauer, M. W. (2008). Paradigm change for science communication: Commercial science needs a critical public. In D. Cheng, M. Claessens, T. Gascoigne, J. Metcalfe, B. Schiele & S. Shi (Eds.),
Communicating science in social contexts: New models, new practices (pp. 7-25). Dordrecht, The Netherlands: Springer.
Bauer, M. W., Allum, N., & Miller, S. (2007). What can we learn from 25 years of PUS survey research? Liberating and expanding the agenda. Public Understanding of Science, 16(1), 79- 95. doi: 10.1177/0963662506071287
Beierle, T. C. (1998). Public participation in environmental decisions: An evaluation framework using social goals. Washington, DC: Resources for the Future.
Bevir, M. (2009). Key Concepts in Governance [SAGE Publications] doi:10.4135/9781446214817 Biegelbauer, P., & Hansen, J. (2011). Democratic theory and citizen participation: democracy models
in the evaluation of public participation in science and technology. Science and Public Policy, 38(8), 589-597. doi: 10.3152/030234211X13092649606404
Bielak, A. T., Campbell, A., Pope, S., Schaefer, K., & Shaxson, L. (2008). From science communication to knowledge brokering: The shift from ‘science push’ to ‘policy pull’. In D. Cheng, M. Claessens, T. Gascoigne, J. Metcalfe, B. Schiele & S. Shi (Eds.), Communicating science in social sontexts: New models, new practices. (pp. 201-226). Dordrecht, The Netherlands: Springer.
Boaz, A., Fitzpatrick, S., & Shaw, B. (2009). Assessing the impact of research on policy: A literature review. Science and Public Policy, 36(4), 255-270. doi: 10.3152/030234209X436545
Bogner, A. (2012). The paradox of participation experiments. Science, Technology, & Human Values, 37(5), 506-527. doi: 10.1177/0162243911430398
Bottrill, M. C., & Pressey, R. L. (2012). The effectiveness and evaluation of conservation planning.
Conservation Letters, 5(6), 407-420. doi: 10.1111/j.1755-263X.2012.00268.x
Bradshaw, G. A., & Borchers, J. G. (2000). Uncertainty as information: Narrowing the science-policy gap. Conservation Ecology, 4(1), 7.
Bryman, A., & Burgess, G. (1999). Qualitative Research London: Sage Publications.
Bucchi, M. (2008). Of deficits, deviations and dialogues: Theories of public communication of science. In M. Bucchi & B. Trench (Eds.), Handbook of public communication of science and technology (pp. 57-76). Oxon, UK: Routledge.
Bucchi, M., & Trench, B. (2014). Science communication research: Themes and challenges. In M. Bucchi & B. Trench (Eds.), Routledge Handbook of Public Communication of Science and Technology (2nd ed., pp. 1-14). Oxon, UK: Routledge.
Burgess, J., & Chilvers, J. (2006). Upping the ante: A conceptual framework for designing and
evaluating participatory technology assessments. Science and Public Policy, 33(10), 713-728. doi: 10.3152/147154306781778551
158
Burgess, M. (2014). From ‘trust us’ to participatory governance: Deliberative publics and science policy. Public Understanding of Science, 23(1), 48-52. doi: 10.1177/0963662512472160 Carew, A. L., & Wickson, F. (2010). The TD wheel: a heuristic to shape, support and evaluate
transdisciplinary research. Futures, 42(10), 1146-1155. doi: 10.1016/j.futures.2010.04.025 Cash, D. W., Borck, J. C., & Patt, A. G. (2006). Countering the loading-dock approach to linking
science and decision making comparative analysis of El Niño/Southern Oscillation (ENSO) forecasting systems. Science, Technology, & Human Values, 31(4), 465-494. doi:
10.1177/0162243906287547
Cash, D. W., Clark, W. C., Alcock, F., Dickson, N. M., Eckley, N., Guston, D. H., . . . Mitchell, R. B. (2003). Knowledge systems for sustainable development. Proceedings of the National Academy of Sciences, 100(14), 8086-8091. doi: 10.1073/pnas.1231332100
Caudron, A., Vigier, L., & Champigneulle, A. (2012). Developing collaborative research to improve effectiveness in biodiversity conservation practice. Journal of Applied Ecology, 49(4), 753- 757. doi: 10.1111/j.1365-2664.2012.02115.x
Chapman, S., Sullivan, C., Palm, C., Huynh, U., Diru, W., & Masira, J. (2016). Monitoring and evaluation to support adaptive co-management: Lessons learned from the Millennium Villages Project. Journal of Environmental Management, 183, 142-151. doi:
10.1016/j.jenvman.2016.08.014
Chess, C., Dietz, T., & Shannon, M. (1998). Who should deliberate when? Human Ecology Review, 5, 45-48.
Chess, C., & Purcell, K. (1999). Public participation and the environment: Do we know what works?
Environmental Science and Technology, 33(16), 2685-2692. doi: 10.1021/es980500g Chilvers, J. (2008). Deliberating competence: Theoretical and practitioner perspectives on effective
participatory appraisal practice. Science, Technology, & Human Values, 33(2), 155-185. doi: 10.1177/0162243907307594
Chilvers, J. (2012). Reflexive engagement? Actors, learning, and reflexivity in public dialogue on science and technology. Science Communication, 35(3), 283-310. doi:
10.1177/1075547012454598
Chilvers, J., & Kearnes, M. (2016). Science, democracy and emergent publics. In J. Chilvers & M. Kearnes (Eds.), Remaking participation: Science, environment and emergent publics (pp. 1- 28). Oxon, UK: Routledge.
Cid, C. R., & Pouyat, R. V. (2013). Making ecology relevant to decision making: the human-centered, place-based approach. Frontiers in Ecology and the Environment, 11(8), 447-448. doi: 10.1890/1540-9295-11.8.447
Collins, W., & Anderson, S. (Eds.). (2006) Collins English Dictionary: Complete & Unabridged. Collins. Commonwealth of Australia. (2010). Inspiring Australia. A National Strategy for Engagement with
the Sciences. Canberra: Commonwealth of Australia Retrieved from http://www.innovation.gov.au/
Commonwealth of Australia. (2012a). An overview of the clean energy legislative package. Canberra: Commonwealth of Australia.
Commonwealth of Australia. (2012b). Securing a clean energy future: Implementing the Australian government’s climate change plan. Canberra: Commonwealth of Australia.
Commonwealth of Australia. (2013). Science and Technology Engagement Pathways. Community involvement in science and technology decision-making. Canberra: Department of Industry, Innovation, Research and Tertiary Education Retrieved from
https://archive.industry.gov.au/industry/IndustrySectors/nanotechnology/Publications/Com munity-Engagement/Documents/DraftSTEPFramework.pdf.
Commonwealth of Australia. (2014). National Environmental Science Programme Guidelines. Canberra: Commonwealth of Australia Retrieved from
http://www.environment.gov.au/system/files/pages/2f561690-b47e-4bf2-b028- d18739b3486f/files/national-environmental-science-programme-guidelines.pdf.
159
Commonwealth of Australia. (2017a). Australia's national science statement. Canberra: Commonwealth of Australia Retrieved from
http://www.science.gov.au/scienceGov/NationalScienceStatement/index.html. Commonwealth of Australia. (2017b). Engagement and impact assessment pilot 2017 report.
Canberra: Commonwealth of Australia.
Cook, C. N., Carter, R. B., Fuller, R. A., & Hockings, M. (2012). Managers consider multiple lines of evidence important for biodiversity management decisions. Journal of Environmental Management, 113, 341-346. doi: 10.1016/j.jenvman.2012.09.002
Cook, C. N., Mascia, M. B., Schwartz, M. W., Possingham, H. P., & Fuller, R. A. (2013). Achieving conservation science that bridges the knowledge–action boundary. Conservation Biology, 27(4), 669-678. doi: 10.1111/cobi.12050
CSIRO. (2014a). About AdaptNRM Retrieved 28/11/16, from http://adaptnrm.csiro.au/about- adaptnrm/
CSIRO. (2014b). Shared learning campus Retrieved 27/06/17, from
http://adaptnrm.csiro.au/shared-learning/shared-learning-campus/
Cullen, P., Cottingham, P., Doolan, J., Edgar, B., Ellis, C., Fisher, M., . . . Whittington, J. (2001). Knowledge seeking strategies of natural resource professionals. Synthesis of a workshop held in Bungendore, NSW from 5-7th June 2000. Canberra: The Cooperative Research Centre for Freshwater Ecology.
Curtis, A., Ross, H., Marshall, G., Baldwin, C., Cavaye, J., Freeman, C., . . . Syme, G. J. (2014). The great experiment with devolved NRM governance: lessons from community engagement in Australia and New Zealand since the 1980s. Australasian Journal of Environmental Management, 21(2), 175-199. doi: 10.1080/14486563.2014.935747
Cvitanovic, C., Hobday, A. J., van Kerkhoff, L., Wilson, S. K., Dobbs, K., & Marshall, N. (2015). Improving knowledge exchange among scientists and decision-makers to facilitate the adaptive governance of marine resources: a review of knowledge and research needs. Ocean & Coastal Management, 112, 25-35. doi: 10.1016/j.ocecoaman.2015.05.002
Davies, S. R. (2013). The rules of engagement: Power and interaction in dialogue events. Public Understanding of Science, 22(1), 65-79. doi: 10.1177/0963662511399685
Dawson, E. (2018). Reimagining publics and (non) participation: Exploring exclusion from science communication through the experiences of low-income, minority ethnic groups. Public Understanding of Science, 27(7), 772-786. doi: 10.1177/0963662517750072
Delgado, A., Lein Kjølberg, K., & Wickson, F. (2011). Public engagement coming of age: From theory to practice in STS encounters with nanotechnology. Public Understanding of Science, 20(6), 826-845. doi: 10.1177/0963662510363054
Dickinson, J. L., Shirk, J., Bonter, D., Bonney, R., Crain, R. L., Martin, J., . . . Purcell, K. (2012). The current state of citizen science as a tool for ecological research and public engagement.
Frontiers in Ecology and the Environment, 10(6), 291-297. doi: 10.1890/110236
Diesing, P. (1973). Reason in society: Five types of decisions and their social conditions. Westport, Conn: Greenwood Press.
Driver, A., Nel, J. L., Snaddon, K., Murray, K., Roux, D. J., Hill, L., . . . Funke, N. (2011). Implementation manual for freshwater ecosystem priority areas (WRC Report No. 1801/1/11). South Africa: Water Research Commission.
Dryzek, J. S. (2002). Deliberative democracy and beyond: liberals, critics, contestations. Oxford: Oxford University Press.
Dunn, G., & Laing, M. (2017). Policy-makers perspectives on credibility, relevance and legitimacy (CRELE). Environmental Science and Policy, 76, 146-152. doi: 10.1016/j.envsci.2017.07.005 Dye, T. (1972). Understanding public policy. Englewood Cliffs, NJ: Prentice-Hall.
Einsiedel, E. F. (2008). Public participation and dialogue In M. Bucchi & B. Trench (Eds.), Handbook of public communication of science and technology (pp. 173-184). Oxon, UK: Routledge.
160
Emery, S. B., Mulder, H. A., & Frewer, L. J. (2015). Maximizing the policy impacts of public
engagement: A European study. Science, Technology, & Human Values, 40(3), 421-444. doi: 10.1177/0162243914550319
European Union. (2012). Responsible research and innovation: Europe's ability to respond to societal challenges (pamphlet): European Union.
Fazey, I., Bunse, L., Msika, J., Pinke, M., Preedy, K., Evely, A. C., . . . Reed, M. S. (2014). Evaluating knowledge exchange in interdisciplinary and multi-stakeholder research. Global
Environmental Change, 25, 204-220. doi: 10.1016/j.gloenvcha.2013.12.012
Fazey, I., Evely, A. C., Reed, M. S., Stringer, L. C., Kruijsen, J., White, P. C., . . . Trevitt, C. (2013). Knowledge exchange: a review and research agenda for environmental management.
Environmental Conservation, 40(1), 19-36. doi: 10.1017/S037689291200029X
Fearon, J. D. (1998). Deliberation as Discussion. In J. Elster (Ed.), Deliberative democracy. Cambridge Cambridge University Press.
Finlayson, C. M. (2001). Views from divergent stakeholders on the Macquarie–Cudgegong River Management Committee. Ecological Management & Restoration, 2(2), 87-98.
Fiorino, D. J. (1990). Citizen participation and environmental risk: A survey of institutional mechanisms. Science, Technology, & Human Values, 15(2), 226-243.
Fischer, A. R. H., Wentholt, M. T. A., Rowe, G., & Frewer, L. J. (2014). Expert involvement in policy development: A systematic review of current practice. Science and Public Policy, 41(3), 332- 343. doi: 10.1093/scipol/sct062
Forss, K. (2005). An Evaluation Framework for Information, Consultation and Public Participation. In J. Caddy (Ed.), Evaluating public participation in policy making (pp. 41-84). Paris:
Organisation for Economic Co-operation and Development.
Freeman, R. E. (1984). Strategic management: a stakeholder approach. London: Pitman. Friedman, A., J. (ed),. (2008). Framework for Evaluating Impacts of Informal Science Education
Projects: The National Science Foundation.
Funke, N., & Nienaber, S. (2012). Promoting uptake and use of conservation science in South Africa by government. Water SA, 38(1), 105-114. doi: 10.4314/wsa.v38i1.13
Gardner, J., Dowd, A.-M., Mason, C., & Ashworth, P. (2009). A framework for stakeholder engagement on climate adaptation (C. A. Flagship, Trans.): CSIRO
Gascoigne, T., & Metcalfe, J. (2017). The emergence of modern science communication in Australia.
Journal of Science Communication, 16(03), A01.
Godfrey, L., Funke, N., & Mbizvo, C. (2010). Bridging the science-policy interface: a new era for South African research and the role of knowledge brokering. South African Journal of Science, 106(5-6), 44-51. doi: 10.4102/sajs.v106i5/6.247
Goodwin, M. (2015). Political Science? Does Scientific Training Predict UK MPs Voting Behaviour?
Parliamentary Affairs, 68(2), 371-392. doi: 10.1093/pa/gst011
Grand, A., Davies, G., Holliman, R., & Adams, A. (2015). Mapping public engagement with research in a UK university. PloS one, 10(4), e0121874. doi: 10.1371/journal.pone.0121874
Grunig, J. E. (1997). A situational theory of publics: Conceptual history, recent challenges and new research. In D. Moss, T. MacManus & D. Vercic (Eds.), Public relations research: An international perspective (pp. 3-46). London: International Thomson Business.
Guba, E. G., & Lincoln, Y. S. (1989). Fourth generation evaluation. Newbury Park: Sage Publications. Guldin, R. W. (2003). Forest science and forest policy in the Americas: Building bridges to a
sustainable future. Forest Policy and Economics, 5(4), 329-337. doi: 10.1016/S1389- 9341(03)00042-X
Guston, D. H. (1999). Evaluating the first US consensus conference: The impact of the citizens’ panel on telecommunications and the future of democracy. Science, Technology & Human Values, 24(4), 451-482.
Habermas, J. (1984). The theory of communicative action, Volume I: Reason and the rationalization of society. Boston, MA: Beacon.
161
Habermas, J., Lennox, S., & Lennox, F. (1974). The public sphere: An encyclopedia article (1964). New German Critique(3), 49-55.
Hall, N. L. (2014). Can the “Social Licence to Operate” Concept Enhance Engagement and Increase Acceptance of Renewable Energy? A Case Study of Wind Farms in Australia. Social
Epistemology, 28(3-4), 219-238. doi: 10.1080/02691728.2014.922636
Hansen, J., & Allansdottir, A. (2011). Assessing the impacts of citizen participation in science governance: exploring new roads in comparative analysis. Science and Public Policy, 38(8), 609-617. doi: 10.3152/030234211X13111546663377
Haraway, D. J. (1991). Simians, cyborgs, and women: The reinvention of nature. New York: Routledge.
Haynes, A., Brennan, S., Redman, S., Williamson, A., Makkar, S. R., Gallego, G., & Butow, P. (2017). Policymakers’ experience of a capacity-building intervention designed to increase their use of research: a realist process evaluation. Health Research Policy and Systems, 15(1), 99. doi: 10.1186/s12961-017-0234-4
Head, B. W. (2007). Community engagement: participation on whose terms? Australian Journal of Political Science, 42(3), 441-454. doi: 10.1080/10361140701513570
Head, B. W. (2008). Three lenses of evidence-based policy. Australian Journal of Public Administration, 67(1), 1-11. doi: 10.1111/j.1467-8500.2007.00564.x
Head, B. W. (2010). From knowledge transfer to knowledge sharing? Towards better links between research, policy and practice. In G. Bammer (Ed.), Bridging the ‘Know–Do’ Gap. Knowledge brokering to improve child wellbeing. Canberra: ANU E Press.
Head, B. W. (2013). Evidence-based policymaking – Speaking truth to power? Australian Journal of Public Administration, 72(4), 397-403. doi: 10.1111/1467-8500.12037
HEFCE. (2017). Research Excellence Framework Retrieved 31/08/2017, from http://www.hefce.ac.uk/rsrch/REFimpact/
Hegger, D., Lamers, M., Van Zeijl-Rozema, A., & Dieperink, C. (2012). Conceptualising joint knowledge production in regional climate change adaptation projects: success conditions and levers for action. Environmental Science and Policy, 18, 52-65. doi:
10.1016/j.envsci.2012.01.002
Heink, U., Marquard, E., Heubach, K., Jax, K., Kugel, C., Neßhöver, C., . . . Vandewalle, M. (2015). Conceptualizing credibility, relevance and legitimacy for evaluating the effectiveness of science–policy interfaces: Challenges and opportunities. Science and Public Policy, 42(5), 676-689. doi: 10.1093/scipol/scu082
Hess, D. J. (2011). To tell the truth: On scientific counterpublics. Public Understanding of Science, 20(5), 627-641. doi: 10.1177/0963662509359988
Hilligoss, B., & Rieh, S. Y. (2008). Developing a unifying framework of credibility assessment: Construct, heuristics, and interaction in context. Information Processing & Management, 44(4), 1467-1484. doi: 10.1016/j.ipm.2007.10.001
Holmes, B., Scarrow, G., & Schellenberg, M. (2012). Translating evidence into practice: The role of health research funders. Implementation Science, 7(39).
Horlick-Jones, T., Walls, J., Rowe, G., Pidgeon, N., Poortinga, W., & O'Riordan, T. (2006). On
evaluating the GM Nation? Public debate about the commercialisation of transgenic crops in Britain. New Genetics and Society, 25(3), 265-288. doi: 10.1080/14636770601032858 Hutchings, J. A., & Stenseth, N. C. (2016). Communication of science advice to government. Trends in
Ecology & Evolution, 31(1), 7-11. doi: 10.1016/j.tree.2015.10.008
International Association for Public Participation. (2016), from https://www.iap2.org.au/About- Us/About-IAP2-Australasia-/Core-Values
IPBES. (2012). Functions, operating principles and institutional arrangements of the
Intergovernmental Science-Policy Platform on Biodiversity and Ecosystem Services
Retrieved 09/01/18, from https://www.ipbes.net/document-library-categories/policies-and- procedures
162
Irwin, A. (2008). Risk, science and public communication: Third-order thinking about scientific culture. In M. Bucchi & B. Trench (Eds.), Handbook of public communication of science and technology (pp. 199-212). Oxon, UK: Routledge.
Irwin, A. (2014). From deficit to democracy (re-visited). Public Understanding of Science, 23(1), 71- 76. doi: 10.1177/0963662513510646
Irwin, A., & Michael, M. (2003). Science, social theory and public knowledge. Maidenhead, Berkshire: Open University Press.
Jabbar, A. M., & Abelson, J. (2011). Development of a framework for effective community engagement in Ontario, Canada. Health Policy, 101(1), 59-69. doi:
10.1016/j.healthpol.2010.08.024
Jasanoff, S. (1990). The fifth branch: Science advisers as policymakers. Cambridge, MA: Harvard University Press.
Jasanoff, S. (2003). (No?) Accounting for expertise. Science and Public Policy, 30(3), 157-162. Jasanoff, S. (2014). A mirror for science. Public Understanding of Science, 23(1), 21-26. doi:
10.1177/0963662513505509
Jensen, E. (2014). The problems with science communication evaluation. Journal of Science Communication, 1, C04.
Jones, R. A. (2014). Reflecting on public engagement and science policy. Public Understanding of Science, 23(1), 27-31. doi: 10.1177/0963662513482614
Klein, J. T. (2008). Evaluation of interdisciplinary and transdisciplinary research: a literature review.
American Journal of Preventive Medicine, 35(2), S116-S123. doi: 10.1016/j.amepre.2008.05.010
Knight, A. T., Cowling, R. M., Rouget, M., Balmford, A., Lombard, A. T., & Campbell, B. M. (2008). Knowing but not doing: Selecting priority conservation areas and the research–
implementation gap. Conservation Biology, 22(3), 610-617. doi: 10.1111/j.1523-