Has the gap been closed? Evaluation in the SPG discourse

There is no consolidated ‘science-policy gap evaluation discourse’ with well recognised frameworks. In a cross-disciplinary review of published work on the impact of research on policy Boaz et al. (2009, p. 266) noted: “the majority of studies reflect on the relationship between research and policy rather than the impact of research on policy”. A lack of well

37

recognised frameworks to evaluate efforts to bridge the SPG may reflect the fact that discussions are situated across a range of science disciplines and policy sectors, and because the approaches to bridging the gap are varied and may require different approaches to evaluation (Fazey et al., 2014). Different research fields define or value impacts differently for example, health research impact is focused on clinical practice, while other sciences focus on science policy, technical innovation, or economic development (Boaz et al., 2009). Evaluation of policy impact is therefore spilling over into evaluating implementation impacts (see Figure 1).

One particular area of study within the broad discourse around the science-policy gap related to policy impacts concerns the evaluation of various forms of “intervention”. Interventions include the implementation of science-informed policy or plans (Fig. 1), for example in public health and as part of international development programs. While my focus is on the role of science communication in how science informs policy, rather than the implementation of that policy, it is worth noting the wealth of research in this area. Evaluation in these contexts is commonly centred on a “theory of change” (Stein & Valters, 2012). Evaluation typically involves a realist approach whereby desirable outcomes are articulated and then possible pathways or mechanisms by which these outcomes can be achieved are hypothesised (Chapman et al., 2016; Haynes et al., 2017). Indicators are then case specific to test assumptions about causal pathways.

In contrast to the realist approach to evaluating interventions, evaluation of transdisciplinary research incorporates a more relational emergent perspective where value emerges and is defined through the participatory process. Transdisciplinary research explicitly involves multiple science disciplines and stakeholders, with different knowledge bases and values including decision-makers. The research process itself can therefore be viewed as a mechanism for bridging the science-policy gap through involving decision- makers in processes such as knowledge co-production. While a number of evaluative frameworks are being developed for transdisciplinary research, a key problem is capturing the impact on highly complex, long-term goals such as sustainability (Polk, 2014) and devising evaluative criteria for an approach that is necessarily flexible and evolving (Carew & Wickson, 2010). Current evaluation principles have a strong emphasis on process, for example how projects are managed to encourage communication, cognitive processing and reflexivity and epistemological considerations such as how well different knowledges are integrated and their value determined (Klein, 2008). Societal and scientific impacts are also

38

valued. For example, Walter et al. (2007) identified eight societal impacts that could be evaluated, half of which were related to the transdisciplinary process (building of networks and communities based on trust and understanding) with the others related to the knowledge produced and how this was spoken about after the project.

Knowledge exchange (KE) is another mechanism proposed for bridging the science-policy gap where evaluation has been considered, although there is recognition that more conceptualization and methodological development is required (Fazey et al., 2013). A cross- disciplinary review of published KE evaluations identified four broad categories of outcomes that had been evaluated. These were outcomes manifest in: individuals (changes in understanding); institutions (changes in practice or policy); impacts arising from implementation of policy/changed practices and; a diversity of process oriented outcomes including communication (Fazey et al., 2014). The value of KE in developing social networks among scientists and decision-makers and participant satisfaction have also been recommended (Cvitanovic et al., 2015).

Finally, another discourse that does not fit neatly into the SPG but includes how research may influence policy and how this can be evaluated is that concerned with “research impact”. This discourse is driven in part by funders of research (Morgan, 2014) with research impact defined in the UK as “an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia’ (HEFCE, 2017). The activities required to achieve research impact have been recognised in the UK and recently in Australia. In Australia this “research engagement” is defined in the same terms as knowledge exchange and is to be evaluated from 2018 by a suite of quantitative financial indicators (e.g. cash support received for research and research commercialization) and a supporting narrative (Commonwealth of Australia, 2017b). In the UK, public engagement by researchers (including with decision-makers) is seen as evidence of research impact, and is supported by various institutional structures such as the National Coordinating Centre for Public Engagement. However, one study indicated few researchers systematically evaluate this engagement (Grand, Davies, Holliman, & Adams, 2015). These observations indicate the value of engagement with people like decision-makers in the research process is beginning to be recognised by research funders but is still framed as a means to an end (generation of revenue and other impact).

Perhaps unsurprisingly, evaluation in the SPG focuses on science push and is strongly linked with the research agenda rather than decision-maker needs and perspectives. Greater

39

consideration of policy-pull activities may well yield different criteria as evidenced by the Fisher et al. (2014) review which found decision-makers valued interactions with scientists as either assisting in decision-making (consensus, confirmation and boundary setting) or information gathering (iterative scoping, opinion poll, boundary seeking). In all instances when it comes to evaluation there is a blurring of the distinction between science informing policy and the impacts arising from the implementation of those policies. This is even when omitting the evaluation literature focused specifically on implementation such as in international development. Transdisciplinary research places greatest emphasis on processes such as how scientists and decision-makers share and create knowledge and make meaning in a fair, equitable way while valued outcomes relate to new knowledge and constructive relationships.