Evaluation of direct outcomes

I define ‘direct outcomes’ as changes that can arise directly from engagement, as opposed to impacts that rely on processes in addition to engagement with science, such as subsequent

49

decision-making processes. In other words, if decisions are made in a process separate to engagement with science then the nature of the decisions doesn’t necessarily reflect upon the effectiveness of engagement. This focus on direct outcomes may mean that outcomes of engagement are evaluated primarily at the individual and institutional level rather than impacts at the societal level such as improved public health or species conservation. I have identified two broad types of direct outcomes from engagement that might be evaluated (Table 4). The first I have termed “benefits” which may manifest in individuals and broadly reflect the potential outcomes of communication—changes in knowledge, attitude and behaviour—as well as benefits to the organisations where decision-makers are situated. Benefits may also be in the form of relational outcomes such as when the process of engagement results in building trust between people and institutions, or forming new partnerships or alliances (Mitton et al., 2007; Roux et al., 2006). Such outcomes may be thought of collectively as increasing the capacity of decision-makers and may have benefits beyond the scope of the target decision-making process.

The aims of an engagement processes may be to produce outputs such as recommendations, new knowledge or decisions (Table 4). These outputs may simply be evaluated according to how well they achieve the pre-defined goals of an engagement process although, it is also important that evaluation of decision-maker engagement be designed to capture unintended outcomes in all these categories (Stern et al., 2012; Stilgoe et al., 2014). Beyond the fact of creating a quality output however, there is also the consideration of acceptance of the output which is a key criterion for evaluating engagement by the general public (Rowe et al., 2004). There it is assumed that participants will accept decisions coming out of a legitimate process i.e. one that is representative, independent, transparent, etc. It is unclear whether or not acceptance of decisions or other outputs will be important in engagement involving decision-makers. Broader public acceptance and legitimacy of decisions could well be important outcomes for institutions so it is included as a measure for now (Table 4)

2.5 Conclusion

The science–policy gap discourse illustrates the variety of mechanisms by which institutionally-based decision-makers can engage with science in a communicative sense. This discourse includes an extensive, primarily science-centric discussion about how science can inform policy, but principles or criteria for evaluating these processes and their outcomes are in their early stages. While recognizing the fundamental importance of processes that seek

50

to promote informed public decision making, I have taken an alternative conceptual approach embedded in communication theory and focused on the needs of the audience or publics - the decision-makers. The extensive communication literature around public engagement with science includes a number of evaluation frameworks and sets of evaluation principles. However, PES has traditionally framed decision-makers as sponsors or passive recipients of scientific advice rather than individuals actively making meaning of science through communication. Therefore neither of the relevant discourses specifically focus on how decision-makers make sense of science through communication. I have analysed the rationales for both public engagement with science and bridging the science-policy gap to propose a new benchmark for evaluating the multiple processes whereby decision-makers engage communicatively with science.

I have proposed that dialogue and deliberation on a range of relevant knowledge and experience, including science, as part of public decision-making is a good thing to do. This combines principles from both PES and the SPG but ultimately differs from both because the starting point is: What is valuable about engagement with science to decision-makers as distinct from the general public (PES) or scientists (SPG). From this perspective, communication processes that enable meaning making are valued instead of social empowerment, while science informing policy is a potential outcome but not one that ultimately defines success. I have posited three core values related to this normative rationale that can be used as broad evaluation criteria for decision-makers engagement with science. Firstly, engagement can be valued as a means of bringing together a variety of knowledge for consideration by decision-makers (an epistemological value). Secondly, communication processes involving deliberation and dialogue have inherent value as a means to develop understanding of the potential application of science in policy. Openly and fairly considering a range of knowledge can promote robust decision-making as well as legitimizing the process (communication and ethical values). And thirdly the direct outcomes of such communicative processes will lie in transformation (new knowledge, awareness and attitudes) for participants.

The discussion in this chapter has therefore laid out a set of values on which to base the evaluation of decision-maker engagement with science. The next two chapters will explore the usefulness and validity of these criteria through the examination of two case studies of decision-makers engaging with science. The case studies may also shed light on some of the questions that have arisen by applying communication theory to previously unconsidered

51

publics. These include questions related to the significance of institutional imperatives and processes on how the people within them engage with science, whether or not change can occur at an institutional level and what types of decision making context warrant such engagement process. Other questions relate to the nature of decision-makers themselves such as how does their empowerment affect their engagement with science? If we reframe bridging the science-policy gap as a process of engagement with science, do ‘good’ engagement processes legitimise decisions and institutions in the eyes of both participants and stakeholders outside the engagement process?

52

Chapter 3