Frameworks for evaluating decision maker engagement with science

(1)

Frameworks for evaluating decision-maker engagement with

science

A thesis submitted for the degree of

Doctor of Philosophy

of

The Australian National University

by

Ruth Ann O’Connor

Australian National Centre for the Public Awareness of Science

ANU College of Science

April, 2018

(2)

ii

Declaration

This thesis is original work. None of the work has been previously submitted for the purpose of obtaining a degree or diploma in any university or other tertiary education institution. To the best of my knowledge, this thesis does not contain material previously published by another person, except where due reference is made in the text.

Ruth O’Connor

(3)

iii

Acknowledgements

The opportunity to do post-graduate research is a great privilege. Part of that privilege has been both learning from, and enjoying the company of, a suite of supervisors and mentors. My thanks go to these people. Joan Leach and Fabien Medvecky who introduced me to science communication research. Lilly Lim-Camacho and the team from CSIRO for sharing the challenging and rewarding journey that is facilitating a national project like AdaptNRM. Jeanne Nel and the wonderful group in South Africa who made me welcome and shared the rich insights gained from years working at the science-policy interface in a domain close to my heart – freshwater conservation. Lorrae Van Kerkhoff and Wendy Russell for being engaged and thoughtful panel members when I made the move to the ANU.

This research was my first experience in conducting interviews. Listening to the stories and experiences of people working in the NRM sector was a highlight of my research process. A huge thanks to those in Australia and South Africa who gave of their limited time to be interviewed. These insights are the heart of the thesis.

This research was supported by an Australian Government Research Training Program Scholarship. I am also grateful for a CSIRO PhD top-up.

Academic and financial support is only one requirement for such a long endeavor. Thanks seem inadequate for the inspiration, emotional and material support given generously by he who is dearest to me, Simon.

(4)

iv

Abstract

In what ways are decision-makers participants in science communication? Do they actively engage with scientific concepts? If they do actively engage, what is valuable about those processes and their outcomes? In this thesis I explore institutionally-based decision-makers through a communication lens and reflect on how they compare with current definitions of lay publics. I focus primarily on the processes of communication and their value rather than processes of decision-making. This work marks a departure from the current framings of decision-makers in the public engagement with science discourse where they are considered sponsors or stakeholders outside public engagement processes, or in a deficit mode passively receiving expert advice. Likewise the Science-Policy Gap discourse, with its focus on the relationship between science and policy, recognises the importance of decision-maker engagement, but still views them as stakeholders who should use science to inform decisions.

I asked decision-makers for their perspectives on the value of science communication and what it looks like to them. They were public servants from the natural resource management (NRM) sector in Australia and South Africa who had participated in two specific science engagement programs. Their science communication landscape included a wide variety of modes reflecting their institutional status. They initiate engagement with science and are also invited to participate and engage in activities ranging from seminars to knowledge co-production processes. Their institutional setting also imposes constraints, however, determining what issues are salient and providing limited time to weigh up different sources of science and other knowledge. Within this landscape, NRM decision-makers demonstrated the key traits of publics. They are highly educated but still lay people in complex decision contexts, forming and re-forming around relevant issues. They are also emergent or unknown with high degrees of individual staff turnover, private consultants filling gaps in capacity, and unintended audiences engaging and using science-based outputs. I argue that framing decision-makers as publics for science, therefore is not only a more accurate representation of reality, it gives us the tools to both recognise and analyse the communicative nature of engagement activities like knowledge co-production. It also opens the scope of what might be considered valuable about these processes and what it takes for science to “inform” decisions.

(5)

v

(6)

vi

Figures and Tables

FIGURE 1:DIAGRAMMATIC REPRESENTATION OF HOW SCIENCE INFORMS POLICY BASED ON KEY TERMS USED IN THE SCIENCE-POLICY GAP AND RELATED DISCOURSES. ... 34

FIGURE 2:BREAKDOWN OF THE FUNDING STRUCTURE FOR THE CLEAN ENERGY FUTURE PLAN.ELEMENT 2 INCLUDED THE ADAPTNRM PROJECT. ... 53

FIGURE 3:MAP OF REGIONAL NRM ORGANISATIONS SHOWING CLUSTERS USED FOR THE NATURAL RESOURCE MANAGEMENT PLANNING FOR CLIMATE CHANGE PROGRAM. ... 57

FIGURE 4:VALUES ASSOCIATED WITH DIALOGUE, DELIBERATION AND ASSOCIATED OUTCOMES AT INDIVIDUAL (INSIDE THE DOTTED SQUARE) AND INSTITUTIONAL LEVELS. ... 70

FIGURE 5:AN EXAMPLE OF A CASE STUDY APPEARING IN THE NRMADAPTATION CHECKLIST (RISSIK ET AL.,2014) ... 75

FIGURE 6:THE MECHANISMS BY WHICH DECISION-MAKERS ACCESS AND ENGAGE WITH SCIENCE BASED ON CURRENT LITERATURE AND FINDINGS FROM THE ADAPTNRM AND NFEPA CASE STUDIES. ... 128

FIGURE 7:SOME OF THE DECISION-MAKING PROCESSES INVOLVED IN THE FUNDING AND IMPLEMENTATION OF THE ADAPTNRM PROJECT [SOURCES:(COMMONWEALTH OF AUSTRALIA,2012A,2012B); INTERVIEW TRANSCRIPTS] ... 135

FIGURE 8:SCHEMATIC REPRESENTATION OF POTENTIAL KNOWLEDGE FLOWS IN NRM DECISION-MAKING WITH YELLOW BOXES REPRESENTING A DECISION LEGITIMIZATION OUTCOME. ... 146

TABLE 1:SUMMARY OF KEY FEATURES OF PUBLIC ENGAGEMENT IN FOUR DIFFERENT CONTEXTS WITH FEATURES UNIQUE TO ENGAGEMENT HIGHLIGHTED.OTHER MODELS OF SCIENCE COMMUNICATION THAT SHARE FEATURES ARE INDICATED: DEFICIT (D), DIALOGUE (DL) AND PUBLIC PARTICIPATION (PP). ... 11

TABLE 2:A TYPOLOGY OF EVALUATIONS FROM GUBA AND LINCOLN (1989) SHOWING THE EVOLUTION OF THE PRACTICE AND INCLUDING THE HYPOTHETICAL EXAMPLE OF EVALUATING A NEW EXHIBIT AT A MUSEUM. ... 22

TABLE 3:SUMMARY OF EXISTING EVALUATION FRAMEWORKS FOR PUBLIC PARTICIPATION SHOWING THE SOURCE, BASIS AND CONTEXT OF EVALUATION CRITERIA. ... 29

TABLE 4:DRAFT SET OF VALUES, EVALUATION CRITERIA, AND MEASURES FOR EVALUATING THE EFFECTIVENESS OF DECISION-MAKER ENGAGEMENT WITH SCIENCE. ... 46

TABLE 5:MODE OF INTERVIEW WITH ADAPTNRM DECISION-MAKERS BY JURISDICTION. ... 58

TABLE 6:BREAKDOWN OF ADAPTNRM INTERVIEWEES BY INSTITUTIONAL ROLE. ... 58

TABLE 7:SUMMARY OF HOW INTERVIEWEES FROM REGIONAL NRM GROUPS ENGAGE WITH SCIENCE OR IN SCIENCE AS PART OF THEIR INSTITUTIONAL ROLE. ... 61

TABLE 8:MECHANISMS BY WHICH INTERVIEWEES ENGAGED WITH SCIENCE AS PART OF ADAPTNRM. ... 62

TABLE 9:REVISED VALUES AND EVALUATION CRITERIA FOR EVALUATING DECISION-MAKERS’ ENGAGEMENT WITH SCIENCE BASED ON FINDINGS FROM THE ADAPTNRM CASE.TEXT SHADED GREEN REPRESENTS CHANGES TO THE DRAFT FRAMEWORK (TABLE 4). 81 TABLE 10:BREAKDOWN OF NFEPA INTERVIEWEES BY JURISDICTION, SECTOR AND MODE OF INTERVIEW. ... 96

TABLE 11:SOURCES OF SCIENCE AND SCIENCE ENGAGEMENT MENTIONED BY NFEPA INTERVIEWEES. ... 100

TABLE 12:MECHANISMS BY WHICH INTERVIEWEES ENGAGED WITH SCIENCE THROUGH NFEPA- ADAPTED FROM NEL ET AL.(2016). ... 102

TABLE 13:EXAMPLES OF WHERE NFEPA PRODUCTS HAVE BEEN USED THAT WERE MENTIONED IN INTERVIEWS. ... 114

TABLE 14:VALUES AND EVALUATION CRITERIA FOR EVALUATING DECISION-MAKERS’ ENGAGEMENT WITH SCIENCE (BASED ON TABLE 9) WITH SUMMARY OF FINDINGS FROM THE NFEPA CASE. ... 119

(8)

1

Chapter 1 Introduction and definition of key concepts

1.1 Introduction

Before starting my PhD, I worked as a knowledge broker in several collaborative research programs. These positions involved what I regarded as some of the traditional aspects of science communication such as website development and writing documents that summarised scientific findings into plain English. However, what I found equally important was work that didn’t fit neatly into the institutional view of science communication. This work involved building relationships and trust between scientists, decision-makers and community groups; bringing people together to harness science for problem solving; and developing new ideas by capturing different people’s experience and knowledge. As one example, my roles involved facilitating engagement between scientists and Indigenous Australians. Many of the scientists were confronted by the initial skepticism of Indigenous people toward them which were often based on previous experiences where insufficient effort had been made to understand their perspectives and knowledge. However, providing non-threatening and purposeful environments in which people could interact lead to outcomes such as a better mutual understanding of each other’s knowledge and values. Several Indigenous participants were inspired to enroll in related degrees that could empower them to meet their aspirations. Several relationships formed that lead to collaboration on grant applications driven by Indigenous priorities where problems were jointly framed. I only discovered these outcomes and impacts anecdotally from the individuals involved.

(9)

2

described were unplanned and certainly not universal but were, in my opinion, highly significant indicators of success and how we should value the often labour and time-intensive activities that enable meaningful dialogue.

I was left with the question: can we do better? I have no idea how many other similar successes or indeed failures arose from the programs. Worse still, I cannot say what might have contributed to those successes or failures. When I returned to student life and was afforded the luxury of reading about the theory and practice of science communication, I realised that my experience of inadequate evaluation was far from unique. While communication is recognised as an integral part of the scientific process (Leach, 2009) and science communication as a discipline has become institutionalised in Australia and around the world (Gascoigne & Metcalfe, 2017; Trench, 2014), science communication broadly and public engagement with science specifically, are often not rigorously evaluated (Jensen, 2014; McKenzie, 2014; Metcalfe, Alford, & Shore, 2012). So we can do better in how we value and evaluate science communication and this research is my attempt to move in that direction.

The focus of this thesis is on how to evaluate a particular form of science communication (public engagement with science), involving particular publics (institutionally-based decision-makers). This is a logical progression from my practical experience but both “public engagement with science” and “publics” are complex concepts interpreted in multiple ways. The first chapter of the thesis lays the groundwork for my arguments by defining such key terms and reviewing the science communication literature on different models of science communication and their aims. While there is no agreed definition of what science communication is, it has been recognised as performing two functions. It can be about science for those who don’t participate in it and for science to influence people’s attitudes toward it (Leach, 2013). In other words, science communication functions to include or extend the reach of science to the non-scientist or non-expert in a particular field.

(10)

3

In this strategy, ‘science’ or ‘the sciences’ refers to the following:

 the natural and physical sciences, such as biology, physics, chemistry and geology

 the applied sciences, such as engineering, medicine and technology

 newly emerging and interdisciplinary fields, such as environmental science, nanotechnology and phenomics

 mathematics, a field of study in its own right, as well as an essential tool of the sciences

 the social sciences and humanities, critical to the interface between science and society.

The recent Australian National Science Statement, in answering the question “what is meant by science?”, similarly downplays the scientific process and in listing disciplines ignores the social sciences (Commonwealth of Australia, 2017a, p. 4):

Natural, physical and life sciences, including medical and health sciences, mathematics, engineering and technology-related disciplines.

This includes the full spectrum from basic to applied scientific research in both the public and private sectors, and the infrastructure, skills, institutions, knowledge and policies that make it possible.

The object of science communication research has been described as “how society talks about science” (Bucchi & Trench, 2014, p. 10). A key consideration of science communication, therefore, is who takes part in it. The people who are involved in science communication are publics and they will also be discussed in detail later in this chapter. What surprised me as I delved into the science communication literature, however, was that decision-makers weren’t viewed as audiences or publics for science communication. Rather, decision-makers were framed as outside engagement. They were stakeholders or sponsors of such processes (e.g. Lezaun & Soneryd, 2007; Mohr, Raman, & Gibbs, 2013) or those who seemingly passively “took advice” from scientific experts and committees (Jasanoff, 1990; Stilgoe, Irwin, & Jones, 2006). The research focus was on the legitimacy of scientific advice to decision-makers and who could give it (Spruijt et al., 2014) rather than how they might process scientific information. As Helga Nowotny (2014) noted, the processes by which decision-makers engage with and make meaning of science remain in the “political black box”.

(11)

4

gap. This discourse is explored in Chapter 2 for insights about decision-makers and how they engage with science. In that chapter I also bring together the public engagement with science and science-policy gap discourses to propose a draft framework for evaluating decision-maker engagement with science. That framework is then tested using two case studies of decision-maker engagement with science in Chapters 3 and 4.

I decided to pursue case studies in one particular policy sector—natural resource management or NRM. This decision was guided partly by my personal interest and background knowledge as well as a recognition that different policy sectors may have unique issues or variables which could obscure my focus on communication processes. NRM also provides potentially fertile ground for exploring interactive communication processes because of its management imperative. Scholars recognise that while science can be an important source of knowledge to inform NRM decisions, science is also culturally and socially situated and its meaning and implications need to be negotiated among those with a stake in decisions (Curtis et al., 2014; McCool & Guthrie, 2001).

A fundamental component of my thesis is the importance of considering institutionally-based decision-makers as publics for science. This is because they do engage with science in a communicative sense (at least in some policy arenas) and the nature of these processes is important for understanding how and why science is or isn’t applied through policy instruments to impact the public sphere. While such decision-makers are not located in the public sphere I will explore how they meet the other criteria for publics such as being lay people or non-experts. I will also argue that if we recognise the public engagement with science model of science communication as a process of non-experts making sense of science through communication rather than focusing on particular outcomes (decisions or democratizing science), then decision-makers stand out as publics of great importance and interest. I will also explore the distinguishing characteristics of decision-makers as publics, what they value in engagement processes and how we can evaluate those processes and their outcomes. In this way I hope to provide insight into what we should be valuing about the communicative processes that are an integral part of collaborative research programs like the ones I have participated in as well as a variety of other activities at the interface between science and policy.

(12)

5

1.2 What is public engagement with science?

In the context of this thesis, public engagement with science (PES) is one form of science communication. While the term ‘public engagement with science’ has grown in popularity in the last two decades, it is still poorly defined and used interchangeably with other terms such as public participation (Delgado, Lein Kjølberg, & Wickson, 2011; Vincent, 2014). In this section, I review current models of science communication and develop a definition for PES based on its unique qualities. Models of science communication have been variously defined based on: different paradigms for the public understanding of science; the mode of communication (particularly is it one-way, two-way or multi-directional); and what types of knowledge (in addition to science) are being communicated (Bauer, Allum, & Miller, 2007; Bucchi, 2008; Lewenstein, 2003; Ziman, 1992). A discussion of these models follows to tease out where engagement sits within the discipline of science communication. I will explore the concept of ‘publics’ later in this chapter but for now ‘the public’ will be used to refer to non-experts in science.

1.2.1 Models of science communication

Models are representations of real-world phenomena and can be any simplification, substitute or stand-in for the object of study. “Communication models” are not used in a predictive sense, rather as simple descriptions of analytic categories evoked by their use (Leach, Yates, & Scanlon, 2009). I will explore the key elements of science communication using the framework developed by Bucchi (2008) and also Trench (2008) who identify three models. While science communication events won’t always fit neatly into one or the other of these categories they are useful for exploring the aims and assumptions of science communication (Irwin, 2008; Kurath & Gisler, 2009).

1. Deficit

(13)

6

model is that the public are passive, ignorant and/or hostile to science and this can be counteracted by science communication (Bucchi, 2008).

The deficit model assumes a pedagogical role for science communication. Knowledge from the science community is fixed and certain and flows from experts to non-experts (Logan, 2001). The public is problematised while the capacity of science and scientists to identify and solve societies problems remains unquestioned (Irwin, 2014; Wynne, 1991). These communication processes are not only one-way, the public is disempowered and without valid knowledge to contribute. They have no say in the process, only the power to ignore it. The contextual model of Lewenstein (2003) is a variation on the deficit model that acknowledges the public as more than the one-dimensional receiver of knowledge. Science is still the dominant form of knowledge but it is acknowledged that the context of individuals in the public such as their cultural background or age will affect their response to science communication.

Another assumption of the deficit approach is that ignorance leads to hostility. In other words, greater understanding of science will lead to a greater regard for science, or as Bauer et al. (2007, p. 83) frame it, “the more you know, the more you love it”. However, several decades of research suggest many assumptions of the deficit model are flawed (Nisbet & Scheufele, 2009). Meta analyses of the link between science knowledge and attitudes to science across Europe and the US have shown science literacy is a minor contributor in how the public form opinions about controversial areas of science (Allum, Sturgis, Tabourazi, & Brunton-Smith, 2008). Studies have also shown that the more ‘scientifically literate’ a public is, the less likely they are to think that ‘science can solve our problems’ (Bauer, 2008).

2. Dialogue

(14)

7

representatives) and the public. This difference in the mode of communication provides a clear point of distinction between the dialogue and deficit models.

Irwin (2008) posited that the shift from deficit to dialogue is more than a shift in modes of communication, it has deeper intellectual and political implications. In this view, the dialogue model is a mechanism of deliberative democracy and part of a normative approach to public policy-making, particularly in circumstances of social and technical uncertainty. Instead of having a knowledge deficit, the public can contribute to the framing of science-informed issues and their resolution. In the dialogue model the public are consulted regarding the application of science (Trench, 2008) and the implications of research (Bucchi, 2008). Both imply that the knowledge and values of the public are sought out and potentially acted upon in the conduct of science and development of technologies. In this sense dialogue is about “opening-up” a discussion about science (Stirling, 2008) rather than assuming it has the answers.

Not all public dialogue processes match the criteria described above, however. Equitable public dialogue requires work or it can inhibit the expression of lay perspectives (Davies, 2013) or close the decision-making process to citizens by preventing participation of people with strong views on the issue at hand (Stilgoe, Lock, & Wilsdon, 2014). An instrumental approach to dialogue where there is a pre-determined endpoint is counter to the spirit of dialogue and risks being viewed as manipulation. In this vein, open and inclusive dialogue does not necessarily lead to participants reaching consensus on an issue. As noted by Stoker & Tusinski (2006, p. 157), communication (including dialogue) is both a “bridge and chasm,” bringing people closer to agreement and exposing the disagreement lying in between. In dialogue, mutual understanding may be a more valid goal than convincing or consensus (Stoker & Tusinski, 2006).

(15)

8

3. Public participation

Public participation is in its broadest sense a range of activities that seek citizen involvement in decision making processes (Arnstein, 1969). As with dialogue, the content and context of public participation may have nothing to do with science. The term is also used in a more specific sense to describe a model of science communication. As defined by Bucchi (2008, p. 73):

“Public participation in science may be defined broadly as ‘the diversified set of situations and activities, more or less spontaneous, organised and structured, whereby non-experts become involved, and provide their own input to, agenda setting, decision-making, policy forming and knowledge production processes regarding science”

Public participation in science is therefore specifically concerned with “democratizing science” (Lewenstein, 2003) which is similar to the broader concept of public participation which, while taking myriad forms is fundamentally about the empowerment of those excluded from political and economic processes (Arnstein, 1969). Participation processes necessarily include dialogue so the distinction between the models is not clear-cut. One important point of difference appears to be the greater role of the public relative to scientists in public participation. Another is that there can be mechanisms in place in organised public participation activities for the views of the public to feed back into the issue at hand— decision-making, policy-making or the research agenda itself (Bucchi, 2008; Einsiedel, 2008). It can also be seen as an extension of dialogue with all participants being active players. Information flow is multi-directional between the public, scientists and science institutions, knowledge can be co-produced and processes can be open-ended (Bucchi, 2008).

(16)

9

In this view, different forms of expertise and understanding represent an important resource for change rather than an impediment or burden (Irwin, 2008).

The public participation model, therefore, takes the idea of deliberative democracy a step further than dialogue. The mode of communication is multi-directional between and among participants (which may have implications for learning) and there are mechanisms for the views and issues raised to be used in some way. The public have a greater level of agency than in the dialogue model with the potential to influence decision-making including decisions related to science policy. Whether the public are setting the agenda and guiding the nature of participation, however is unclear.

These three models of science communication highlight key aspects of science communication that now need to be considered in relation to defining engagement. The key aspects being: the mode of communication; the role of the public and their empowerment; and how sources of knowledge other than science are regarded and incorporated in the process. Defining these key elements of engagement is crucial for understanding what engagement aims to achieve and what value it may have for decision-makers.

1.2.2 Different contexts for public engagement with science (PES)

The philosophical foundations and methods for public engagement with science issues have been developed and debated by a variety of authors for at least the last 15 years (Rogers-Hayden, Mohr, & Pidgeon, 2007). However, the shift from ‘public participation’ to ‘public engagement’ remains poorly articulated and the two terms are often used interchangeably (Delgado et al., 2011; Vincent, 2014). I will now review four different contexts for public engagement that have received substantial attention in the literature to tease out how engagement may differ from deficit, dialogue and participation. These types of activity may be described by some as examples of public participation because of the interchangeable use of the terms to date.

Citizen science

(17)

10

in determining what research is done and how (Riesch, Potter, & Davies, 2013). The defining feature of this form of engagement (summarised in Table 1) is that the co-creation of data and/or knowledge is at least one core aim.

Museums

In comparison with other engagement contexts, museums (and science centres) are unique in that the provision of a physical place for engagement is fundamental (Rodari, 2010 and Table 1). Engagement with science in museums also has a strong emphasis on learning which is not a defining feature of engagement in most other contexts, although it may well be an outcome (Rennie, 2001). Indeed, learning as an aim of science engagement is viewed negatively by some in contexts such as public policy as it harks back to the pedagogical emphasis of the deficit approach (Rowe, Marsh, & Frewer, 2004). Another defining feature of engagement at museums is that it does not necessarily seek to inform the direction of scientific investigations or science policy (McCallie et al., 2009).

Science policy and governance

Public engagement with science in the development of science policy has been the subject of a significant body of scholarly work. PES in the context of science policy occurs via a number of specific approaches such as ‘Participatory Technology Assessment’ and ‘Upstream Engagement’. These approaches involve dialogue and deliberation among the public, scientists and policy-makers at the early stages of research and development into potentially disruptive or controversial technologies such as nanotechnology (Russell, 2013). The rationale of the approaches is that public debate can alter the future trajectory of such technological developments in ways supported by society (Burgess & Chilvers, 2006; Rogers-Hayden et al., 2007). In contrast, Responsible Research and Innovation (RRI) is an approach to science governance promoted in the European Union (European Union, 2012). Public engagement is part of this approach to “collective stewardship of science and innovation” but RRI also encompasses codes of conduct and responsible behaviour within science institutions (Stilgoe, Owen, & Macnaghten, 2013, p. 1570).

(18)

11

[image:18.595.65.549.412.689.2]

value of public engagement but one that may not always be realised in formalized settings controlled by those developing new technologies (Bogner, 2012). Implicit in these PES processes is the aim of building public trust in the fairly narrow range of science and associated technology that is the focus of engagement (such as nanotechnology and biotechnology). The nature of the public involved in engagement is often a general public assumed to represent a cross section of views—as seen with the GM nation program in the UK (Mohr et al., 2013). The key features differentiating public engagement with science policy versus other forms of engagement are the empowerment of the public to set the agenda for engagement and provide input to the research agenda as well as the opportunity for multi-directional learning (Table 1). Multi-multi-directional learning generally refers to scientists and the public learning from each other, promoting greater reflexivity in science (McCallie et al., 2009; Rogers-Hayden et al., 2007). The importance of learning by all participants has also been noted which may include facilitators and policy-makers (Chilvers, 2008).

Table 1: Summary of key features of public engagement in four different contexts with features unique to engagement highlighted. Other models of science communication that share features are indicated: deficit (D), dialogue (DL) and public participation (PP).

Engagement context Feature of engagement Citizen

Science

Museums Science Policy

Social Licence to

Operate

Process

Public can set the agenda (PP) X

Public can state their views (DL, PP) X X

Feedback mechanisms to incorporate public input (PP)

X X

Venue provided for engagement X

Public can influence the process of engagement ? X

_*

Outcomes

Science made accessible (D, DL, PP) ? X X X

Public can experience informal learning (D, DL) OR X X

Multi-directional learning (PP) X ?

Knowledge co-production a key goal X

Public input to research/development agenda a key goal (some PP)

X X

(19)

12

Social licence to operate (SLO)

A social licence to operate reflects the idea that society is able to grant or withhold support for an organization or industry to work in the local area (Rooney, Leach, & Ashworth, 2014). Engagement is considered to be central to developing and implementing an SLO. One of the key purposes of engaging the public is to build trust and relationships between those proposing an activity and those who may be affected by it (Hall, 2014; Williams & Walton, 2013). Ultimately, the aim of the proponent is to be able to continue their activity, which limits how far dissenting voices within the public can be taken into account. Engagement can consist of convincing the public that their concerns are unfounded which could be viewed as ‘selling the science’ but may also take the form of negotiation—changing some practices or providing compensation in some form.

In summary, public engagement with science in different contexts has overlapping characteristics with the deficit, dialogue and public participation models of science communication. Depending on the specific nature of an activity and different authors’ perspectives, dialogue seems an important component of both participation and engagement. Likewise, making science accessible (which may be seen as a fundamental component of the deficit model) is also intrinsic to different forms of public engagement (Table 1) as the public needs to have some awareness or knowledge of science to engage with it or about it. Learning processes are also common to engagement and the three other models. Defining a particular communication exercise as being one model or another may therefore be somewhat arbitrary as it may have elements of multiple models depending on how it is implemented. Key elements for consideration in both participation and engagement are the empowerment of publics to influence both the engagement process and the issue of concern (Table 1). The exercise also highlights some of the key features of processes and outcomes of public engagement, which may be of value and form part of evaluation frameworks.

1.2.3 A definition of public engagement with science

(20)

13

intensely; engross; occupy” (Collins & Anderson, 2006). The definition of participation overlaps with that of engagement also describing a level of “involvement” although the act of participating may vary from being a passive recipient of information to actively participating in decision-making processes (Arnstein, 1969; Rowe & Frewer, 2005). In recognition of the important conceptual differences in different forms of participation, Rowe and Frewer (2005) split them into public communication, consultation, and participation based on the direction of information flow and collectively called these processes public engagement. This solution fits awkwardly with the science communication models described here i.e. saying deficit, dialogue and participation collectively are engagement.

While dictionary definitions don’t neatly separate engagement and participation they do point to an important conceptual difference around passive involvement versus active interest, intense engrossment or occupation. I propose therefore that a starting point for defining public engagement with science is that participants value science content, processes or outcomes as this provides the impetus for active involvement. Engaging with science need not be equated with an acceptance of the authority of science, however. One could be engaged with science if one is forcibly rejecting it, such as some activists, campaigners or counterpublics (Grunig, 1997; Hess, 2011; Mohr et al., 2013). The active participation of the public is also reflected in a shift in terminology from ‘public engagement in science’ to ‘public engagement with science’. While this distinction in terminology is not discussed broadly in the literature, ‘with’ has become commonly used (e.g. Irwin, 2014; McCallie et al., 2009; Nowotny, 2014; Wynne, 2006). The use of ‘public engagement with science’ implies public engagement is not only with science but with policy and with organizations, as well as issues (Rogers-Hayden et al., 2007; Stilgoe, 2007).

(21)

14

on. The focus of this thesis will be on the forms of engagement undertaken by decision-makers in their institutional roles which may involve them actively seeking knowledge or scientists/third parties inviting them to engage with science.

Different science communication models capture different scenarios for how the knowledge of the non-scientist is viewed and valued (Chilvers, 2008; Lewenstein, 2003). In the deficit model, non-scientists are passive recipients rather than contributors of knowledge. In public engagement, the knowledge and expertise of the public is valued in some way. In public policy this may be to frame problems and promote rational decision-making. This change in role of the public from passive to active participants is associated with different modes of communication and these may also be distinguishing features of public engagement. Dialogue is certainly one of these modes in addition to processes of reflection and deliberation (Chilvers, 2008; Rathouse & Devine-Wright, 2010). Reflexivity is seen as a necessary component of responsibility in decision-making i.e. “holding a mirror up to one’s own activities, commitments and assumptions, being aware of the limits of knowledge and being mindful that a particular framing of an issue may not be universally held” (Stilgoe et al., 2013, p. 1571). Reflexivity can be achieved through deliberative processes that help individuals interactively explore their ideas and preferences (Einsiedel, 2008).

(22)

15

1.3 Who are the publics?

One of the key considerations of all forms of communication are the people or publics being addressed. The concept of ‘the public’ or ‘publics’ has sparked intellectual discussion and social concerns across a range of fields including public relations, political science and psychology (Vasquez & Taylor, 2001). The idea of a single or mass public emerged in public relations in the 1930s and is grounded in the democratic view of a sum of individuals who can respond to civic matters (Grunig, 1997; Vasquez & Taylor, 2001). In more recent times, the situational theory of publics (coming from public relations to explain who an organisation’s publics are at a given time) and other perspectives have focused on multiple publics that emerge and develop in relation to problems (Grunig, 1997; Vasquez & Taylor, 2001). The notion of multiple publics has now largely replaced the notion of a single public in science communication and public relations (Bucchi & Trench, 2014; Leitch & Neilson, 2001).

The literature discussing public engagement with science often fails to define the publics involved. In the progression from deficit approaches to engagement, the academic focus has often been on democratizing science governance. In this context there is an assumption of a single public who are assumed to be ‘general’ or ‘expert’, defined only by being non-scientists (Hess, 2011; Kurath & Gisler, 2009). Those in the field who have turned their attention to the question of publics, however, have recognised that in PES publics are not ﬁxed or known entities, they change according to context (time, place and issue), and are formed and reformed through the process of engagement with various perspectives (Chilvers & Kearnes, 2016; Jasanoff, 2014; Mohr et al., 2013). For any engagement activity, therefore, there is a unique public. Hence in this thesis the term ‘publics’ is preferred to ‘the public’.

(23)

16

network who protest, lobby or campaign to raise public awareness and put pressure on governments (Delgado et al., 2011; Mohr et al., 2013). Also known as activist publics (Grunig, 1997), people with a known interest in an issue are sometimes deliberately excluded from engagement activities targeting ‘disinterested’ citizens (Hess, 2011; Mohr et al., 2013).

Finally, it has been argued that even if publics are not explicitly defined, any activity seeking to engage ‘the public’ implicitly constructs the ‘public’ and ‘public interests’ as well as their role (Burgess, 2014). For example, citizen science projects such as those that use the internet to ‘crowd-source’ various scientific tasks, may not explicitly define its publics. Yet clearly individuals that don’t have access to the technology and the internet will be excluded from these engagement activities. Likewise, research into attendance at open invitation engagement activities shows participants to be disproportionately drawn from groups with higher socio-economic status, greater interest in the topic area, more strongly held views and high educational levels (Martin, Christidis & Pecl, 2016; Sturgis, 2014).

Publics in the context of this thesis, therefore, are all the people involved in a given science engagement activity—invited or uninvited, explicitly defined or not. This definition may then encompass the seemingly disinterested citizen (often conflated with a notion of the ‘general public’), particular target audiences (such as school children), activist groups, communications specialists and decision-makers.

1.3.1 Decision-makers as publics

As mentioned in the introduction, individuals who make public policy are not currently framed as publics in the science communication literature (Nowotny, 2014; Rogers-Hayden et al., 2007). The key reason for this appears to be that policy-makers don’t generally engage with science in the public sphere defined as the ensemble of public spaces available for debate between citizens (Habermas, Lennox, & Lennox, 1974). Also, as the previous discussion of science communication showed, the normative underpinnings of science communication models go beyond communication processes to extend to ideas of democracy through deliberation on issues informed by science. If science communication is about empowering citizens then those in public office cannot be publics.

(24)

17

their stake (Mohr et al., 2013, p. 9). This framing seems to be linked to the idea that the public are ‘general’ or disinterested (Einsiedel, 2008). Another framing of policy-makers is as sponsors of PES activities. For example, Gene Rowe and colleagues mention policy-makers as sponsors or funders of engagement activities with ‘the general public’ (Rowe & Frewer, 2004; Rowe & Frewer, 2005; Rowe et al., 2004) but how policy-makers view science or how it is communicated to them is not explored. When policy-makers sponsor engagement, the implication is that they are the arbiters of what knowledge or recommendations will inform decision-making. Yet how knowledge and information is communicated to them is conceptualised in a one-way or transmission mode of “advice” or recommendations (Jasanoff, 1990; Stilgoe et al., 2006).

In contrast, a suite of literature that can be grouped under the heading of ‘bridging the science-policy gap’ focuses specifically on policy-makers as potential end-users of science. The science-policy gap is a term used to describe “the set of known difficulties in the dialogue and relationship between science and policy which includes the use and effect of science in policy” (Sumby, 2012, p. 13). Here scientists and science organizations engage with policy-makers to promote science adoption in policy. However, policy-makers are still framed as stakeholders (with an implied stake of using science to inform policy) despite recognition of a range of activities where they engage in scientific research and knowledge co-production (e.g. Carew & Wickson, 2010; Polk, 2014).

(25)

18

engagement processes where the application and implications of science are pursued but the adoption or uptake of science isn’t assumed.

I will limit my consideration of policy-makers to those situated in the public sector defined by law, public authority and public finance while recognizing that by my definition policy-makers also exist in the private business sector (defined by private investment and purchasing power), and the community sector (Head, 2007). A simple definition of public policy is “what governments do, why they do it, and what difference it makes” (Dye, 1972, p. 2). A diverse range of people are involved in public policy-making including politicians, their advisors, public servants, think tanks, interest groups and academics (Maddison & Denniss, 2013). Therefore, if we think about who may engage with science in order to inform decisions that manifest in the public domain, we can envisage multiple publics of decision-makers with diverse institutional roles and drivers for engaging with science. For example, public sector decision-makers in the fields of natural resource management and health may include policy-makers, planners and people in management or practitioner roles such as those managing protected areas or medical practitioners (Cook, Carter, Fuller, & Hockings, 2012; Holmes, Scarrow, & Schellenberg, 2012) and our more familiar elected representatives.

(26)

19

decision-makers engage with science through communication, what is valuable about these processes and how we can evaluate them.

1.4 What is evaluation?

While considering how to evaluate decision-maker engagement with science, it is important to recognise that there is no single approach or absolute answer to any particular object of evaluation (Guba & Lincoln, 1989). Evaluation is about assigning value and for that reason results will be context dependent, for example, do we value learning or do we value consensus (Neresini & Pellegrini, 2008)? While the specifics of each evaluation exercise will be unique, a helpful way to think about evaluation that will be used in this thesis is “studies designed and conducted to assist some audience to assess an object’s merit and worth” (Stufflebeam, 2001, p. 11).

Gene Rowe and colleagues give four reasons why evaluation of public participation is important. These are: i) financial considerations involving proper use of public or institutional money); ii) practical considerations related to learning and improving practice; iii) ethical and moral reasons such as ensuring those involved are not deceived as to the impact of their contribution; and iv) for research such as improving understanding of human behavior (Rowe & Frewer, 2004; Rowe, Horlick-Jones, Walls, & Pidgeon, 2005). In the next chapter I will explore the evaluation literature related to the science-policy gap. However, a discourse more closely related to PES is the evaluation literature around public participation by the general public—often around issues of environmental management. I will now explore the key elements of evaluation in public participation and then hypothesise how it may or may not relate to PES.

1.4.1 Approaches to evaluation – process versus outcome

(27)

20

Research Councils UK, 2011). As noted previously, the participation process itself may also be regarded as a valuable end-point rather than a means to an end (Lawrence, 2006). As mentioned previously in the discussion of science communication models, public participation emerged from a normative foundation that values democratization of science through inclusive dialogue and deliberation. From this perspective, the process of participation is critical and process indicators dominate early evaluation frameworks (discussed in more detail in Chapter 2).

There is inconsistency in the science communication literature as to how the outcomes and impacts of engagement are differentiated. Outcome and impact arising from engagement can manifest in changes to the individual (e.g. changes in the behaviour, skills or attitudes of participants), societal or institutional change (e.g. in the political sphere) or material change such as in the physical environment. Outcomes can also be viewed as the substantive decisions, conclusions or recommendations coming out of the participation process (Beierle, 1998) although if these are not implemented in some way they may be considered outputs. Those concerned with research impact view outcomes as those changes manifest in participants or individuals while impacts are changes manifest at a social, institutional and physical level. For example, the definition of research impact used currently in both the UK (HEFCE, 2017) and Australia looks at the broader consequences of research beyond those directly involved in the endeavour: “an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia” (Watt, 2015, p. 65). I will use this distinction between outcome and impact for now as a potentially useful way to separate results that the design and implementation of an engagement process may directly influence (outcomes) and those also subject to complex social, political and environmental forces unrelated to engagement itself (impacts).

(28)

21

minimization of conflict (Beierle, 1998) while society is assumed to benefit from better decisions. Measuring and attributing some of these outcomes to participation is difficult. Participation processes may be easier to measure but there is little evidence that they can or should be used as a surrogate for measuring outcomes (Emery, Mulder, & Frewer, 2015; Hansen & Allansdottir, 2011). Instead, if one is interested in outcomes then they need to be evaluated directly. In part, this is because a range of extrinsic factors such as the history of the issue, existing relationships among stakeholders, or the institutional capacity of agencies may affect the outcomes of public participation over and above good process (Beierle, 1998; Chess & Purcell, 1999).

1.4.2 Issues associated with evaluation

The literature on evaluating public participation—with science or otherwise—often includes discussion of the difficulties or issues involved (Friedman, 2008; Rowe et al., 2005). Rosener (1981) noted four difficulties associated with evaluating public participation that have been frequently cited and will form the basis of the following discussion. At the heart of much of the discussion and disagreement about how to evaluate public participation, are differing opinions on what problem public participation is meant to solve. As already discussed, participation is often linked to democratic processes (Beierle, 1998; Rosener, 1981; Rowe et al., 2005) and so the appropriateness of criteria will depend in part on which perspective of democracy is adopted such as a managerial versus a popular perspective (Beierle, 1998). With decision-makers, the democratic imperative is likely of less importance than other publics however, the tension, between ‘instrumental’ and ‘transformative’ participation noted by Lawrence (2006) may be worth consideration. The former is participation in a task defined by others and the latter is a process that changes meaning, power or social organisation. Whether an engagement process is valuable as a means to a goal, or as an end in itself will have important implications for designing its evaluation.

(29)

22

alternative is to try to take the subjectivity out of ‘success’ by combining all values in some manner or adopting a normative definition informed by the academic literature (Burgess & Chilvers, 2006; Rowe et al., 2005).

[image:29.595.67.529.426.639.2]

Rosener’s third issue is that there is no agreed method for evaluating public participation. Evaluation theorists have charted the evolution of evaluation from 1st to 4th generation (Guba & Lincoln, 1989). A description of this evolution with an example is presented in Table 2. While some may argue that 1st generation measurement on its own does not constitute evaluation, an analysis of Australian PES activities indicates 1st and 2nd generation evaluation are the forms predominantly used (Metcalfe et al., 2012). In contrast, evaluation activities in museums and science centres seem to have progressed into 3rd and possibly 4th generation (Sanchez-Mora, 2014). In sum, the Guba and Lincoln (1989) schema highlights that when evaluating engagement we need to consider whether we merely measure and describe elements, make value judgments about process and outcome or move toward 4th generation by incorporating decision-maker and other participants’ values.

Table 2: A typology of evaluations from Guba and Lincoln (1989) showing the evolution of the practice and including the hypothetical example of evaluating a new exhibit at a museum.

Evaluation type What evaluation involves Example

1st generation evaluation

Measurement by an evaluator

The number of people who visit the exhibit, how long they spend at the exhibit

2nd generation Description by an evaluator A survey asking visitors what they learned and what they liked. Results are reported based on their numeric value e.g. 60% of visitors found the exhibit interesting

3rd generation Judgment by an evaluator on the merit or worth of the evaluand

Interviewing visitors to the exhibit and using the results to form a judgment about the exhibit e.g. the exhibit was successful as most visitors reported that they found the exhibit interesting and had learned about DNA.

4th generation Responsive constructivist approach - evaluation is built on the claims, concerns and issues of all

stakeholding audiences

The evaluator works with all stakeholders (museum staff, visitors, sponsors, etc) to define and determine what constitutes success and they determine the success of the exhibit.

(30)

23

used to explore meaningful patterns and themes, participant’s views and diverse impacts (Research Councils UK, 2011; Stufflebeam, 2001). Quantitative approaches and their associated data may be perceived as being more objective than qualitative approaches to evaluation (Lawrence, 2006; Porter, 1996) however, in some engagement contexts, the most significant outcomes may be the internal ones related to learning and values which are better captured through qualitative approaches (Lawrence, 2006). It may be that in PES, and indeed more broadly, rigid opposition between quantitative and qualitative research methods is not very useful (Neresini & Pellegrini, 2008).

Finally a number of issues in evaluation relate to measurement. Two of the key questions that can be asked about any instrument or process used in evaluation are is it valid and is it reliable (Rowe et al., 2005). Validity refers to whether or not the target concept is actually measured (Rowe & Frewer, 2004). Does the number of people attending an engagement event reflect the level of engagement in the topic, for example? Reliability on the other hand refers to whether a method is liable to give similar results on different occasions (Rowe et al., 2005). These measurement issues are particularly important when evaluating engagement or participation because we are assigning value to complex human beings interacting in complex ways for complex reasons. For example, it has been noted that each visitor to a museum has a unique experience and as a consequence outcomes for each person will likely be unique (Rennie, 2001). If we are to move beyond 1st generation evaluation of engagement we inevitably need to measure or assign value to some aspect of humanity such as behaviour, attitudes, learning or empowerment. Diversity in who we are and how we respond can make drawing inferences problematic.

Another issue for evaluating public participation is how to validly infer causality or attribute a particular outcome to a particular activity (Chess & Purcell, 1999; Gardner et al., 2009). In other words, regardless of the goals selected, and even with reliable and valid instruments, evaluating the outcome of any public participation or engagement process faces the challenge of showing an effect is due to the participation process rather than other variables, such as simultaneous events (Chess & Purcell, 1999).

(31)

24

2005). The process of collecting data or reporting can be difficult if the sponsor feels insecure about their process or the findings are negative.

Implications for evaluating PES

Public engagement with science has multiple meanings, one of which is as a model of science communication. Under my definition of PES three qualities emerge of relevance for evaluation. Firstly, the quality of the communication process itself such as how well participants can explore science and make meaning of it. Secondly, engagement requires the valuing of knowledge sources other than science so there are epistemological issues around which knowledge is considered. Related to this are ethical issues about who gets to decide what knowledge is considered and are they considered fairly and equally. Thirdly, while not a particular quality of PES per se, outcomes and impacts are valued by sponsors and participants. Evaluation should therefore consider the outcomes and impacts of PES at individual, institutional and societal levels. It is unclear at this point what engagement involving decision-makers may look like in relation to these principles, so this needs to be explored. Further, if we want evaluation to move from 1st to 4th generation we need to consider how decision-makers themselves value engagement with science.

1.5 Directions for the thesis

Given the gaps in our current understanding of engagement and what functions it performs for decision-makers in the public sector, my research will address the following questions:

1. How do decision-makers engage with science in a communicative sense? (Note: by examining cases of engagement, I will assume that at least some decision-makers do engage with science to some degree)

1.1What is the nature of science engagement by decision-makers?

1.2 How important is empowerment in the engagement process and does the degree of empowerment influence outcomes?

2. What drives or inhibits engagement with science?

2.1 What is the importance of personal interest in science versus institutional imperatives to use science?

2.2 Is valuing science linked to science engagement?

(32)

25

3. What is valuable about engagement processes and how can we evaluate them? 3.1 What are the key aspects of engagement that should be evaluated?

3.2 Is dialogue an important aspect of engagement, and if so, how?

(33)

26

Chapter 2 A draft framework to evaluate decision-maker engagement

with science

2.1 Introduction

In the previous chapter I explored public engagement with science (PES) as a model of science communication, concluding engagement was founded on “a set of ideological commitments by non-experts based on assigning value to scientific knowledge, processes or outcomes”. This valuing of science can motivate publics to actively set the engagement agenda. I also observed that the current framing of PES does not generally consider institutionally-based decision-makers as publics who actively engage in communicative processes about science. I argue that it is worthwhile to explore this idea of decision-maker publics given that they too can make sense of science through communication and because decision-makers have the power to influence the conduct and application of science for the public good.

(34)

27

2.2 Public engagement with science (PES)

Public participation and engagement can refer to a range of often democratically founded activities that don’t necessarily involve science such as those described by the International Association for Public Participation (2016) IAP2, for example. However, I am using the term PES in its application to science communication activities that involve the general public in “public issues involving science” (Wynne, 2014, p. 64) or “scientifically-framed issues” (Chilvers, 2008, p. 157). PES may manifest in a variety of activities such as museum visits and citizen science (highlighted in Chapter 1), but the context I am specifically focused on is engagement related to public decision-making. The discourse around this form of PES has developed in the context of public mistrust of institutionalised decision-making (e.g. Wynne, 2006). This means the PES model encompasses more than a mode of science communication that can actively involve citizens. By seeking to actively engage with people with knowledge and perspectives outside of science, there is an implicit recognition of the values of these inputs to issue-framing and decision-making. PES also encompasses ethical values around fairly and openly exploring these different sources of knowledge. These values are reflected in existing evaluation frameworks.

2.2.1 Existing evaluation frameworks

A number of frameworks and sets of principles exist for evaluating public participation Evaluative criteria for these activities have been developed by different means including: using theoretical and normative perspectives; empirically deriving criteria from participants and practitioners; or based on researcher opinion (Burgess & Chilvers 2006). Criteria derived from participants are likely to vary with context (Santos & Chess, 2003) and so those criteria are not dealt with explicitly here, although the frameworks of Rowe and colleagues are based on an amalgamation of criteria commonly appearing in the literature. It should also be noted that evaluation guidelines exist (generally produced by government-funded agencies) that summarize available criteria and give practical guidance and principles for conducting evaluations (e.g. Forss, 2005; Research Councils UK, 2011; Sciencewise, 2013). What follows is a discussion of existing frameworks and principles that propose generalisable criteria for the evaluation of PES.

(35)

28

of Beierle’s (1998) outcomes-based approach, incorporate criteria related to the nature of dialogue, analysis or deliberation. One feature of communication proposed for evaluation is how effectively technical and often complex information is communicated to non-expert publics (Chilvers, 2008). The intent here is beyond a purely pedagogic or deficit view of science communication but rather a recognition that in order for non-expert participants to engage effectively—and so achieve a democratic ideal of “equal contribution” as per Schroeter et al. (2016)—they need to be able to make sense of the scientific claims put forward by scientists. Valuing the ability of participants to make, challenge and defend claims whilst engaging with science can be traced back to the ideal speech acts of Habermas (1984). Competence in discourse is viewed as necessary in order to construct “the most valid understandings and agreements possible given what is reasonably knowable at the time” (Webler, 1995, p. 58).

(36)

[image:36.595.63.522.107.767.2]

29

Table 3: Summary of existing evaluation frameworks for public participation showing the source, basis and context of evaluation criteria.

Source Context of framework/ principles/ guidelines

Metacriteria/

categories Laird (1993) US

theoretical/normative

Public participation in (science & technology) policy making

Democratic process - pluralism

Democratic process - direct participation

Webler (1995) US theoretical/normative

Public participation Fairness (ethical normative) Competence (functional analytical) Beierle (1998) US

researcher/substantive

Public participation in environmental decision-making using social goals

Educating and informing the public Incorporating public values into decisions Improving quality of decisions

Increasing trust in institutions Reducing conflict

Achieving cost-effectiveness Rowe et al., (2008) UK

researcher/normative

Public participation in policy-making

Acceptance criteria

Representativeness, Independence, Early involvement, Influence, Transparency

Process criteria

Resource accessibility, Task definition, Learning, Quality of deliberation, How information is collated, Cost-effectiveness

Other

Aims of event organizers Criteria derived from participants Burgess & Chilvers

(2006) UK researcher/opinion

Participatory technology assessments

Context and decision situation Engagement process

Outputs

Outcomes (e.g.learning, reflexivity) Chilvers (2008) UK

practitioner/opinion

Participatory appraisal (of science and technology)

The analytic-deliberative process Scientific analysis

Access to information & specialist expertise Deliberation

Sciencewise UK government guidelines

The effectiveness of public dialogue

Meeting project objectives Standards of good practice Participant satisfaction Impact

Cost-benefit Schroeter et al. (2016)

EUR

Theoretical/ normative

Public participation in policy-making

Inclusiveness

Platform for communication & negotiation Equal contribution

Information exchange and learning

Exchange of knowledge, Common base of information, Transparency, Common understanding of the process

Influence on political decisions

Effectiveness/Efficiency

Frameworks for evaluating decision maker engagement with science