4.7 Policy evaluation
4.7.1 Command and control policy
Top down command and control policy (CCP) approaches in natural resource management failed to resolve the problems in natural resources in many different national contexts (Holling and Meffe, 1996; Bardsley et al., 2001; Clausen and McAllister, 2001;
Lal et al., 2001). In the 1970s when many of these landmark policies were established,
issues were considered to be largely local, reversible and direct whereas today impacts are changing rapidly, are considered to be irreversible and geographically are at a global scale (Daily, 2000). Although objectives were set, they were without context often in the absence of a regional planning and process framework. In Western Australia this deficiency is now being addressed through the regional natural resource management strategies policy.
These early policies were criticised on the basis that they neglected the intrinsic cycles of natural and social systems, were inefficient or even worse than doing nothing, and were based on static rather than dynamic models (Holling and Meffe, 1996). Furthermore, CCP approaches have also been described as ineffectual or unsatisfactory, often achieving undesired management outcomes (Lal et al., 2001). Mullner et al. (2001) more disparagingly described similar policies in the U.S. as “autocratic natural-science-based management of renewable resources” institutionalised in the early 20th century following the principle of management based on science with administrative decisions by professional
agency employees. Consider, for example, the conclusion about failure in forestry planning in the U.S. in which the technical and systematic processes that were designed to reach the ‘right’ answer were inadequate (Patterson and Williams, 1998).
Bardsley et al. (2001) proposed that natural resource policy was based on political expediency and identified four characteristics of natural resource policy that contributed to their failure.
1. Environmental policy treated each segment of the environment separately as individual concerns. Each separate policy was judged on its merits within a particular context. Unless the chain of contextual factors are identified and the decision-making rules changed, there will be little change in actions and
outcomes (Edwards and Steins, 1999). Change will require a divergence from normal science methodology and the adoption an alternative philosophy and
methodology that identifies the contextual factors.
2. Policy has not taken account of how incentives change behaviour.
3. Policy formulation and implementation have not been dealt with together. 4. There has been little or no monitoring and auditing of environmental outcomes
resulting in lack of accountability of public money.
In general, natural resource issues were a secondary consideration to the primary productive purpose of a region and consequently the former were dealt with in a reactive manner rather than a proactive preventative manner (Clausen and McAllister, 2001). For example, on private land in the WA agricultural region the primary objective was agriculture, and issues of natural resource management, other than for productivity, were add-ons when and if agricultural profit permitted. Concern over the alleged failure of public policy in the 1960s and 1970s led to a resurgence of policy analysis studies in the 1980s. Under the general heading of policy research, there are many
Accordingly, the intent can vary greatly, depending on the aim, the methods and the affiliation of the researcher (Hogwood and Gunn, 1992). Mobbs and Dovers (1999) made a broad distinction between descriptive and analytical policy research, consistent with the categories of Hogwood and Gunn (1992) who distinguished between policy studies and policy analysis respectively, although these typologies are by no means definitive nor exhaustive. Mobbs and Dovers (1999) also identified some common methodological approaches that might be applied to natural resource management: for example, political science approaches; psychological/sociological approaches; policy/program evaluation; public choice approaches; legal policy research; institutional analysis; policy cycle analysis; and decision process approaches. There is no agreed approach to applied policy analysis and policy evaluation has been criticised as too often coming from a single disciplinary perspective (Syme and Sadler, 1994).
In public policy there are two overarching schools of thought. The dominant of these is the positivist, rational and empirical school in which there are five commonly used methods: surveys, experimentation, interview, use of data and the use of case studies. Post-positivist or subjective research methods include history, naturalistic inquiry, and stories. Post-positivists such as Sabatier and Jenkins-Smith (1993) and Fox (1990) argued that the analysis of policy implementation had become too complex for the rational positivist approach and suggested a more intuitive or participatory approach of the post-positivist or subjective school was required.
Whether of not specific actors have been attributed with a role in the policy implementation process in policy implementation studies depends on the fundamental epistemological or ontological premises of the researcher (Howlett and Ramesh, 1998). In
a review of policy change Howlett and Ramesh (1998) proposed that most early positivist policy analysis studies argued that actors and their behaviour made very little difference to policy outcomes. These studies would occupy a position within the functionalist paradigm in the Burrell and Morgan framework shown in Figure 4.2. In contrast the more recent post-positivist approaches to the study of policy making focus on the way in which the language of politics ‘constructs’ public policy. In particular, the language of politics is predisposed with interpretations of what the policy ‘problem’ is, in an ontological sense.
Recent policy analysis models have made greater attempts to integrate the broader suite of important factors that link policy to performance or effectiveness. To achieve better integration the models were designed to take a macro-level perspective focussing on the tractability of the problem with reference to the social and political context rather than becoming immersed in the detail (Sabatier and Mazmanian, 1980). The changes between the macro- and the micro-level analysis frameworks highlighted the perennial problem of the tension between theory and practice in policy studies. Lester and Goggin (1998) argued for the need for reform in policy implementation studies suggesting that fundamental changes were required that moved the orientation away from the reductionist positivist tradition of normal science towards a ‘post-positivist epistemology’ .
Dissatisfaction with the outcomes of implementation research lead to the development of the Advocacy Coalition Framework, evolved from an earlier implementation analysis framework (Sabatier and Mazmanian, 1980). The new approach emphasised the need for new conceptual approaches that included policy dynamics and learning over time within a systems framework (Sabatier and Jenkins-Smith, 1993). The approach focused on the establishment and maintenance of competing ‘coalitions’ of
interest groups, government institutions/agencies, and key individuals, each attempting to realise their preferred policy options in a subsystem of political pressure, strategy, and information sharing called the ‘policy subsystem’. Sabatier and Jenkins-Smith (1993) proposed that values underpin the guiding instruments or strategies that influence decision-making and provide the causal theory upon policy positions are constructed. Shared values can be used to map the relationship between various organisations within the subsystem. This latter policy analysis method is appropriate for the evaluation of the integrated and adaptive management approaches that have dynamic characteristics, involve coalitions of groups at the local, regional, state and national levels, and involve knowledge acquisition since the Advocacy Coalition Framework predicts that policy core beliefs hold coalitions together. Coalition survival during policy implementation, especially in the face of unanticipated complexity and uncertainty, requires learning and the ability to adapt to changing policy conditions. This approach is also consistent with the theoretical changes taking place in natural resource management described in Chapter 6.
I suggest in this thesis that the epistemology is based on the normal science paradigm (positivist orientation) and is concerned with the direct cause and effect level of understanding within the concept of single-loop learning in which information feedback is interpreted by existing mental models and assumptions. In single-loop learning, feedback operates in the context of existing decision rules, strategies, culture and institutions which in turn are derived from our mental models. In contrast double-loop learning is concerned with feedback from the ‘real world’ in which it can stimulate changes in mental models. Such learning involves new goals and new decision rules, not just new decisions (Sterman, 2000)(Figure 4.6).
a
b
Real world Decisions Information feedback Mental models of the real world Strategy, structure, decision rules Real world Decisions Information feedback Mental models of the real world Strategy, structure,decision rules
Figure 4.6: Single- and double-loop learning. Single-loop learning (a) occurs when information feedback is interpreted by existing mental models. The learning feedback operates in the context of existing decision rules, strategies, culture and institutions which in turn are derived from our mental models. In contrast double-loop (b) learning is concerned with feedback from the real world in which it can stimulate changes in mental models. Such learning involves new goals and new decision rules, not just new decisions. Source: Sterman (2000)
In the United Kingdom some of the key lessons of the Economic and Social Research Council’s (ESRC) Global Environmental Change Program (ESRC Global Environmental Change Programme, 2000) are:
1. that all policy domains need to incorporate environmental and social aspirations and hence involve human values;
2. decisions have to deal with conflicting opinions about environmental problems; 3. the risks they pose to different groups in society; and
4. the need to build trust and engage citizens in solutions.
Although the programme originally focussed on global issues it increasingly turned its attention to the implementation of sustainable development and in doing so research investigated smaller-scale measures and processes–at local, national and regional levels–which contribute to the achievement of sustainability at the global scale.
the interdependencies of natural, political and social systems, and technology, often being described as ‘messy’ or ‘wicked problems’ (Mason and Mitroff, 1981; Bellamyet al., 1999; Bellamy and Johnson, 2000) and will require very different ways in which to conceptualise and understand them (discussed in Chapter 5).
4.7.2 Integrated natural resource management
Although the rhetoric of integrated and adaptive approaches to natural resource management are embracing general theoretical developments in systems theory, ecology and the participative and adaptive management philosophies (Gunderson et al., 1995; Bellamy et al., 2001; Patterson and Williams, 1998), there are few methodologies to evaluate policy implementation based on the same foundations. Bellamy et al. (2001) believed this limitation contributed to the failure of policy initiatives to address sustainable and equitable resource use. Bellamy et al. (2001) identified the lack of evaluating frameworks to help guide continuous program development in the way natural resource management initiatives contribute to ongoing improvements in resource use, sustainability and social wellbeing of communities concerned. Bellamy et al. (2001) reviewed previous evaluation techniques and found conventional economic methodologies to be inadequate to meet the multiple objectives of natural resources management. In addition, two specific models of policy implementation evaluation, the Mazmanian-Sabatier model (Sabatier and Mazmanian, 1980) and the Planning, Research, Implementation, Monitoring and Evaluation Framework (PRIME) (Symeet al., 1994) for the development, implementation and evaluation of whole catchment management plans, were criticised for deficiencies in addressing instrumental assumptions (that is, causal theory in terms of the model) and deficiencies in problem context or formulation and structures, respectively. The more
recent Advocacy Coalition Framework of Sabatier and Jenkins-Smith (1993) may have been a more appropriate framework to use to evaluate these approaches but was not included in the research.
In order to fill the gap of lack of comprehensive evaluating frameworks for policy implementation, Bellamyet al.(2001) devised an integrated systems-based framework for the evaluation of natural resource management policy initiatives with three objectives to provide:
1. a basis for an integrated evaluation of the different perspectives (for example, social, economic, environmental, institutional and technical) on the performance of the natural resource management initiative;
2. a framework for guiding implementation; and 3. a rigorous basis for synthesising findings.
I suggest that a fourth objective might be realised in the construction of an evaluation framework. It may also act as a tool to help understand the problem in the epistemological sense.
The framework of Bellamy et al. (2001) comprises the identification of seven components:
1. the context which is composed of environmental, economic, social, institutional and technological factors;
2. the issue characterisation; 3. the object or intent; 4. instrumental assumption; 5. process of implementation; 6. products; and
It marks a major shift towards incorporating the theory of systems into natural resource management. It does so by emphasising the importance of the context or ‘problem situation’ and by providing for two important steps in the process, the implementation stage and a rigorous basis for synthesising the findings. However, this framework omitted one important component. Although Bellamy et al. (2001) identified in the text a range of factors that reflect the worldviews or mental constructs of the evaluation team (for example, the values, priorities, experiences and organisational culture) which will influence the formulation of the problem and the analytical models used, it was not incorporated explicitly as a step in the framework. Consequently this framework suppresses the importance of mental constructs on the process and the potential impacts that this may have on the outcomes of the evaluation. Because of this the mental construct including assumptions of the researcher are not made explicit. Hullet al.(2002) showed empirically that assumptions about nature were embedded in people’s preferences for environmental policy and management and constrained people’s vision of what environmental conditions could and should exist, thereby constraining the future that could be negotiated, further emphasising the need for the practitioner’s assumptions to be made explicit.
In this thesis a general systemic framework for understanding problem-solving (Jayaratna, 1994) is used and described in Section 5.2. There are three contexts in which any method is used and their identification is important in understanding how they are used and whether or not they are effective (Jayaratna, 1994). The three contexts are creation, selection/interpretation and action. The constructed method reflects the mental construct of the creators. Whereas the people who select the method to solve their perceived problem may try to interpret the method through their own mental constructs.
Those who ultimately have to use the method may apply it according to their mental constructs. Consequently the context in which the method users interpret the method may change its nature, form, structure and content, and therefore its effectiveness (Jayaratna, 1994). A fuller examination is given in Chapter 6.
4.8
Conclusion
In this chapter I have examined a number of the theoretical factors that underpinned policy and natural resource management, emphasising the importance of paradigms and conceptual metaphors. While CCP and integrated natural resource management are components of a policy mix which have been inadequate as drivers of behavioural change in agriculture towards sustainable land management outcomes. Not only were the policies based on the science of the 1950s, 60s and 1970s but also the policy-makers and managers were products of the same system, trained mostly in narrowly focused disciplinary research, shaping their mental constructs. Although integrated approaches had an alternative epistemology to that of CCP this was not adopted and the approaches practised with the epistemology of normal science. That is to say, those who selected and used the method may have interpreted the method through their own mental constructs. Therefore, the policy models, whether they were CCP or one of the integrated methodological alternatives, were shown to be practised within the normal scientific paradigm. Consequently changes in natural resource management policy from CCP to integrated approaches operated only at the level of tools, techniques and methods and were not marked by a paradigm shift.
alternatives, were shown to be practised within the normal scientific paradigm. Out of this paradigm came such axioms as the integrity and stability of nature and the view that people were outside the system in an objective approach. However, the ecology of the 1990s and 2000s has replaced these notions with concepts of resilience, non-linearity and multiple-stable states, and the dynamics of systems of linked people and nature. Science, society and nature are interlinked into a whole system, and a new epistemology, theory and praxis are required to meet the challenges of producing resilient sustainable systems. From the review of natural resource degradation in Chapter 3, it is clear that existing natural resource policies have failed to manifest sustainable land management practices that will mitigate natural resource degradation. The current integrated resource management policies, which support adaptive management, are the latest attempt to address the issues of natural resource degradation. The audit of natural resource management since 1997 (National Land and Water Resources Audit, 1997) has shown no improvement in natural resources despite the amount of public funding directed at these issues.
It has been suggested that the failure of science to resolve certain problems may be due to uncertainty about whether the appropriate questions are being asked, and whether problems are addressed with appropriate theoretical and methodological tools and within an appropriate paradigm (Wynne, 1974). It is time to rethink the questions when decisions result in persistent problems. Science, society and nature are interlinked, and a new epistemology, theory and praxis are required to meet the challenges with a greater emphasis on the way that we understand, define and formulate the problem. In Chapter 5, I review the literature for the emerging theories of understanding the dynamics
of natural resource management in complex systems and construct a research framework from which to examine the dynamics of the WA agricultural region.
CHAPTER 5
A CONTEMPORARY EPISTEMOLOGY FOR
NATURAL RESOURCE MANAGEMENT OF 21
T HCENTURY AND THE RESEARCH FRAMEWORK
Separations of disciplines and politics are artefacts of the human mind, not characteristics of the real world.
Donella Meadows and Jennifer Robinson, 1985.
5.1
Introduction
In Chapter 4, I examined the epistemology of natural resource management of the 20th century and I made the case that natural resource management was based on the normal science paradigm and that there was a need to change towards a pluralistic and holistic approach. At the time when many natural resource policies were established, issues were considered to be largely local, reversible and direct whereas today impacts are changing rapidly, are considered to be irreversible, and geographically and economically interactions occur at a global scale (Daily, 2000; Lambin et al., 2001). Conceptual development has not kept pace with the speed of changes that alter and control the processes in large-scale systems (Gunderson and Pritchard, 2002). Poor conceptual development of these systems has hindered our understanding of their dynamic behaviour and weakened our ability to respond to increasingly uncertain behaviour, neither with appropriate policy nor management.
The way we perceive problems and how we go about problem-solving and decision-making depends on human interpretation of information (Meadows and Robinson, 1985). Information from the past and present is required to make decisions and to take action, that requires knowledge of the future and the consequences of those decisions.