Types and Functions of Evaluations

16  Download (0)

Full text

(1)

Universität für Bodenkultur

Department für Wirtschafts-und Sozialwissenschaften

Types and Functions of Evaluations in Three

Austrian Sustainability Strategies

(2)

Universität für Bodenkultur

Department für Wirtschafts-und Sozialwissenschaften

Types of Evaluations

ƒ

Progress Monitoring:

ƒ

Output-type monitoring and reporting

ƒ

Progress reporting

ƒ

Spending reviews

ƒ

Auditing

ƒ

Formative and Summative Evaluations:

ƒ

Process evaluation

ƒ

Implementation evaluation

ƒ

Impact evaluation

(3)

Different Functions of Evaluations

Different Functions of Evaluations

Planning

Function

ƒ

Decision-making: develop and evaluate decision alternatives

ƒ

Implementation: inform and motivate persons involved

ƒ

Monitoring: examine outcomes and effectiveness

ƒ

Feedback and adaptation: provide basis for learning from evaluation results

Economic

Function

ƒ

Optimization: safeguard efficient allocation of resources

Political

Function

ƒ

Legitimation: provide a source of authority for political decisions and actions

ƒ

Accountability: the allocation or acceptance of responsibility for actions

ƒ

Whitewash: cover up programme failure

ƒ

Postponement: provide arguments to avoid or postpone action

ƒ

Ritual use: evaluation without any intention to use the findings

Administrative

Function

ƒ

Survival: provide arguments for continued existence of institution or process

ƒ

Growth: provide arguments for securing resources

ƒ

Territoriality: provide arguments for protecting administrative competencies

ƒ

Self-presentation: document administrative willingness to learn

ration

alist-i

nstru

menta

l func

tions

politic

al, sy

mboli

c func

tions

(4)

Universität für Bodenkultur

Department für Wirtschafts-und Sozialwissenschaften

Which type(s) of evaluations have been used in the three Austrian

strategy processes and what are their functions?

How to explain the different types and functions of evaluations in the

strategy processes?

Discussion

(5)

Universität für Bodenkultur

Department für Wirtschafts-und Sozialwissenschaften

Strategy processes as elements

of environmental governance

ƒ

Key tenets of environmental governance:

(Jänicke & Jörgens 2006)

ƒ

Strategic approach: consensual, broad-based target

and strategy formulation with a long-term horizon

ƒ

Integration: of environmental concerns into other policy areas and sectors

ƒ

Participation: by NGOs and citizens

ƒ

Cooperation: between state and private-sector actors

ƒ

Multi-level coordination: from global to local

ƒ

Monitoring and evaluation: diverse range of reporting obligations and indicators

ƒ

National sustainable development strategies:

ƒ

transition form “misconceptions of ideal and static master plans and one-off initiatives,”

to “sets of co-ordinated mechanisms and continuing processes of monitoring, learning

(6)

Universität für Bodenkultur

Department für Wirtschafts-und Sozialwissenschaften

3 Austrian strategy processes

ƒ

Austrian Biodiversity Strategy

ƒ

Austrian Forest Dialogue

ƒ

Austrian Strategy of

(7)

Universität für Bodenkultur

Department für Wirtschafts-und Sozialwissenschaften

Progress Monitoring

ƒ

National Biodiversity Commission: central role as

information pool & in monitoring of implementation activities

ƒ

Online database „Living Document“: collection of on-going activities

ƒ

501 activities reported as of Feb. 2003, but no further activities added since then

Evaluation

ƒ

Two-step procedure:

ƒ

1

st

evaluation (2001) assessed to what degree the measures

formulated in the 1

st

ABS had already been implemented and by whom

ƒ

2

nd

evaluation (2003) assessed the strategy document itself

Austrian Biodiversity Strategy

(8)

Universität für Bodenkultur

Department für Wirtschafts-und Sozialwissenschaften

ƒ

Type:

ƒ

mix of process and output evaluation

ƒ

semi-outsider evaluation (Federal Environmental Agency)

ƒ

partly based on “self-reporting”

ƒ

Functions:

ƒ

1

st

evaluation:

ƒ

Ministry of Environment wanted to keep the topic of biodiversity

on the national political agenda (administrative function)

ƒ

“work-to-rule exercise” to fulfill international obligations (political function)

ƒ

2nd evaluation:

ƒ

was supposed to lay foundation for revision of ABS (planning function)

(9)

Universität für Bodenkultur

Department für Wirtschafts-und Sozialwissenschaften

ƒ

Impact:

ƒ

2

nd

ABS (2005) partly based on evaluation results

ƒ

some of the most severe flaws of the first strategy could be cured

ƒ

new type of implementation instrument was introduced (Action Plans)

ƒ

but 2

nd

strategy still falls far behind an ideal-type strategic planning approach

ƒ

Institutional and Governance Context:

ƒ

1

st

ABS adopted by Council of Ministers, 2

nd

only by Commission

ƒ

“Biological diversity” as a poorly comprehensible term

ƒ

Low problem pressure and lack of public attention

ƒ

Low political weight and weak institutionalisation

ƒ

Complex allocation of competences and high number of veto players

ƒ

Antagonistic stakeholder interests: a weak environmental coalition and strong business

interests groups

(10)

Universität für Bodenkultur

Department für Wirtschafts-und Sozialwissenschaften

Progress Monitoring

ƒ

Internal process surveys in the course of programme formulation

ƒ

Internal monitoring of implementation by BMLFUW

Evaluation

ƒ

No external evaluation so far (to be decided by mid 2008)

Austrian Forest Dialogue

(11)

Universität für Bodenkultur

Department für Wirtschafts-und Sozialwissenschaften

Assessment of the AFD Evaluation

1/2

ƒ

Type:

ƒ

Internal process surveys among participants: evaluating and

monitoring the process

ƒ

Internal (informal) monitoring of Work Programme implementation by BMLFUW

ƒ

Functions:

ƒ

Internal process surveys - “keeping actors on board / the process on track”

(political function):

ƒ

participants’ attitudes and perceptions as regards procedural aspects; internally used

by process management bodies

(12)

Universität für Bodenkultur

Department für Wirtschafts-und Sozialwissenschaften

Assessment of the AFD Evaluation

2/2

Impact:

ƒ

Procedural management partly influenced by results of process

surveys (t

opics to the agenda, priority of measures, AFD goals)

ƒ

No outcome evaluation so far

ƒ

Institutional and Governance Context:

ƒ

Forest Programme and Work Programme adopted by Round Table in 2005;

Council of Ministers “took notice”

ƒ

Moderate institutionalisation (in comparison)

ƒ

No specific budgets for implementation of measures

ƒ

Low public attention to forest policy

(13)

Progress Monitoring

ƒ

Working Programmes (WP 2003, WP 2004)

ƒ

Online Database of Measures (currently 281 measures)

ƒ

Bi-Annual Progress Reports (2004, 2006)

ƒ

Bi-Annual Indicator Reports (2004, 2006)

ƒ

Semi-external process and implementation reviews

Evaluation

ƒ

External evaluation done by independent researchers and consultants in 2005

ƒ

Review by the Austrian Court of Audit (October-November 2005)

Austrian Strategy for Sustainable Development

(14)

Universität für Bodenkultur

Department für Wirtschafts-und Sozialwissenschaften

ƒ

Type:

ƒ

Mix of progress reviews and process/impact evaluations

ƒ

Evaluations focused on implementation instruments of the ASSD, not on

the strategy and its policy goals

ƒ

External evaluations (Independent researchers; Court of Audit)

ƒ

Functions:

ƒ

Evaluation as part of the strategy process (“Learning strategy”) (planning

function)

ƒ

Present ASSD as successful role model; Fulfill reporting obligations under

the EU SDS (political functions)

(15)

Universität für Bodenkultur

Department für Wirtschafts-und Sozialwissenschaften

ƒ

Impact:

ƒ

Recommendations will be used to strengthen the strategy process

ƒ

Stronger position for the Forum Sustainable Austria

ƒ

More active role of the federal states („Regionalization“)

ƒ

Prioritization of the Working Programmes and Measures

ƒ

Institutional and Governance Context:

ƒ

Supportive international and domestic political context

ƒ

ASSD has been adopted by Council of Ministers in 2002

ƒ

High degree of institutionalization

ƒ

No capacity problems (staff and financial resources)

(16)

Comparative Perspective

Type

Function

Institutional Capacities

and Political Profile

ABS

Process and Output

Evaluation

Mainly Adminstrative

Low

AFD

Process Reviews

Mainly Political

Moderat

ASSD

Process, Output and

Impact Evaluation

Figure

Updating...

References