Universität für Bodenkultur
Department für Wirtschafts-und Sozialwissenschaften
Types and Functions of Evaluations in Three
Austrian Sustainability Strategies
Universität für Bodenkultur
Department für Wirtschafts-und Sozialwissenschaften
Types of Evaluations
Progress Monitoring:
Output-type monitoring and reporting
Progress reporting
Spending reviews
Auditing
Formative and Summative Evaluations:
Process evaluation
Implementation evaluation
Impact evaluation
Different Functions of Evaluations
Different Functions of Evaluations
Planning
Function
Decision-making: develop and evaluate decision alternatives
Implementation: inform and motivate persons involved
Monitoring: examine outcomes and effectiveness
Feedback and adaptation: provide basis for learning from evaluation results
Economic
Function
Optimization: safeguard efficient allocation of resources
Political
Function
Legitimation: provide a source of authority for political decisions and actions
Accountability: the allocation or acceptance of responsibility for actions
Whitewash: cover up programme failure
Postponement: provide arguments to avoid or postpone action
Ritual use: evaluation without any intention to use the findings
Administrative
Function
Survival: provide arguments for continued existence of institution or process
Growth: provide arguments for securing resources
Territoriality: provide arguments for protecting administrative competencies
Self-presentation: document administrative willingness to learn
ration
alist-i
nstru
menta
l func
tions
politic
al, sy
mboli
c func
tions
Universität für Bodenkultur
Department für Wirtschafts-und Sozialwissenschaften
Which type(s) of evaluations have been used in the three Austrian
strategy processes and what are their functions?
How to explain the different types and functions of evaluations in the
strategy processes?
Discussion
Universität für Bodenkultur
Department für Wirtschafts-und Sozialwissenschaften
Strategy processes as elements
of environmental governance
Key tenets of environmental governance:
(Jänicke & Jörgens 2006)
Strategic approach: consensual, broad-based target
and strategy formulation with a long-term horizon
Integration: of environmental concerns into other policy areas and sectors
Participation: by NGOs and citizens
Cooperation: between state and private-sector actors
Multi-level coordination: from global to local
Monitoring and evaluation: diverse range of reporting obligations and indicators
National sustainable development strategies:
transition form “misconceptions of ideal and static master plans and one-off initiatives,”
to “sets of co-ordinated mechanisms and continuing processes of monitoring, learning
Universität für Bodenkultur
Department für Wirtschafts-und Sozialwissenschaften
3 Austrian strategy processes
Austrian Biodiversity Strategy
Austrian Forest Dialogue
Austrian Strategy of
Universität für Bodenkultur
Department für Wirtschafts-und Sozialwissenschaften
Progress Monitoring
National Biodiversity Commission: central role as
information pool & in monitoring of implementation activities
Online database „Living Document“: collection of on-going activities
501 activities reported as of Feb. 2003, but no further activities added since then
Evaluation
Two-step procedure:
1
st
evaluation (2001) assessed to what degree the measures
formulated in the 1
st
ABS had already been implemented and by whom
2
nd
evaluation (2003) assessed the strategy document itself
Austrian Biodiversity Strategy
Universität für Bodenkultur
Department für Wirtschafts-und Sozialwissenschaften
Type:
mix of process and output evaluation
semi-outsider evaluation (Federal Environmental Agency)
partly based on “self-reporting”
Functions:
1
st
evaluation:
Ministry of Environment wanted to keep the topic of biodiversity
on the national political agenda (administrative function)
“work-to-rule exercise” to fulfill international obligations (political function)
2nd evaluation:
was supposed to lay foundation for revision of ABS (planning function)
Universität für Bodenkultur
Department für Wirtschafts-und Sozialwissenschaften
Impact:
2
nd
ABS (2005) partly based on evaluation results
some of the most severe flaws of the first strategy could be cured
new type of implementation instrument was introduced (Action Plans)
but 2
nd
strategy still falls far behind an ideal-type strategic planning approach
Institutional and Governance Context:
1
st
ABS adopted by Council of Ministers, 2
nd
only by Commission
“Biological diversity” as a poorly comprehensible term
Low problem pressure and lack of public attention
Low political weight and weak institutionalisation
Complex allocation of competences and high number of veto players
Antagonistic stakeholder interests: a weak environmental coalition and strong business
interests groups
Universität für Bodenkultur
Department für Wirtschafts-und Sozialwissenschaften
Progress Monitoring
Internal process surveys in the course of programme formulation
Internal monitoring of implementation by BMLFUW
Evaluation
No external evaluation so far (to be decided by mid 2008)
Austrian Forest Dialogue
Universität für Bodenkultur
Department für Wirtschafts-und Sozialwissenschaften
Assessment of the AFD Evaluation
1/2
Type:
Internal process surveys among participants: evaluating and
monitoring the process
Internal (informal) monitoring of Work Programme implementation by BMLFUW
Functions:
Internal process surveys - “keeping actors on board / the process on track”
(political function):
participants’ attitudes and perceptions as regards procedural aspects; internally used
by process management bodies
Universität für Bodenkultur
Department für Wirtschafts-und Sozialwissenschaften
Assessment of the AFD Evaluation
2/2
Impact:
Procedural management partly influenced by results of process
surveys (t
opics to the agenda, priority of measures, AFD goals)
No outcome evaluation so far
Institutional and Governance Context:
Forest Programme and Work Programme adopted by Round Table in 2005;
Council of Ministers “took notice”
Moderate institutionalisation (in comparison)
No specific budgets for implementation of measures
Low public attention to forest policy
Progress Monitoring
Working Programmes (WP 2003, WP 2004)
Online Database of Measures (currently 281 measures)
Bi-Annual Progress Reports (2004, 2006)
Bi-Annual Indicator Reports (2004, 2006)
Semi-external process and implementation reviews
Evaluation
External evaluation done by independent researchers and consultants in 2005
Review by the Austrian Court of Audit (October-November 2005)
Austrian Strategy for Sustainable Development
Universität für Bodenkultur
Department für Wirtschafts-und Sozialwissenschaften
Type:
Mix of progress reviews and process/impact evaluations
Evaluations focused on implementation instruments of the ASSD, not on
the strategy and its policy goals
External evaluations (Independent researchers; Court of Audit)
Functions:
Evaluation as part of the strategy process (“Learning strategy”) (planning
function)
Present ASSD as successful role model; Fulfill reporting obligations under
the EU SDS (political functions)
Universität für Bodenkultur
Department für Wirtschafts-und Sozialwissenschaften
Impact:
Recommendations will be used to strengthen the strategy process
Stronger position for the Forum Sustainable Austria
More active role of the federal states („Regionalization“)
Prioritization of the Working Programmes and Measures
Institutional and Governance Context:
Supportive international and domestic political context
ASSD has been adopted by Council of Ministers in 2002
High degree of institutionalization
No capacity problems (staff and financial resources)
Comparative Perspective
Type
Function
Institutional Capacities
and Political Profile
ABS
Process and Output
Evaluation
Mainly Adminstrative
Low
AFD
Process Reviews
Mainly Political
Moderat
ASSD
Process, Output and