EVALUATING LONG-TERM TRANSITION PROGRAMS ON A SHORT-TERM BASIS Towards a six-step transition program evaluation tool

(1)

EVALUATING LONG-TERM

TRANSITION PROGRAMS ON A

SHORT-TERM BASIS

Towards a six-step transition program

evaluation tool

Tom Creten, Sander Happaerts & Kris Bachus

HIVA – KU Leuven

Research paper 9

Leuven, September 2014

(2)

The P KU Le Plea Cret term Rese For This Cent This resp © 20 p.a. This Policy Researc euven, Ghent U ase refer to th ten T., Happa m basis. Towa earch Centre o more inform s publication tre Programm s publication r ponsible for an 014 POLICY RESE Administratio HIVA - Onder Parkstraat 47 s publication i ch Centre Tra University, VIT his publicatio aerts S. & Ba rds a six-step on Transitions ationkris.bac was sponsore me. reflects the v ny use which m EARCH CENTRE O on Policy Rese rzoeksinstituu 7 box 5300, B s also availab ansitions for S TO and Erasm on as follows: chus K. (2014 transition pro s for Sustainab chus@kuleuve ed by the Flem views only of t may be made N TRANSITIONS FO earch Centre ut voor Arbeid E 3000 Leuven le via www.st Sustainable D mus University 4), Evaluating ogram evalua ble Developm n.be mish Governm the author, a of the inform OR SUSTAINABLE on Transitions d en Samenlev n, Belgium teunpunttrado evelopment b Rotterdam. g long-term t ation tool (Res

ent.

ment, under t and the Flemis mation contain DEVELOPMENT s for Sustaina ving o.be brings togethe ransition prog search Paper 9 the 2012-2015 sh Governmen ed therein. ble Developm er researcher grams on a sh 9). Leuven: P 5 Policy Rese nt cannot be ment s from hort-olicy earch held

(3)

(4)

Voorwoord (in Dutch)

Deze paper stelt een ontwerptool voor die kan dienen om transitiebeleidsprogramma’s te evalueren. De tool is voornamelijk bedoeld voor de ‘managers’ of coördinatoren van transitieprocessen (in de eerste plaats opgezet vanuit het Vlaamse beleid). Maar hij kan evengoed gebruikt worden door andere betrokkenen in gelijkaardige processen of door externe evaluatoren. De toepassing van de tool moet kunnen aantonen of het transitieproces goed verloopt en of de overheid wel de juiste beleidsondersteuning biedt om de vooropgezette doelstelling te kunnen behalen. Hij moet kunnen aangeven welke goede stappen er al gezet zijn, wat nog ontbreekt en hoe de overheid kan bijsturen.

Op die manier zorgen we ervoor dat de transities, en vooral het beleidsprogramma dat opgezet is om ze te sturen, op korte termijn en pragmatisch geëvalueerd kunnen worden, hoewel ze verandering op lange termijn beogen.

De tool focust op de niches die gepromoot worden om een bepaalde transitie te versnellen of in de juiste richting te sturen. Hij kan daarom toegepast worden om één, meerdere of alle verschillende niches te evalueren die in een bepaald beleidsprogramma aan bod komen. Dat komt tegemoet aan de vraag om een rollende evaluatie te kunnen doen, zonder meteen een hele transitie (waar de overheid slechts in beperkte mate controle over heeft) te moeten beoordelen. Zes verschillende stappen moeten doorlopen worden om tot die evaluatie te komen.

In een eerste stap gaan we de interne samenhang van het transitiebeleidsprogramma na. Zijn de geselecteerde niches wel de juiste? Hebben ze, ongeacht hun uitwerking, de mogelijkheid om de langetermijndoelstellingen van het transitiebeleid (ten dele) te bereiken?

De tweede stap kijkt naar een aantal cruciale procesacties die moeten gebeuren om een transitiebeleidsprogramma te doen slagen, bijvoorbeeld op het vlak van visievorming en samenwerking.

De derde stap doet hetzelfde voor een reeks cruciale beleidsactiviteiten die, in verschillende domeinen, de mechanismen van transitiebeleid in gang moeten zetten.

Om de beleidsactiviteiten uit te voeren, moeten een aantal instrumenten ingezet worden. Als vierde stap gaan we dan ook de goede implementatie van die beleidsinstrumenten na. Het beleid kan immers wel goed bedacht worden, maar moet ook nog correct uitgevoerd worden.

Ten vijfde proberen we te achterhalen welke van de verwachte effecten er al gerealiseerd zijn. Dat doen we door te meten in hoeverre het gangbare regime al dan niet weerstand biedt aan de niches die gepromoot worden door het beleidsprogramma.

Als laatste stap bestuderen we het oorzakelijke verband tussen de effecten enerzijds en het beleidsprogramma anderzijds. Een transitie kan immers in een bepaalde richting evolueren, zonder dat die evoluties een noodzakelijk gevolg zijn van overheidsbeleid. De tool biedt daarom wel de mogelijkheid om de rechtstreekse invloed van het beleid in te schatten.

Het finaliseren van deze tool is het einddoel van onderzoekslijn 2 van TRADO. Op basis van deze eerste versie zullen we de tool daarom in 2014 en 2015 toepassen in twee transitieprocessen van Vlaanderen in Actie, nl. ‘Slimme Mobiliteit’ en ‘Flanders’ Care’. Op basis van de ervaringen van die twee casestudies zullen we de tool verder verfijnen en aanvullen waar nodig. Op het einde van het project leveren we een afgewerkte tool met een gebruikershandleiding af aan de Vlaamse overheid.

(7)

vi | Tom Creten, Sander Happaerts & Kris Bachus

Executive summary

Despite a growing body of literature on transitions, surprisingly little has been published on the evaluation of policy programs that aim to enhance ongoing transitions in society. Possibly, such is due to the inherent paradox in the evaluation of transition policy. While policy-makers are interested in rather short-term evaluations to know when and how to adapt ongoing policy programs, transition policy by definition aims at achieving system change only on a very long time horizon.

To tackle that paradox, we develop in this paper a hands-on six-step evaluation tool for transition policy programs that can be used both by civil servants during the coordination of a transition program and by external evaluators performing ex-post evaluations. The tool is based upon an innovative combination of the transitions literature with the knowledge from the field of traditional (environmental) evaluation. Such an original combination of two strands of literature also forms an academic added value to the transitions field of study.

Specifically, we make use of the distinction made by environmental evaluation scholars between output, outcome and impact effects (Gysen et al., 2006b). In the case of transition programs, the ultimate goal of system change is the impact effect that is hoped for, but which is almost impossible to evaluate given its long time horizon. Therefore, we only investigate the potential impact of the program and otherwise focus upon the intermediate output effects (direct policy results) and outcome effects (influence on target groups) that should be visible on a shorter term, next to the important aspects of process evaluation that we also take into account. Apart from that, we also make use of the program theory evaluation methodology which is based upon the idea that policy-makers have in mind a well-defined theory about how their policy will influence reality. In the case of transition programs, the assumptions that form the basis of that policy program theory come from the transitions literature. Our evaluation tool therefore investigates three possible problems:

1. The evaluator investigates the possible occurrence of an application deficit. In that case, the transitions literature was not well applied in the development of the policy program theory. Essential elements that should be common to any policy theory based upon transitions thinking are missing in reality, which explains why success is impossible.

2. The evaluator should examine the possibility of implementation deficits. Apart from the common elements that should be present in each transition policy theory, the policy-makers will also at some point make a choice for specific policy instruments. Those specific instruments are needed to carry out in practice the general crucial policy activities that are common to all transition programs, but will themselves be different for each transition program. Whatever the chosen instruments however, they should be used and implemented in an appropriate way, according to the ‘spirit’ of the instrument.

3. Finally, there is the possibility of a policy theory failure that should be evaluated. In that case, it is the transition program theory itself that proves to be invalid, at least for the policy problem under consideration. It means the policy-maker’s assumptions and expectations (based for a large part upon the transitions literature, but also on assumptions on specific policy instruments) about the consequences of the chosen policy were wrong.

(8)

For the latter problem, use will be made of a very specific methodology called the ‘modus narrandi’ (Gysen et al., 2006b) to investigate causal chains between the transition program on the one hand and the outcome effects on policy target groups on the other hand.

In the first step of the evaluation tool, the internal consistency of the program is investigated. Specifically, we focus on the question whether or not the chosen niches of the program can contribute to the final aim of system change for the system under consideration, i.e. we investigate the potential impact, of the transition program. In case all scientific literature aims in the same direction, namely that the chosen niches are irrelevant or pernicious for the long-term objective of the program, then we can conclude the crucial policy theory assumption that the chosen niches will contribute to the wanted system change is false and a theory failure occurs. Also, any further evaluation in fact becomes obsolete. That is why we start off with that first step.

In the second and the third steps, we look for possible application deficits. The second step identifies a number of crucial process activities that should be present in any transition program. The third step does the same for crucial policy activities (output effects) that should be witnessed. In case those crucial process and policy activities do not take place, the transitions theory was not well applied in the development of the policy theory.

The crucial policy activities can be carried out by making use of a several different policy instruments. Those instruments have to be implemented well in case any outcome results are to be expected. That is the research object of step 4.

In step 5 and 6 we finally evaluate the policy program theory itself. We evaluate whether or not the expected outcome effects from the transition program have occurred. Such is investigated by monitoring indicators that point towards a decrease of regime resistance against the chosen niches. In case the indicators do go in the expected direction, we still need to evaluate to what extent such can be assigned to the transition program. As such, we need to tackle the issue of causality and that is exactly what we do in the last step when reconstructing causal chains using the modus narrandi methodology.

For all six steps we give very concrete evaluation questions that should be posed by the evaluator and/or clear indicators that can be used. Table 8 in the paper provides a visual overview of the total six-step evaluation tool and is shown at the end of this summary as well. In the paper, many concrete examples are provided from the transition policy programs in the Belgian region of Flanders, in the context of which this research project has been set up. In the continuation of this project, we will apply the evaluation tool on existing transition programs so to make further improvements and refinements. In particular, we will attempt to improve the evaluation questions and indicators so to make the tool even more widely applicable to all kinds of transition programs (for technological as well as social transitions) and we will investigate the possibilities to evaluate also the role of other partners in the transition arena except for the government. Finally, while we focus strongly upon transition programs that nurture niches in this paper, the practical examples throughout the paper already clarify that niches are to be defined in a wide sense. In further research, we will investigate whether the evaluation tool can be used for transition programs in a broader sense.

(9)

viii | Tom Creten, Sander Happaerts & Kris Bachus

Evaluation step

Evaluation criteria

Evaluation method Type of

evaluation Transition phase 1 Internal program consistency Evaluation questions:

- What are the long-term policy objectives of the transition program? - What are the chosen niches in the

transition program?

- What is the potential impact of the chosen niches and are those consistent the long-term policy objectives of the program?

Potential impact Stabilization 2 Crucial process activities Checklist:

- Setup of a transition ‘arena’ for a specific socio-technical system

- Development of a long-term vision on the system within the arena - A actor, level and

multi-domain approach is chosen, but with a single point of coordination

- Setup of learning moments

Process Predevelopment

3 Crucial policy activities

Checklist:

- Stimulating R&D for the chosen niches

- Levelling and turning the playing field - Shifting public resources

- Avoid regime reinforcement

Output Predevelopment 4 Policy instruments implement-tation Evaluation questions:

- What is the “spirit” or the aim of the policy instrument?

- What are the essential elements for good implementation?

- Are those elements present? - If not, wherein lies the problem?

Process Predevelopment

Implemen-tation

deficit?

Application

deficit?

Program

theory

failure?

(10)

5

Decreasing regime resistance

Indicators:

- Powerful actors have embraced the niches

- Niche players are becoming powerful actors

- Increasing market share for the niches

- Increasing efficiency (performance-price ratio) of the niches

- Decreasing complexity of the niches for end users

- Emergence of network organizations for the niches

- Increasing public support

Outcome Take-off & Acceleration 6 Causality between program and outcomes

Reconstruction of causal chains by using the Modus Narrandi method

Causality assessment From predevelopment to take-off & acceleration

Table 1: Transition program evaluation tool – extended version

Program

theory

failure?

(11)

(12)

Introduction

Sustainability transitions are becoming more common as a framework for sustainable development policies, encompassing but not limited to environmental issues. In transition programs (policy programs inspired by transitions), the paradigm of transitions thinking is used by policy-makers to decide upon policy interventions in socio-technical systems (e.g. the housing, food or energy system) to solve the persistent problems of sustainable development. In the Belgian region of Flanders, for instance, the Flemish government has formulated a sustainable development strategy based on the transitions framework, and it has additionally identified 13 transversal transition programs to achieve sustainable change in systems as diverse as energy, care, mobility, etc. (Vlaamse overheid, 2012d). 1

This paper is part of a research project, commissioned by the Flemish government, on the evaluation of the transition approach. It is a first attempt to develop a transition program evaluation tool based on a thorough analysis of different strands of literature. In next papers, the tool will be improved after a concrete application on a number of Flemish transition programs. After completion, the tool is intended to be used in the first place by the Flemish administration in the evaluation of its transition programs, and second as a contribution to the scientific literature on the evaluation of transition policy that could be useful for external evaluators.

We build upon the program theory evaluation literature, in which it is taken as an assumption that policy-makers have in mind a well-defined theory about how their policy program will influence the real world (Crabbé et al., 2006). In the transitions literature the process of system change is described step by step and the way policy can have an influence on those transitions is well described in certain streams of the literature, such as the transition management approach (Loorbach, 2007). Taking this literature as a basis, there usually is indeed a clear policy theory in policy program and they can for that reasons be considered to be ‘transition policy programs’ in the way ‘programs’ are perceived in the literature on program theory evaluation. We assume transition (policy) programs to include at the minimum aspects that are intended to stimulate niches on the one hand and destabilize the existing regime to enhance the possibilities of those niches to break through on the other hand. Such a strategy for the program should also fit within a long-term view on system change, concretized by a number of long-term objectives for the system under consideration that ought to be reached by the program. Despite a large body of literature on sustainability transitions, surprisingly little attention has been given to the evaluation of such comprehensive transition programs.

We attempt to fill that research gap by proposing a six-step evaluation tool for transition programs. The tool should be suitable for both extended external evaluation of transition programs, as well as for more modest internal evaluations and learning activities performed by the civil servants and

1_{The Flemish sustainable development strategy puts forward seven transitions, namely in 1/ housing and building 2/}

material use; 3/ energy; 4/ mobility; 5/ food; 6/ health care and 7/ knowledge and learning.

The thirteen transversal transition programs were identified in the context of the most important Flemish policy strategy called ‘Flanders in Action’. Those thirteen transition programs are: 1/ New Industrial Policy; 2/ ‘Gazellesprong’ (on the growth and internationalization of enterprises); 3/ Streamlining a directed innovation policy 4/ ‘iedereen mee, iedereen actief’ (on the match between education and labor market); 5/ The fight against child poverty; 6/ Flanders’ Care (on the health care system); 7/ Renewable energy and smart grid; 8/ sustainable housing and building; 9/ sustainable material use; 10/ ‘Ruimte voor morgen’ (on spatial planning); 11/ Smart mobility; 12/ Speeding up investment projects; and 13/ Towards a sustainable and creative city.

(13)

2 | Tom Creten, Sander Happaerts & Kris Bachus

other actors involved in the programs. The tool is developed through the combination of the limited literature on evaluation in transition theory and the more elaborated environmental evaluation literature. Using program theory evaluation as an overarching paradigm, that combination makes it possible to investigate three different potential policy problems. First, we can discover so-called

application deficits that indicate a policy theory (i.c. the program theory based on transitions thinking) was not well applied by the policy-makers. Next, there could be an implementation deficit, meaning that the specific policy instruments chosen to stimulate transitions were not well implemented. Finally, we might identify a policy theory failure, which means the transition program theory itself is not the best approach for the policy problem under consideration. The first and second problem can be discovered mainly by a thorough process evaluation, which is the most common type of evaluation in the transitions literature. The third problem of a theory failure (but also partly the application deficit problem) is only possible to investigate through an evaluation of the (intermediate) results of the transition program. The latter is rather rare in transition studies and this is where the combination with concepts and methods from the traditional environmental evaluation literature constitutes a genuine added value for the evaluation of transition programs.

While the tool is unique in the fact that it is able to evaluate a transition program comprehensively, it could nevertheless also be used to evaluate parts of the program. In particular, it could be useful to investigate the effects of the stimulation of one specific niche rather than all niches that are nurtured within the program. The evaluation of that part would still be a full or comprehensive evaluation nonetheless, given that the tool for that part still makes it possible to evaluate both the process aspects and the results of nurturing that specific niche and to discover both the problem with application and implementation deficits and the problem with theory failure.

The paper is structured as follows. Part 1 elaborates on the theoretical framework. We give an overview of the most inspirational literature in the field of transition studies and in the field of environmental evaluation. After that, we introduce a first compact version of the transition program evaluation tool. Part 2 continues with a thorough discussion of all six steps of the tool and ends by presenting a more elaborated version of the tool. In part 3 we reflect upon the generalizability of the tool to any kind of transition program. Finally, part 4 draws a number of conclusions and formulates an agenda for further research.

1. Theoretical framework

1.1 Evaluation in the transitions literature

In the literature on transitions, there are four main strands of theory that dominate the work of scholars (Markard et al., 2012). First, there is the Multi-Level Perspective (MLP), strongly influenced by the work of Geels (2002), that analyzes transitions of socio-technical systems (e.g. the housing, transport or food system) as the consequence of an interplay between a macrolevel (landscape), a mesolevel (regime) and a microlevel (niches). Second, the transition management approach focuses on the role of policy-makers in transition processes and emphasizes the need to set up multistakeholder collaborations to find long-term solutions to solve persistent sustainability problems in socio-technical systems (Loorbach, 2007; Rotmans & Loorbach, 2009). A third strand of literature is known as strategic niche management and concentrates on the importance of developing and nurturing experiments on the microlevel as a way to strengthen system innovation

(14)

(Kemp et al., 1998). Finally, the Technological Innovation Systems (TIS) literature is concerned with informing policy-makers on how to encourage the development and diffusion of new technologies for system change (Bergek et al., 2008; Hekkert et al., 2007).

As Mazur et al. (2014) point out, the attention in the different transitions literature streams for the evaluation of policy interventions in the context of transitions has been limited (see also Bussels et al. (2013). The MLP has mainly focused on developing ideal-typical transition pathways in which the interplay between the three levels happen in different ways and which can explain transitions taking place in the present and the past, but with very limited attention for drawing policy lessons (Geels & Schot, 2007; Van Bree et al., 2010; Verbong & Geels, 2010). Strategic niche management has given some attention to evaluating the success of policy interventions in encouraging the breakthrough of certain niches. Ieromonachou et al. (2004), for instance, have formulated the Strategic Policy Niche Development (SPNM) methodology to explain the success of the introduction of the Durham road access charging scheme and to make recommendations for introducing comparable schemes or other (policy) niches elsewhere. Albeit interesting, the approach remains limited to evaluating policies in stimulating specific niches and also does not evaluate the effect of such a policy on actual system change. The same is true for the TIS literature that focuses on drawing policy lessons for developing specific technologies (e.g. solar panels), but again does not provide evaluations of the results of complete transition programs and their effects on system change (Bergek et al., 2008; Hekkert et al., 2007). Both streams of literature also have limited attention for the transition policy process. The transition management literature then again is limited in its scope for evaluation exactly because it focuses almost exclusively on the evaluation of process aspects. Indeed, in that strand of literature there are long lists of prescriptions on what policy-makers should and should not do to support a transition process (e.g. setting up a transition arena, developing a long-term vision together with the transition stakeholders, working with back- and forecasting, etc.) (Loorbach, 2007). To evaluate whether or not all of those prescriptions are fulfilled, transition management stresses strongly the importance of reflexive learning among the actors involved as a way to achieve ex durante self-evaluation about the process (Bussels et al., 2013). Hence, there is a great deal to learn from the transition management literature when it comes to process evaluation and we will integrate this in our work. The transition management literature brings us however little insight in the evaluation of results.

As such, for as far as transition scholars have given attention to policy evaluation, it has been limited in scope. There are few evaluations of complete transition policy programs. However, with transition governance becoming more and more common as a framework for government policies on sustainable development, there is a growing need for evaluation tools that are able to evaluate complete programs. In that context, there are two noteworthy attempts that we will discuss below. Commonly interesting to both methods is that they try to evaluate to some extent both the policy process and the results of the policy program. Obviously, it is taken into account that the real impact of transitions can only be witnessed in the long run, but the authors note that some first other results should be observed already in the predevelopment and take-off phases, i.e. the first two out of four phases according to the transition literature. The predevelopment phase is mainly characterized by experimentation, which could be witnessed and should be actively encouraged by government. In the take-off phase the first indications of system change should be visible. On the longer term the transition will be speeded up during the acceleration phase when changes in the system accumulate and finally come to a new (dynamic) equilibrium in the stabilization phase (Loorbach & Rotmans, 2006). Such a massive system turnover and the emergence of a new

(15)

equilibrium usually take a very long time and as such it is a reasonable approach only to evaluate the effects of transition policy in the first two phases.

First of all, Mazur et al. (2014) have identified four steps to evaluate policy interventions for system change. More specifically, they analyze which of the ideal-typical transition pathways developed by Geels & Schot (2007) are the most likely to occur for the system under consideration (i.c. the transport system) and they investigate to what extent current policy interventions are encouraging those pathways. Perhaps the most interesting part of their analysis is the fact that they do not attempt to decide upon themselves what a sustainable transport system should look like in 25 years. Rather, they choose to examine the long-term sustainable transport policy objectives that the government has put forward and take those as a given. Next, they investigate whether or not current policy interventions are likely to lead to those policy objectives. That approach avoids a tricky discussion on what sustainability is and what a sustainable system should look like, which is in fact a normative and political debate. For that reason, we will operate in the same manner. However, the four ideal-typical transition pathways are in our opinion too compelling. As a consequence, policy interventions that actually encourage a transition can be evaluated negatively if they do not fit into the pathway that should have been chosen according to the authors. On the other hand, discouraging policy interventions can be legitimized as long as they fit into the preferred pathway. For instance, the attempts of Germany to weaken EU CO2 emission limits for vehicles is considered by

Mazur et al. (2014) not as a German attempt to block the transition towards electrical vehicles, but actually as a logical step in the incremental transformation pathway that Germany is following to adapt its regime. Arguably that way any policy intervention could be considered to be beneficial for transitions. That is why we do not copy that method.

The second four-step evaluation tool was developed by Ros et al. (2006) in the context of the fourth Dutch National Environmental Policy Plan (NMP4) which brought transitions to the forefront in the Netherlands. In that approach, there is an assessment of specific policy programs that aim to stimulate innovating ‘system options’ (strongly comparable to niches) so to make specific systems more sustainable. Although strongly focused on the microlevel, the approach is very much in line with what we need when it comes to evaluating comprehensive policy programs. The tool is specifically interesting for its flexibility in evaluating the programs. Rather than working with predefined transition paths, a number of crucial policy activities are defined that are commonly necessary during the predevelopment phase for any transition pathway taken. Those crucial activities include the development of a long-term vision among the relevant actors, the stimulation of R&D for niches and the set-up of experiments (Ros et al., 2006). Next, it is investigated (using a number of indicators) whether or not those policy interventions decrease regime resistance against the niche innovations under consideration. Decreasing regime resistance indicators (or in other words indicators that show niches are breaking through) can be expected to evolve positively during the take-off phase and/or acceleration phase. The concepts of crucial policy activities and regime resistance are interesting and flexible to use for any kind of transition program evaluation and we will integrate those two steps in our evaluation tool. We will however adapt and broaden the identified crucial activities and indicators for regime resistance, as we believe a number of essential transition policy elements are missing and some of the regime resistance indicators need refinement. Also, as mentioned before, we prefer to take policy objectives as a given and we do not attempt to determine ourselves how a sustainable system should look like as Ros et al. (2006) do.

Summarizing, the development of our evaluation tool in this paper builds upon the integration of several concepts and ideas from the existing transitions literature. One step in our tool will be on process evaluation and will be strongly based on the policy prescriptions from the transition

(16)

management strand of literature, complemented with insights from earlier phases of this project (Bussels et al. 2013; Bussels and Happaerts 2014). From Mazur et al. (2014) we learn not to get lost in the discussion on what a sustainable system looks like and we take over the idea of testing policy interventions against long-term sustainability goals formulated by government itself (possibly together with transition stakeholders) within the system under investigation. Finally, we integrate the concepts of crucial activities and indicators of regime resistance developed by Ros et al. (2006). Nevertheless, we still do not have enough elements to perform a comprehensive evaluation of a policy program aimed at achieving a transition. Most importantly, the matter of causality has not been tackled. Ros et al. (2006) claim that certain policy interventions could have had an influence on specific indicators of regime resistance. Nevertheless, their claims are highly speculative. Causality between policy interventions and results is however of crucial interest to policy-makers. To find an appropriate methodology that can elucidate causal links, we should turn to more traditional environmental evaluation literature.

1.2 Environmental evaluation literature

In the environmental domain there is a very broad literature on evaluation. In this section we will not provide a comprehensive overview of the environmental evaluation literature, but we will touch upon a number of elements that we use in the development of our evaluation tool.

First of all, the literature on environmental evaluation can provide us with arguments that the evaluation of the results of policy programs for transitions is indeed possible. Some transition scholars have neglected the evaluation of results, justified by claiming that evaluating the ‘impact’ of government actions on the sustainability of a socio-technical system is a mission impossible given the long-term framework and complex circumstances (Bussels et al., 2013). Yet, as Ros et al. (2006) or Mazur et al. (2014) have demonstrated, policy interventions aimed at encouraging a transition can have other (more short-term) effects in the predevelopment or take-off phases that are not yet a full proof of impact in the sense of system change, but that are at least showing that the probability of a transition taking place is increasing. Here it is relevant to point out the traditional environmental policy evaluation distinction between impact, outcome and output effects of a policy program (Gysen et al., 2006b):

 The impact effects refer to the final policy objectives, the reason why the policy program is set up in the first place. In the case of transition programs such would be the aim of system change or the effects of the policy program on the sustainability of the system. A concrete example could be the improvement in air quality or even the improved human health that is realized thanks to a transition program on mobility. Typically, transitions need time and system change can only be witnessed on a long term, thereby making it indeed almost impossible to evaluate the impact of transition programs on a short term.

 The outcome effects occur one step earlier and refer to the effects of the policy program on the behavior of target groups within the system. In transition programs those could occur on a short to mid-term basis. As such, we might be able to grasp the first outcome effects in a short-term evaluation. An example could be the increased number of people that buy electrical cars and the resulting increasing market share of electrical vehicles.

(17)

 Finally, the output effects take place in a very early stage and refer to the short-term direct policy actions that are taken in the context of the program. The output effects are easily witnessed and form a first step in investigating causal chains that link a policy program to outcome and final impact effects. For instance, the amount of tax cuts given within a transition program to car users that choose to buy an electrical car would be an output effect.

Now, the three types of effects also fit perfectly into the framework of ‘program theory evaluation’. The idea behind program theory evaluation is that policy-makers have a very clear theory about how their policy program will influence the real world (Crabbé et al., 2006). It is the task of the evaluator to check whether or not that theory makes sense and as such the right approach is taken to achieve the desired results. In our case, we know the transitions literature is the basis of the transition program theory. More specifically, the transition management literature stream provides assumptions on how certain process and policy activities will lead to certain results. Apart from that, there are also assumptions in the minds of the policy-makers and civil servants on how (within the general transition framework) specific policy instruments will have an effect on the behavior of target groups (outcome effects) and finally on the sustainability of the system (impact effects). The mixture of both forms the policy theory (i.e. the transition program theory) which drives the decisions that are taken within the program.

Although the specific program theory will be different in each transition program, there is a common basis to all transition program theories given that all are to a large extent based upon the transitions literature. In all transition program theories, the three types of effects described above are the results of a certain policy input, which are the resources and energy that are put into the transition program. The program should consist out of a number of crucial process aspects and that process should lead again to a number of crucial policy activities to stimulate niches. The specific ways in which that happens, will depend upon the specifics of the program theory, but all transition programs should carry out those activities in some way as those are crucial to the transition framework. The output effects then refer to which extent those ‘crucial policy activities’ took place. Next, the outcome effects are the effects on target groups who are progressively taking up the chosen niches, indicating that the regime resistance is decreasing and niches are breaking through. Finally, the impact is the system change towards a more sustainable character which is the final objective of the program. The transition framework states that in case a transition process is set up in the right way and the right crucial activities are undertaken to stimulate niches, then automatically there should be a decrease in regime resistance against the niches and finally this should lead to sustainable system change. That causal chain is part of each transition program theory and is presented in figure 1.

(18)

Obviously, we would like to check whether or not the theory that forms the basis of the transition program is valid or whether the theory fails to yield (sufficient) results. However, to identify a theory failure, we first need to know whether or not the theory that the policy-makers put forward was converted to practice as foreseen. If not, it is not the theory itself that fails but rather the translation of the theory into actual policy deeds. Given that the policy theory in transition programs is developed from two elements, there are also two possible problems with conversion of the theory that should be investigated. As such, there is triple challenge for the evaluator2_:

1. The evaluator should investigate the possible occurrence of an application deficit. In that case, the transitions literature was not well applied in the development of the policy theory. Essential elements that should be common to any policy theory based upon transitions thinking are missing in reality, which explains why success is impossible. It corresponds to an evaluation of the crucial process activities and crucial policy output

activities that should be present in each transition program.

2. The evaluator should examine the possibility of implementation deficits. Apart from the common elements that should be present in each transition policy theory, the

2_{Typically, there is only a double challenge identified in program theory evaluation, namely to detect on the one hand}

‘implementation deficits’ and on the other hand ‘theory failures’ (Crabbé et al., 2006). Usually, the policy theories evaluated are rather simple theories assuming that one specific policy instrument will lead to a number of specific effects. For instance, a new tax will lead to less polluting behavior and finally will improve environmental quality. In such a case, the implementation deficit would refer to a bad implementation of the specific policy instrument, namely the tax (e.g. no enforcement policy). In transition programs however, the policy theory is much more elaborated. Apart from the effect of very specific policy instruments, there is also the general assumptions about the effects generated by carrying out the crucial process and output prescriptions that are common to all transition programs. Therefore it is also necessary to investigate the possible occurrence of an ‘application deficit’.

Input

Output

Outcome

Impact

Transition

program

Crucial

policy

activities

Decreasing

regime

resistance

System

change

Application &

implementation deficits

Theory failure

Figure 1: Theory aspects common to all transition program theories Policy process

(19)

makers will also at some point make a choice for specific policy instruments. Those specific instruments are needed to carry out the more general crucial policy activities in practice and will be different for each transition program. In other words, it will be evaluated whether the actual policy is implemented in the right way. It corresponds to an evaluation of the policy process during the phase of policy implementation. Here it is not the idea to evaluate the presence of process aspects that should be common to all transition programs, but rather to evaluate the process of implementation of the program specific policy deeds.

3. Finally, there is the possibility of a policy theory failure that should be evaluated. In that case, it is the transition program theory itself that proves to be invalid, at least for the policy problem under consideration. It means the policy-maker’s assumptions and expectations (based for a large part upon the transitions literature, but also on assumptions on specific policy instruments) about the consequences of the chosen policy were wrong. Such corresponds to evaluating the outcome effects and (if possible) the

impact that the process has yielded and also evaluating the causal relation between those latter effects and the crucial policy activities undertaken within the transition program.

As such, it becomes clear why evaluation of both the process and the results yielded by that policy process (in terms of output, outcomes and impact) becomes indispensable for policy-makers. The first is important to reveal implementation deficits and to adjust the program if necessary, while the second is necessary to know when to switch to another general approach in case the transition program theory does not yield sufficient results.

To investigate the application deficit, it is only a matter of monitoring whether or not the crucial process and crucial output elements have been carried out within the transition program and the examination of implementation deficits can be done by thorough process analysis in the phase of implementing the policy instruments. The tracing of a theory failure is probably the most complex. Fortunately, within the context of theory based evaluation, a methodology has been developed which allows us to investigate causality between the crucial government activities in the predevelopment phase and indicators of niches breaking through (or decreasing regime resistance) in the later phases of the transition. Despite the fact that there is no such thing as proving causality, the modus narrandi is a method developed by Gysen et al. (2006b) that attempts to clarify causal mechanisms and shares some similarities with methods like theories of change and realistic evaluation (Blamey & Mackenzie, 2007). The idea behind the method is to reconstruct causal chains and identify in each step both the causal proximity with the next step and the causal contribution of the preceding step in the next one. The causal proximity refers to what extent the first step logically can influence the next one theoretically. The causal contribution is an indication of the strength of that influence in reality (Gysen et al., 2006b). How the modus narrandi works in practice will become clear when applying it to an example in later sections. The modus narrandi methodology has already been used in several Flemish and Belgian environmental and sustainability policy evaluation studies (e.g. Gysen et al (2006a))

Given that in transition programs the impact (system change) can only be investigated on the long term, it is already a big step to elucidate the causal relation between the policy outputs on the one hand and the outcome effects on the other hand and it is exactly our intent to do that by using the

(20)

modus narrandi. Nonetheless, we will not fully neglect the impact effects, as our evaluation tool will start of by reflecting upon the potential impact of the policy program on system change and more specifically on the predetermined policy goals of the program.

1.3 A first glance at the transition program evaluation tool

The combination of the transitions literature with the environmental evaluation literature leads to the development of a new transition program evaluation tool that uses the program theory evaluation approach as an overarching paradigm. The tool is useful for the evaluation of policy programs that are based on the transition literature and that include at the minimum aspects that intend to stimulate niches on the one hand and destabilize the existing regime on the other hand in order to reach a number of long-term system objectives. In Table 2 we introduce briefly the six steps of the evaluation tool that we have developed. In the next part, we will elaborate more thoroughly on all six steps.

Evaluation step

Evaluation criteria Type of evaluation

Transition phase Main source

1 Internal program consistency Potential impact Stabilization Mazur et al. (2014)

2 Crucial process activities Process Predevelopment Transition management

3 Crucial policy activities Output Predevelopment Ros et al. (2006)

4 Policy instruments

implementation Process Predevelopment

Policy theory evaluation 5 Decreasing regime resistance Outcome Take-off & Acceleration Ros et al. (2006) 6 Causality between

program and outcomes

Causality assessment From predevelopment to take-off & acceleration Modus narrandi (Gysen et al., 2006b)

Table 2: Transition program evaluation tool – a first glance

Application deficit? Program theory failure? Implementation deficit? Program theory Failure?

(21)

2. The transition program evaluation tool elaborated

In this part we elaborate on the transition program evaluation tool. We explain one by one all five steps of the tool. To make the different steps more concrete, we sometimes provide hypothetical examples that could be investigated in the context of the existing Flemish transition programs.

2.1 Step 1: Evaluation of internal program consistency

In step 1 we evaluate the internal consistency of the program. We argue there have to be at least three elements to a transition program: long-term policy objectives for the system in transition; the stimulation of niches and finally the active destabilization of the current system in order to enhance the capacity of niches to take over or profoundly change the regime. In this phase of the evaluation, the evaluator should be able to find easily the long-term policy objectives of the program and the exact niches that are encouraged, so to be able to evaluate whether the chosen niches are consistent with the long-term objectives. Whether or not the niches are stimulated in the right way and whether there are also aspects of regime destabilization so to enhance the possibilities for breakthrough of the niches will be tackled later on in step 3.

First, as for long-term policy objectives, we stated before that the paradigm of transitions thinking is used to decide upon policy interventions in socio-technical systems so to solve the persistent problems of sustainable development. Such a description implies that the policy-makers at the least should have an idea about those persistent problems and have identified a number of long-term policy objectives that point towards a diminishment of those problems within the system. As we mentioned before, we will take those objectives as a given and we will not try to define ourselves what a sustainable system should look like. As such, the evaluator should only look for those objectives in policy documents or policy discourse. Preferably, the objectives can be found in documents written in the context of the transition program itself by civil servants together with the transition stakeholders. If not, the evaluator can look for policy documents outside of the transition program but within the policy domains related to the system. For instance, in the case of the mobility transition, the website of the Flemish government states that our current mobility system is not sustainable, because of problems with traffic jams, safety issues, air pollution and inequality issues in mobility (Vlaamse overheid, 2012c). It could be reasonably assumed that solving those sustainability problems are the long-term objectives of the program and indeed this is supported by the fact that such long-term objectives for the mobility system can also be found in the Flemish strategy for sustainable development (Vlaamse overheid, 2011).

Next, as for the stimulation of niches, transition programs typically make use of ‘test cases’ or ‘experimental projects’ to help ‘niches’ develop. No matter whether called test cases, pilots, niches or experiments, the projects that are chosen within the transition program have to be evaluated in terms of consistency with the chosen policy aims. It means that their potential impact should be such that they actually contribute to the long-term transition objectives. Indeed, in case the chosen experiments can in no way be related to the long-term objectives, the complete evaluation becomes

(22)

meaningless and for that reason the external or internal evaluator should first perform the consistency check in the first step of the evaluation. As explained before, the idea is not to evaluate the actual impact of the experiments set up, but rather the potential impact on the system. In other words, we ask ourselves the question whether the chosen niches could contribute to some extent to a transition (defined as reaching the long-term objectives). To evaluate the potential impact of the program experiments, a literature study should be performed by the evaluator on the niches. The hypothesis that chosen niches might contribute to the long-term objectives does not have to be ascertained, but should be at least supported by some scientific sources. If not, we already know from the start that the policy theory is false when it comes to the assumption of causality between the transition program and the final impact and we know a theory failure will occur.

In the case of mobility in Flanders, test cases are set up concerning for instance electrical vehicles, road charging and the use of a mobility budget for employees (Vlaamse overheid, 2012c). Obviously, not all those niches will have an influence on all the long-term policy objectives. An evaluator could therefore for instance decide to investigate the literature on electrical vehicles only for their effect on air pollution and safety issues as a potential impact is the most likely there.

Yet, in case the evaluator is analyzing the total program and investigating all niches, it does become possible to see whether all of the long-term objectives are taken seriously by examining whether all of the objectives could be influenced by at least one of the chosen experiments. It also has the additional benefit of checking compensating effects of interrelated niches. For example, if the introduction of the electrical cars turns out to have negative effects on safety (because of silent engines decreasing alertness among pedestrians), those problems might be compensated by positive effects from other niches such as road charging that decrease the total number of cars on the roads so that total safety is indeed expected to increase thanks to the transition program. In reality of course the evaluation of isolated consequences of specific niches on the long-term objectives is sometimes inevitable in case of limited resources and time.

To summarize, Table 3 shows the three questions an evaluator should pose so to investigate the internal consistency of a transition program and the sources to look for answers. The long term policy objectives and the niches that are nurtured in the transition program can be found in the related policy documents. Whether or not the chosen niches have the potential to contribute to the long-term policy aims can be investigated by performing a scientific literature study. If, according to science, the expected impact of the chosen niches turns out to be consistently negative for the chosen policy objectives, then the rest of the evaluation becomes obsolete.

(23)

Questions needed for an evaluation of internal program consistency

Sources for answers

What are the long-term policy objectives of the transition program?

Policy documents and discourse

What are the chosen niches in the transition program? Policy documents What is the potential impact of the chosen niches and are those

consistent the long-term policy objectives of the program?

Scientific literature review

(24)

2.2 Step 2: Evaluation of crucial process activities

As we mentioned before, there exist quite some transition process prescription lists, specifically in the transition management strand of literature. A transition program that is based upon a good application of the transition framework should indeed contain a number of process characteristics that differ from more traditional policy processes. If not, an application deficit occurs and one cannot expect any serious outcome or impact effects of the program (unless because of coincidental circumstances not related to the transition process). Yet, extensive process prescription lists are often overgeneralizing and ignoring the specifics of real life transition programs, and they could be discouraging to meet for a public servant trying to coordinate a transition process. Therefore, we distinguish between the crucial process activities that can be influenced by the public servant him-/herself on the one hand and favorable process conditions that are the consequence of the actions of other actors or of external (e.g. landscape) circumstances on the other hand. Table 4 provides an overview of the crucial process activities and favorable process conditions that we identified from different sources in the literature. Obviously, we are aware that the choices we make could be contested and that it could be possible to select other crucial process activities and process conditions. We believe that our application of the evaluation tool on existing Flemish transition processes in the future can provide a robustness test of the list and could lead to an improvement of Table 4.

Note that the crucial process activities in principle are expected to be carried out at the level of the program. For instance, the ideal is that actors within a ‘transition arena’ come together to reflect on the long-term aims of the full program and also decide upon which niches to stimulate within that program. Yet, in some circumstances some niches or experiments might be so extensive, that ‘transition arenas’ are (also) formed around the niches rather than (only) around the entire system, still however acting from a joined long-term perspective on the system in which context the experiment is launched. In the latter case, applying the tool to those program parts instead of on the full transition program might even be more appropriate.

(25)

Crucial process activities Favorable process conditions Setup of a ‘transition arena’ for a specific

socio-technical system

Societal and political sense of urgency on the sustainability problems of the system Development of a long-term vision on the

system within the arena

High-level political support for the transition program

A multi-actor, multi-level and multi-domain approach is chosen, but with a single point of

decision

Sufficient qualitative and quantitative capacity within the administration

Setup of learning moments Transparency, trust and respect among the actors

Shared ownership

Open-minded actors are involved that are capable of thinking out of the box

Table 4: Crucial process activities and favorable process conditions

Sources: (Bussels & Happaerts, 2014; Bussels et al., 2013; Loorbach, 2007; Roorda & Bosman, 2014; Ros et al., 2006)3

Given that our tool is in the first place meant for self-evaluation by civil servants coordinating transition programs, the evaluation should focus on the process activities that they can influence and we focus on the development of clear indicators for those activities below. The (un)favorable process conditions can be interesting to analyze in order to understand better why some of the crucial process activities and/or crucial output activities (see step 3) were (not) carried out. For instance, Bussels & Happaerts point at the importance of process conditions such as high level political support and the capacity within the administration to explain the individual discretionary space of civil servants to act in transition programs and to set up learning moments in particular (Bussels & Happaerts, 2014). Other examples could include how the lack of trust between actors within the arena could block the development of a widely supported long-term vision on which all actors agree. Further, limited political support could for example explain why no experimental test cases were set up (a crucial policy activity in step 3). As such, the fact that certain crucial process and/or policy activities were not carried out indicate an application deficit of the transition framework, while the unfavorable process conditions might be the causes why the application deficit occurred.

To evaluate whether or not the crucial process activities were undertaken (by the coordinating civil servants), we should have at our disposal a number of indicators. We use the term ‘indicator’ here in a wide sense as a fact that can be witnessed and gives an indication about whether or not a crucial process activity has taken place, rather than strictly as a quantitative measure that can be monitored through time. Table 5 provides a number of indicators for each process activity that can easily be

3

Some of these crucial process activities and favorable process conditions also formed part of a process checklist developed by the Flemish administration in the context of the transition policy.

(26)

witnessed by the evaluator so to assess whether the activity was undertaken. Again, using the tool in the evaluation of real life transition processes might still improve the list of chosen indicators.

Crucial process activities Indicators

Setup of a ‘transition arena’ for a specific socio-technical system

A committee, steering group, arena or another group of actors has officially been set up and they come together on a regular basis to deliberate on the future of the system under consideration.

Development of a long-term vision on the system within the

arena

The group has produced an official long-term vision on the socio-technical system, preferably including the identification of long-term policy objectives for the system, possible transition pathways towards those objectives and needed experiments. All actors in the arena endorse the document.

A multi-actor, multi-level and multi-domain approach is chosen, but with a single point

of coordination

The group consists out of a multitude of actors that are relevant for the system, both regime players and innovators; from all sides of the quadruple helix (government, business, science & civil society) and from multiple (policy) domains that are related to the system. There is however one actor (possibly a civil servant) clearly in charge of coordinating the process.

Setup of learning moments There are official and/or unofficial learning moments during which the actors reflect upon their actions and explicitly decide together to adapt the policy program if necessary.

Table 5: Indicators for crucial transition program process activities

Sources: (Bussels & Happaerts, 2014; Bussels et al., 2013; Loorbach, 2007; Roorda & Bosman, 2014; Ros et al., 2006)

For as much as possible, indicators were chosen that could be objectively witnessed. The evaluator could use official documents such as founding documents of the arena, long-term vision statements, lists of participants and minutes of meetings. In the case of an external evaluation, those sources to some extent will have to be combined with information from interviews with involved actors. Learning moments for instance could be unofficial and it is rather unlikely that for each change in the policy program there will be an official referral to learning moments as a justification. In case the evaluator would also be interested in identifying the (un)favorable process conditions that explain

why certain process activities were (not) undertaken, then information from interviews becomes indispensable. Such is however not necessarily the case when the tool is used for self-evaluation by civil servants. If they are involved in the process themselves and are in constant interaction with the other involved actors, they have probably a very good view on the process activities and conditions themselves.

(27)

2.3 Step 3: Evaluation of crucial policy activities

Once a good application of the transitions literature has been realized in terms of process aspects, the actors in the transition arena should decide upon a number of policy actions to undertake. Those policy actions lead to the first results of the process that can be witnessed in the short run, during the predevelopment phase of the transition. Those results are the output effects and are crucial in stimulating the niches and destabilizing the regime enabling the niches to break through and trigger system change. Those crucial output effects are mainly related to creating space as a public authority for private players to develop new and improve existing niches and to create the capacity to let the niches compete with the existing regime options. In case the crucial policy activities that should be common to any transition program do not take place, we consider that to be a second form of application deficit, next to a bad application of the transition framework in terms of crucial process activities.

Table 6 provides an overview of the crucial policy activities that should be carried out in the predevelopment phase from the side of the government. Obviously, the attitude and actions of other (non-governmental) players could be at least as important. We focus however on the policy activities, given that we want to evaluate transition policy programs. We implicitly assume here that the actions of private players are influenced by the policy actions, as is done in the transition management literature or for that matter in any policy program theory. We claim there are four crucial activities that should be carried out by government officials in a transition process. For each of the four activities we provide examples of how they could be accomplished by mentioning possible output effects that could be witnessed by the evaluator. The lists of possible output effects are obviously not exhaustive nor should all of the outputs be realized. It is the responsibility of the ‘arena’ to decide which of the policy activities are the most useful in the context of the specific transition program. Yet achieving at least some clear outputs in all four categories of crucial policy activities is essential for a good application of the transition framework. Also, further research and an application of the tool on existing transition programs could reveal other crucial activities that might have to be integrated in this evaluation step in the future.

First of all, stimulation of research and development (R&D) is absolutely essential for the niches to break through, from fundamental research up until the launching of actual test cases for end users (Groot Koerkamp, 2014; Ros et al., 2006). Niches are by definition not yet as efficient, comfortable or technically optimal as regime options. To catch up, government science policy should favor niches and private actors should be stimulated to invest in R&D on the niches instead of on the regime. As such, not only are the niches nurtured, but the regime is also destabilized by changing the R&D direction. Such can be done for instance by a subsidy reform, providing more R&D subsidies for private companies that investigate the possibilities of electrical vehicles and at the same time cutting back on subsidies for more traditional R&D in the automobile industry. The support for R&D does not always have to be approached from a technical and/or economic perspective. For instance, more subsidies would be welcome to investigate the potential of new ideas on mobility such as car sharing or the delivery of packages by individual car users, no matter whether in the end those ideas are commercialized by private companies or instead are launched as societal initiatives by individual groups of citizens and/or civil society organizations.

Second, the playing field should be levelled and preferably should be turned in favor of the niches and at the expense of the regime options (Roorda & Bosman, 2014). This crucial activity mainly refers to changing the environment (e.g. the market) in which the chosen niches compete with the system options by using the regulatory and fiscal power of public authorities. An example could be to tax

(28)

more heavily the use of fossil fuels and to use that revenue to subsidize the purchase of electrical vehicles (for an overview on the use of tax instruments in transition programs, see Vanswijgenhoven (2014)). Also, one could adapt rules so that for instance providers of niches could (temporarily) benefit from lighter administrative procedures. Again, note that although terms as ‘markets’, ‘providers’ or ‘competition between niches and system options’ may have an economic flavor, many of the examples can be applied to transitions from a social perspective. For instance, a shift away from meat consumption and towards vegetarian practices could be stimulated by a tax on meat or the use of bicycles as a dominant mode of transport in the city can be nurtured by stringent regulations on car use. Nevertheless, some of the examples given are more suitable for technical-economic transitions while others (e.g. information campaigns) might be more useful in case of social transitions.

A third crucial activity is to shift government resources. Public authorities are important players simply because of the financial and human resources they have access to (Geels, 2013; Roorda & Bosman, 2014).4

Public procurements for government car fleets, for instance, have the potential of being an important instrument in boosting the electrical car and investments in public infrastructure could be directed towards niches, e.g. by adapting the electricity network in favor of renewables. As for social transitions, investments in bicycle and pedestrian infrastructure could discourage car use in cities.

Finally, a fourth crucial activity is a non-activity. It is namely essential to avoid any active reinforcement of the existing regime. It would for instance be pointless to increase subsidies for electrical car use and in the meantime decrease taxes on fossil fuels. For the first three crucial activities, we mentioned they should be focused on both stimulating the niches and destabilizing the regime. While in some cases it is reasonable to accept that destabilization might be difficult for political reasons, any reinforcement of the regime should really be avoided. As for the evaluation, one could argue for that reason that it might be acceptable if for instance there is no shift of subsidies, but subsidies for niches just come on top of existing subsidies for regime options. Yet, any new subsidies for regime options are certainly harmful as they undo the effects of the niche support and increase the likelihood of a system lock-in.

To summarize, from an evaluation point of view, we argue all four activities should be carried out to avoid an application deficit of the transition theory. Such can be evaluated by looking at the outputs for some of the concrete examples of policy interventions that we mentioned for the first three activities. Obviously, not all possible examples have to be undertaken as not all will be essential or even useful for each possible transition. The examples are meant not only to nurture niches but also to be directed at destabilizing the regime. The presence of destabilization measures is certainly to be preferred, but in some cases it might be reasonable to not consider it as an absolute criterion for the evaluation of specific policy activities. The fourth (non-)activity is however stringent as any active reinforcement of the regime should certainly be avoided.

4

EVALUATING LONG-TERM TRANSITION PROGRAMS ON A SHORT-TERM BASIS Towards a six-step transition program evaluation tool