3.
Reflections on the use of the tool
It was our aim to develop an evaluation tool that is widely applicable to any kind of transition program. We have provided hypothetical examples from existing transition programs within systems as diverse as energy, food and mobility. We also draw attention upon the fact that it is possible to apply the tool to both transition programs that are focused upon nurturing technological niches and to programs that mainly work with social niches.
Nevertheless, a transition program has to meet at least a number of conditions for it to be evaluated with the tool developed in this paper. We mentioned earlier that we consider a transition program to consist out of at least three elements: LT policy objectives for the system under consideration, the nurturing of niches and the destabilization of the regime. While the destabilization of the regime forms part of the evaluation, more specifically in step 3, the presence of long-term system objectives and the encouragement of niches (in form of experiments, test cases, etc.) can be witnessed immediately at the start. Therefore it is also no surprise we start off in step 1 by investigating those long-term aims and the niches. In case it would be impossible for the evaluator to identify the long-term objectives and niches of the program, the rest of the evaluation simply becomes impossible, as those two elements form the basis of our evaluation tool. Also, in case the consistency of the program in terms of nurturing the niches that indeed can contribute to the objectives appears to be problematic, the rest of the evaluation becomes obsolete and there is little use in evaluating whether the right process activities and policy actions occur and whether there is a good implementation of policy if such would only be harmful or irrelevant for the transition to take place. Furthermore, any positive evolution of the outcome indicators would then be in spite of rather than thanks to the transition program. This is not the case for later steps. For instance, even if an application or implementation deficit (partly) occurs, it could still be relevant to check the evolution of outcome indicators and the role of the program therein in the later steps of the evaluation tool.
The fact that the program should be set up in order to meet long-term sustainability goals for a system, also means that indeed the program has to fit within a clear socio-technical system that provides an unambiguous service to society, such as the energy system, mobility system, housing system, care system, etc. In Flanders, for instance, policy programs adopted in a transition context cannot always be placed within such a clear system as some programs are rather set up to meet one or several specific goals by working on those goals within different systems (or policy domains). A fine example of that is the transition program on child poverty (Vlaamse overheid, 2012b). Obviously, fighting poverty is an extremely important policy goal and the transversal approach that emphasizes the fight against poverty has to be fought in every policy domain should be welcomed. Poverty, however, is obviously not a socio-technical system that provides a service to society and that fact makes it very difficult to apply our evaluation tool to such a policy program. With one glance at the tool, it becomes immediately clear that the problems are manifold. For instance, the question immediately rises what are precisely the niches that we have to evaluate in terms of consistency with the long-term objectives and that would have to break through? A number of policy initiatives are indeed identified within the program on child poverty, but they are not social or technological innovative niches that have to compete with a regime (Vlaamse overheid, 2012b). For instance, the automatic granting of benefits simply has to be carried out by policy-makers, but it is clearly not a
32 | Tom Creten, Sander Happaerts & Kris Bachus
niche that has to break through within a system and for which regime destabilization would be necessary. Almost none of the crucial policy activities in step 3 nor the indicators of decreasing regime resistance in step 5 would be fit for evaluating such a program. In terms of the process however, a policy program that is aimed to reach certain specific objectives but does not relate to a specific system, might still benefit from the typical transition approach. For instance, it might indeed be very useful to involve many actors, take a multi-domain approach and set up learning moments in a program on fighting (child) poverty. Also, an evaluation of the process during the implementation phase of policy instruments could be useful in any policy evaluation.
As such, we do not state that our transition evaluation tool is totally useless for some of the Flemish transition programs, but we do warn that, in terms of evaluating the results, the tool can best be applied on those programs that at least target sustainability issues within clear socio- technical systems and that encourage niches and experiments to do so. Otherwise, they are regular policy making and no transition programs, hence they should not be labeled ‘transition programs’.
4.
Research agenda
The tool developed in this paper is a contribution to the existing academic literature on the evaluation of transitions, but is also meant to be used in the future for self-evaluation by civil servants coordinating transition projects in Flanders or beyond. In a first phase, however, the tool will be applied by the TRADO research team to a number of existing Flemish transition programs so to test and refine the current tool and make it even more practical for civil servants.
In particular, we aim to use the tool for an evaluation of the transition programs ‘Smart mobility’ and ‘Flanders’ Care’. Both transition programs have quite a different nature. For mobility, we already named a number of experiments and other aspects of the program in this paper as examples of how we would carry out our evaluation steps. It is a rather clear program with a number of well-defined experiments that are meant to increase the sustainability of the mobility system. Examples include experiments with electrical vehicles, road charging and a mobility budget for employees (Vlaamse overheid, 2012c). ‘Flanders’ Care’ is a transition program aimed at controlling sustainability issues in the Flemish health care system. Interestingly, the type of sustainability problem is quite different in that program as it is mainly focused on maintaining (through innovation) the quality of the health care system on the long term in the context of increasing financial costs caused by an ageing society (Vlaamse overheid, 2012a). That is a very different kind of sustainability issue than in most transition programs, which usually aim at ecological and sometimes social long-term objectives. Also, the Flanders’ Care program is much more diffuse with many projects and less clearly defined borders. As such, the application of the evaluation tool on the Flanders’ Care program will be a challenge, but also a very good test for the generalizability and practical use of our tool for real life transition programs that use the transition approach in a perhaps less theoretically pure way.
Our further research should provide additional insights that can improve each step of the tool. For instance, we expect it may turn out that certain crucial process and policy activities are not as relevant as claimed in this paper and some others may have been neglected so far. Although we have paid specific attention to the intention that our tool would be applicable for both technological and more social niches, we also expect that it may turn out that in some steps we still have a slightly economic-technological bias as a consequence of the fact that still the large bulk of the literature
deals with those kinds of transitions and niches. Specifically the application of our tool to the encouragement of ‘social’ niches such as the mobility budget or data sharing in the health care system may provide interesting additional understandings on that issue as well. Also, it might be an interesting topic of research to investigate the possibilities of using the evaluation tool for an assessment of the role of other partners in the transition than the government or even to evaluate the functioning of the total transition arena. Finally, we have focused upon niches in this paper as most of the Flemish transition programs are indeed based upon the idea of nurturing niches. Yet, the specific examples throughout the paper already clarify that ‘niches’ can be broadly defined and still the evaluation tool could be applicable. In the follow-up of this project, it could also be investigated how broad the type of transition programs is that we can evaluate with our tool, and which elements of a transition program are indeed essential for evaluation. In particular, the application of the tool on the issue of road charging in the mobility transition might be interesting, given that one could consider this to be an act of regime destabilization rather than a niche that has to be nurtured in the pure sense.
| 34