2. The transition program evaluation tool elaborated
2.4 Step 4: Evaluation of policy instruments implementation
In the case of an application deficit, the transition program policy theory is not well thought through as crucial elements of the transition framework are not incorporated within the theory and as such are not present in the transition program. Another problem could occur however when the general transition framework has to be transformed into concrete policy deeds. Such is exactly what happens when the policy-makers make a choice on which of the examples of possible policy instruments presented in table 5 are chosen so to realize the four crucial policy activities. While we might be able to rule out an application deficit as we can witness outputs for all four crucial activities, the way the chosen instruments are implemented could still be problematic. To give just one example, there might be a clear tax shift away from the niches and towards regime options, but in case there is no enforcement policy we can still not expect any outcome or impact effects to occur. For that reason, it is pivotal to track implementation deficits for the chosen policy instruments.
The investigation of implementation deficits has a long history in policy research (Lipsky, 1979; Pressman & Wildavsky, 1973) and traditionally forms part of any program theory evaluation (Crabbé et al., 2006). The central idea behind implementation deficits is that the ‘spirit’ of policy as it has been formulated is not brought into practice in the real world by what Lipsky calls “street-level bureaucrats” (Blackmore, 2001; Lipsky, 1979). The street-level bureaucrats limit themselves to those actions needed to create an “apparent rule-confirmity” (Blackmore, 2001). In others words, the policy formulated by high level policy-makers is implemented in the letter but not in the spirit by the lower level bureaucrats. Lipsky (1979) does not put the blame at the individual bureaucrat. On the contrary, he rather sees the street-level bureaucrats as well-intentioned, even idealistic, individuals that are the victims a system that overwhelms them with huge workloads and ever insufficient resources. The bureaucrats as such have no other option then to develop coping mechanisms such as limiting client demands and reinterpreting and minimizing given objectives and by doing so trying to do the best they can, given the circumstances (Blackmore, 2001; Lipsky, 1979).
Other scholars have pointed at other reasons why implementation is performed badly by street- level bureaucrats. In his work on the restrictions on benefit regimes for job seekers in the UK, Martin Blackmore (2001) mentions active resistance of street-level bureaucrats supposed to take sanctions against job seekers that cannot demonstrate an active job seeking behavior sufficiently. In the
20 | Tom Creten, Sander Happaerts & Kris Bachus
context of high unemployment end the end of the eighties, many bureaucrats considered the sanctions to be unfair and implemented stricter policy regulations in a minimal way. Also, the author points at additional problems making implementation hard such as unclear measures on what is to be considered active search behavior and difficulties for job seekers to prove such behavior
(Blackmore, 2001). In line with the argument of resistance, Pressman and Wildavsky (1973) note that the in many cases policy implementation is to be carried out by dozens of bureaucratic institutions, that all have to implement one part of the policy, inevitably leading to inadequate implementation even if only a few of those institutions oppose or misinterpret the formulated policy and/or make honest mistakes. We would argue this problem even starts at the political level. An example of the latter could be if there is a political decision to implement a carbon tax, but the tax is set much too low to have an influence, for instance because of political compromise or because the bureaucratic institution in charge of identifying the ideal tax level has made mistakes in its calculations of price elasticity.
No matter the reason for implementation deficits occurring, the question is how to open the ‘black box’ of the implementation process. Indeed, we need to turn back to a process evaluation in this step. We do not focus here on the process aspects that are crucial to apply well the transition literature in the formulation of the policy theory as we did in step 2, but rather we look at the specific stage of implementation of the crucial policy activities that were determined in step 3. Yet, while in step 3 we simply checked whether or not the outputs occurred (quantitatively), in this fourth evaluation step we will evaluate (in a qualitative way) the specific process of implementation of the chosen policy instruments to perform the crucial activities.
The specific way to do such an implementation process evaluation will strongly depend upon the chosen niches to stimulate and even more on the specific policy instruments that were selected to carry out the crucial activities. Nonetheless, a number of common evaluation questions can be determined that always should be posed to investigate implementation deficits. Those questions are summarized in table 6.To make things concrete, we give hypothetical answers for two possible policy instruments chosen in specific transition programs. One is for a tax shift away from electrical vehicles and towards cars with a fuel engine and a second is for the amount of subsidies given to R&D on renewables.
Evaluation question Example for tax shift Example for R&D subsidies
What is the “spirit” or the aim of the policy instrument?
To make the purchase and/or use of an electrical vehicle relatively cheaper
To increase the knowledge on renewable energy production
What are the essential elements for good implementation?
- High enough tax to have influence on the relative price
- The tax should be enforced - Fines have to be high
enough to deter non- payment
- …
- High enough subsidies to provoke or increase private R&D
- Controls on what subsidies are used for
- Well written eligibility criteria
- …
Are those elements present? Yes/no Yes/no
If not, wherein lies the problem?
- Political disagreement on the tax level
- Bureaucratic unwillingness to implement
- Insufficient resources for enforcement
- …
- Budgetary constraints making higher subsidies and controls impossible - Insufficient training for
bureaucrats in writing good eligibility criteria
- …
Table 7: Evaluation of policy instruments implementation
Again, the evaluation questions in table 6 are probably to be improved and extended during the testing of our evaluation tool. For real-life transition programs, an external evaluator that wants to answer the evaluation questions in this step will be forced again to seek his/her information at the source, i.e. with the street-level bureaucrats that are implementing the policy instruments. For that reason, again we should emphasize that an honest self-evaluation by the civil servants coordinating the transition programs is to be preferred, as they have at their immediate disposal most of the necessary information given their strong own involvement in the policy (implementation) process.