The evaluation of tools consists of two parts: 1) tool evaluation by means of the evaluation framework on the basis of existing knowledge and experiences, and 2) an elaboration of these desk-study results by means of drawing lessons from actually applying the tools to a real policy case during the case study (discussed in §3.3).
The (preliminary) tool evaluation is done by means of the evaluation framework and a
common reporting format on the basis of existing information and experiences with respect to the tools. As indicated before, the tool-by-tool evaluation comprises an evaluation with respect to the ability of tools to support various policy processes and to address (crosscutting) aspects of sustainable development. In addition, a number of operational aspects need to be specified and other relevant issues to be reported.
3.2.1 How to evaluate which policy processes are supported?
The policy processes that are distinguished in SustainabilityA-Test are given in Table 3.1on page 32. The scoring used for how well a tool can support certain policy processes is given in Table 3.6.
Table 3.6: Overview of the scoring categories that can be applied in SustainabilityA-Test
Score Value 0 Undetermined 1 Particularly well 2 Fairly well 3 Unsuitable 4 Unknown
5 Unknown – subject of further work beyond SustainabilityA-Test 6 Unknown – subject of further work within SustainabilityA-Test
Some tools can be regarded as processes themselves (i.e. not a one-off event, but a process that runs parallel to various stages in policy making, like e.g. Strategic Impact Assessment). In such cases, the score ‘particularly well’ is reserved for the policy process in which the tool normally ‘starts’. The policy processes that run parallel with the tool application should all be given the score ‘fairly well’.
When tools can be used in different policy processes, fulfilling different functions, each tool function must be separately evaluated: one tool evaluation sheet will be made for each
function of the tool. There should be a different name assigned to each function of the tool, to prevent confusion.
3.2.2 How to evaluate the ability to address (crosscutting) aspects of
sustainable development?
The (crosscutting) aspects of sustainable development that can be addressed by a tool are specified in Table 3.2 and Table 3.3 on page 34. How well aspects can be addressed by a tool is specified conform Table 3.6.
The tool teams have mainly scored the coverage of main categories of these aspects. It appeared so far to be not necessary to use the detailed lists of (crosscutting) aspects for the tool evaluation.
Only the coverage of tools of the (crosscutting) aspects of sustainable development has been evaluated by the tool teams. The coverage of aspects related to principles of sustainable development can be extracted from that evaluation.
3.2.3 How to evaluate the operational aspects?
The operational aspects, as specified in Table 3.5 on page 36, are reported in various quantities (e.g. costs in euros, manpower needs in man months) or scoring categories (e.g. user friendliness in high/medium/low categories). The following scoring system should be used.
Table 3.7: Scoring to be applied to the operational aspects
Operational aspect Score 1 Score 2 Score 3
INPUT / OUTPUT
Costs (in monetary terms) for applying the tool Costs in euros*
Manpower needs for applying the tool Manpower needs in man-months* Time needs (for making the assessment) Number of days*
Data needs Low Medium High
Data availability High Medium Low
Technical equipment required Low Medium High COMPLEXITY / TRANSPARENCY
Complexity (of the tool itself) Low Medium High
Transparency High Medium Low
User friendliness High Medium Low
Reliability High Medium Low
Uncertainty Yes / No TOOL CHARACTERISTICS
Experience (with applying the tool) High Medium Low Mandatory usage Yes / No (and specify what obligation)
Time before results become outdate Years
Time scale / time horizon Retrospective: year (e.g. 1990) Prospective: year (e.g. 2010)
Geographical coverage Number of countries and group of countries (e.g. EU25, oldEU, newEU, World)
Geographical resolution NUTS1, NUTS2, NUTS3, EU-regions, countries, EU, World etc. Specificity of results Sectors (agri, transport, energy, etc), socio-economic groups etc *) In first instance, costs, manpower needs and time needs will be specified in euros, man months and days
respectively. This data can in a later stadium of the project be recalculated to high, medium and low categories, which allows for easy comparison across tools.
Determining whether a tool is for example user-friendly is not straightforward and depends, amongst others, on the user. It is the task of the tool teams to make an expert judgement, and to explain why a certain judgement has been made. The I&S team verifies objectivity in co-
ordination with the tool teams. If necessary, the scores and arguments are discussed with all tool team leaders present.
3.2.4 Other relevant tool characteristics
The evaluation is not limited to tool characteristics enquired with the evaluation framework. After all, there might be relevant tool characteristics that do not necessarily come to the surface when only looking at the evaluation criteria as described above. Each tool team therefore also reports those tool characteristics not (yet) tackled with the evaluation framework.
3.2.5 Reporting format
The tool inventory and the results of the evaluation are reported by means of the tool overview and evaluation papers. In addition, a tool information sheet has been created for each tool, summarising the main findings and providing an overview of the evaluation scores given to each tool for each evaluation category. These sheets will be used as reference sheets in the handbook.