6.1 Relevance, efficiency and effectiveness of Eurostars
6.1.4 Efficiency of programme implementation
Implementation efficiency with respect to the virtual common pot
The Eurostars central funding mechanism is called a virtual common pot. Under a (real) common pot the contributions from all countries are put into one account to finance all projects starting with the best (based on the ex-ante evaluation) and ending with the marginal project for which funding is still available. Under a virtual common pot, an additional restriction is that the financial contribution of country x is used only for applicants from country x. If the financial contribution from country x is exhausted, the project is not financed, and the next project in the ranking list gets funded. For a collaborative approach, this implies that there are financial needs to be available from all home countries of the project applicants.
Box 6: What is effectiveness and efficiency?
Effectiveness: The extent to which the intervention's objectives were achieved,
or are expected to be achieved, taking into account their relative importance. Also used as an aggregate measure of (or judgment about) the merit or worth of an activity, i.e. the extent to which an intervention has attained, or is expected to attain, its major relevant objectives in a sustainable fashion.
Efficiency: A measure of how economically resources/inputs (funds, expertise,
time, etc.) are converted into results.
Such a system gives an incentive to individual countries to provide enough money to finance the applicants from the countries. This is even more the case as the EU also puts additional money into the pot. But there is also a drawback. Suppose SMEs from countries "x" and "y" propose a joint project which gets an excellent ex-ante assessment. The project will only receive funding if there are sufficient funds from both countries. If one of the countries does not provide enough funding (“pot” of country x or y is exhausted), the whole project will not receive funding. Hence the
November 2014 Page 53 of 139 relative size of each country’s contribution (compared to the relative size of the country or the demand by applicants from that country) to the common pot becomes crucial for the functioning of this type of funding system. This is highlighted in section 6.1.3 above.
Figure 6-10 provides some insight into the working of the incentive structure for national governments to provide a sufficient amount of national money for the programme. This figure shows, first, the initially agreed size of the national budget contributions for Eurostars, second, the extension of the national budget in the course of the programme. This figure also gives national money unspent with respect to national budget cuts of the Eurostars budget.
The incentive effect of the virtual common pot allocation is demonstrated by Figure
6-10, as several countries extended their initial budget quite significantly. Germany,
Denmark, Sweden or the Netherlands increased their contribution by 60% or more. On the other hand, there are also a significant number of participating countries which did not contribute the initially agreed amount to the common pot. A small number of Member States were unable to realise their original commitment, probably due to the impacts of the financial crises the on the national R&D budgets of Italy, Greece, Cyprus, Spain and Ireland.
Figure 6-10: Initially earmarked budget, budget extensions and unspent budgets (in million Euro)
Source: ESE (2014)
To highlight the working of the virtual pot, the expert group compared the allocation of funding using the virtual common pot system, with a hypothetical common pot
system (Figure 6-11). This figure shows that the allocation of projects following the
virtual common pot system got more closely towards the first best allocation of the real common pot starting from cut-off 1 onwards. However, the deviation of the virtual common pot from the hypothetical real common pot increased following cut-off 3. In an increasing number of cases, funding requests from applications exceeds the
November 2014 Page 54 of 139 available national earmarked funding for Eurostars. In addition, countries are not in a position to enlarge their contribution in parallel to the request for Eurostars funding from their partners. Here, we should also note that the economic and financial crisis hit the public research budget considerably, so that some countries were even not able to finance the initial agreed earmarked budgets. Therefore the crisis seems to limit the functioning of the common pot of Eurostars.
It can be concluded from this analysis that: (1) for the vast majority of projects, it does not matter whether a real common pot or a virtual common pot rule was applied; (2) Eurostars has made considerable progress in cut-offs 2 and 3 towards a purely quality-based, real-pot allocation. In more recent years, the restriction from the availability of fund became more and more binding.
Figure 6-11: Working of Eurostars virtual common pot vs. a real common pot
Explanation:
Under a real common pot allocation system, all projects would be financed according to the ranking list. Any deviations from the
ranking list would
represent deviations
from the real common pot approach due to the use of a virtual common pot system. In
Figure 6-11, the
straight line represents
funding allocation
according to the
ranking list (i.e. the
real common pot
approach). The dashed, coloured lines represent the cumulative number
of projects actually
funded according to
their ranking and
available contribution of member states.
November 2014 Page 55 of 139 Instead of raw numbers, we use the share of cumulated number projects in order to be able to compare patterns across different cut-offs. The larger the deviation of the dashed lines from the straight line the larger is the
deviation of the two
allocation rules.
Source: ESE Eurostars Evaluation Data (2014)
Note: Data for cut-off 9 were not complete at time of data extraction. In interviews, NPCs, NFBs and members of HLG and EAG were quite satisfied with the way the virtual common pot worked. They agreed that the virtual common pot is the preferred funding mechanism since it attracts all countries to contribute to the
programme and to increase their shares beyond the initially agreed level (see Figure
6-10). It is also easier for countries to raise funding from national sources for a virtual
common pot than a real common pot. In that respect, interviewees state that the virtual common pot provides incentives to increase national funding.
According to the NPCs, Eurostars was implemented quite successfully. The SMEs interviewed found that the programme is achieving its goals, even if there are still cases of projects in which partners are not financed at national level. SMEs think that this is the most important bottleneck of Eurostars that needs to be solved. However, some SMEs, despite an excellent evaluation of their projects, did not receive funding because of the restrictions coming from the virtual pot.
Central evaluation
Some NPCs stated that they found their involvement in the eligibility check unnecessary since it should be a straightforward process to control whether a project satisfies eligibility criteria and this could have been done easily by the ESE. The current process required participation of country representatives of NPCs in a session in Brussels, which added costs to the programme management. The possibility for applicants to automatically check for the eligibility of their proposal in Eurostars 2 is appreciated by the interviewees, since the applications will be received through an online system where an applicant will be warned automatically if any information entered does not satisfy the eligibility criteria. However, automatic eligibility checks will not substitute for detailed eligibility checks by a project officer. ESE project officers also need to check all applications to identify the potential experts most suitable to perform the technical evaluation of the application.
The process of central evaluation in Eurostars is appreciated by all governing and administrative bodies involved in Eurostars, due to its speed and objectivity.
November 2014 Page 56 of 139 The improvements in the process, such as the increase in the number of technical experts from two to three in Eurostars 2, and the involvement of experts with market expertise are considered as important developments.
According to ESE data, the number of technical experts increased from 485 in 2010 to 1,736 in 2013. However, there is an imbalance in their geographical distribution (see Figure 6-12).
By 2013, there were a large number of experts from UK, France, Italy and Spain. Also, a considerable large number of experts were listed from newer Member States such as Romania and Hungary. A small number stem from non-Member states participating in Eurostars, like Switzerland or Norway, but also from Germany, at least when that number is compared to the size of the country or to the number of applications to Eurostars. The primary technological areas of expertise of the technical experts were in line with the technological fields covered by Eurostars projects. The majority of experts were in the fields of ICT and biotechnology and chemistry. The technical experts with an industrial background formed 38% of the whole population, whereas the share of experts with a consultancy background was 23%. These two types of expertise provided skills for evaluating the marketing aspects of Eurostars projects.
However, there are diverse views and experiences by NPCs on the sufficiency of technical experts with different expertise. While some interviewees mention that with the opening of the FP7 expert database to Eurostars and through the announcements of NPCs, there are now a significant number of technical experts who also have market knowledge, several NPCs stated that they observed, from time to time, that appropriate technical experts cannot be found by ESE. Those NPCs are of the opinion that the NPC network could be used by ESE to identify enough number of experts with the required
qualifications. However, this has never been the practice in Eurostars.
Some NPCs stated that there are a number of actions which should be taken to improve the central evaluation process. First, the process should become more transparent, particularly by providing detailed feedback to the applicants in a more
Figure 6-12: Country of origin of technical experts in the Eureka database
November 2014 Page 57 of 139 harmonised way as there are differences between countries. The NPCs also find the amount of information about the result of evaluation provided to them insufficient. This particularly leads to problems when SMEs with high levels of specialisation in their fields do not feel that their proposals are appropriately evaluated and contact NPCs for further explanation. Second, streamlining the process in order to decrease the time from application to evaluation is highlighted as an area for improvement. Third, further improvement of the expert database is emphasised as a requirement in order to ensure geographical and gender balance and increase the number of those from the business sector. It was also mentioned that the technical experts have not been provided with “orientation training” on how to evaluate Eurostars projects. As a result, there have been cases where completely different scores have been given to the same project by two experts, making the IEP’s ranking process difficult.
The SMEs interviewed are satisfied with the central evaluation process and the level of information received about the results of the evaluation. They see the evaluation process as clear, transparent and timely. In general, SMEs consider that information about evaluation is pertinent and they appreciate the feedback report. They think that the feedback on evaluation is important to facilitate the redrafting and improvement of the proposal, and a better comprehension of the overall programme objectives and evaluation criteria. Additional technical feedback is seen as an area where there is a potential for improvement.
In some cases (and mainly in the case of unsuccessful applicants), SMEs noted the need for a more comprehensive analysis, not only based in a more detailed feedback report but also through personal interaction explaining the project application’s deviations from evaluation criteria. Some of the interviewed SMEs said that though their applications to Eurostars were unsuccessful, the process was fair and credible, and the evaluation findings helped to further improvement of their projects.
Finally, establishing a central database that can be accessed both by ESE and NPCs is deemed a necessity by interviewees. In this way they would be able to access information on the results of the evaluation as soon as it is completed, and the time- to-contract would decrease, as they might start earlier with their own procedures for
the funding of the selected projects.51
Synchronisation of funding
The data available from ESE only permitted the expert group to look at the time-to- contract, defined here as the time-span between the cut-off date and the date when the funding contract was signed. As this differs significantly between countries, this question was analysed at the level of individual participants and not at the level of projects. ESE was not able to provide us information on time-to-funding, i.e., when the money first arrives at the participants. From interviews with SMEs, it is apparent that time-to-funding might crucially differ from time-to-contract (e.g. there were cases where the project was formally finished and most of the participants already received funding from their governments while there were participants which did not receive any financial support despite of an existing funding contract).
51 In the meanwhile ESE has established such a central database and NPCs have access to the detailed
November 2014 Page 58 of 139 Synchronisation of time-to-contract, that is the time between the cut-off deadline and the signing of the funding contract, was considered as the most important challenge of Eurostars by the interviewees. Also, the interim evaluation recommended significant improvements. In addition, this was repeatedly a key topic of the annual HLG meeting. Hence the expert group examined by data analyses whether a general trend towards short time-to-contract were observed in general and whether the significant differences between participating countries in the time-to-contract were reduced. Finally, time-to-contract was regularly a key topic of talks with SMEs and programme governance and programme administration.
Figure 6-13: Cumulated probability for “still waiting for funding” contract
Figure 6-13 depicts the advances made to bring down the time-to-contract. It shows the estimated probability of the distribution of the time needed before the funding contract was signed for all 10 cut-offs (e.g. in the top chart of Figure 6-13 the red line of cut-off 2 indicates that the probability that about 70% of the funding contracts (=1- 0.30) were signed within 400 days after the deadline; It took nearly 1600 days to sign all funding contracts).
Figure 6-13 indicates a
significant improvement in time-to-contract. NPCs were able to avoid extremely long delays in signing. However, in the initial cut-offs, a small number of SMEs had to wait more than three years before their funding contracts were signed in the initial cut-offs.
November 2014 Page 59 of 139 Such long waiting times were avoided in later cut-offs. Even more important is that a growing share of contracts were signed with a period of less than 250 days from the
cut-off deadline,52 and an
increasing number of funding contracts were signed within 200 days.
Source: ESE Eurostars Evaluation Data (2014)
Although overall improvements have been achieved, time-to- contract is still high in some
countries. Figure 6-14 gives the
average time-to-contract for cut- offs 1 to 5 and cut-offs 6-10 for
participating countries.53
The SMEs interviewed consider synchronisation as a key success factor for the future of the programme. They expect shorter
time-to-contract with effective
negotiation and faster transfer of
funding, enabling them to
accelerate and develop project
tasks and achieve consortia
objectives. In some countries,
time-to-decision for Eurostars
projects takes longer than that for
projects for national R&D
programmes.
Overall, ESE, NFBs and NPCs demonstrated their willingness and ability to shorten the time-to- contract. As a result, the average
52 Table 9-10 in the Annex provides the ranking of countries by time-to-contract in a multivariate
regression setting, accounting for complexity of projects, size of grants, technology sector and the number of projects above the threshold by country.
53 For cut-off 9 and cut-off 10, not all contracts were signed in December 2013, and hence the time-to-
contract for the second period was overestimated, as only signed contracts were used to calculate this figure.
Figure 6-14: Time-to-contract by country
November 2014 Page 60 of 139 time-to-contract has gone down significantly from 458 days in cut-off 1 to 213 days in cut-off 10. The maximum delay in time-to-contract was even reduced much more, so that we can see significant progress between the first phase of Eurostars and the later funding rounds. However, there are still applicants who had to wait more than two years in cut-off 7, the last cut-off for which such an analysis is feasible.
Synchronisation has also been a regular action item in the agenda of NPC and HLG meetings since the launch of the programme. In 2009, and following the first results of an internal working group on synchronisation, the EUREKA Ministerial Conference stated that “constant and increasing attention must be given to synchronisation and rationalisation of national funding schemes and procedures to support EUREKA projects”.
As for the negotiations with NFBs, findings from the SME interviews indicate that the smoothness of the negotiation process differed from country to country. It is observed
that interactions with NFBs reflect a diversity (as shown by Figure 6-11) which is not
only linked to national regulations, but also to cultural aspects. Significant differences are also observed in time to funding, according to empirical evidence, as there are no such data available from ESE. Nevertheless, when asked to rank (on a scale from 1 (low) to 7 (high)) how effective the negotiations with NFBs were, the majority of SMEs selected 6 and 7, indicating that it was indeed effective.
In general, the negotiation process in participating countries is characterised by diversity and heterogeneity. As evident from the SME interviews, different countries use different negotiation procedures (mainly deriving from national regulations and funding schemes), and it is not possible to identify a homogeneous “Eurostars Negotiation Process”. On the other hand, in many cases SMEs declare that the negotiation process went smoothly and was transparent.
The NPCs stated that the target of achieving a time-to-contract of 7 months is feasible, and that all countries involved in the programme should take action to meet this target. Countries like UK and Romania report that they have met the target (6
and 7 months54, respectively) by streamlining their processes. However, the lack of
synchronisation in other countries put their projects at risk, since time-to-contract in some countries is twice and even four times longer than theirs.
There are cases reported by SMEs about long time-to-decision in various countries, which created important obstacles in the implementation of projects. For instance, one