Assessing the effectiveness, efficiency and relevance of the ESF Learning Networks

Assessing the effectiveness, efficiency and relevance of the ESF Learning Networks

VT/2012/104 Request for services in the framework of the Multiple Framework Contract "Provision of evaluation and evaluation related services to DG EMPL”

Final Report, 3rd July 2014

Contents

Abbreviations ... i

Executive summary ... ii

1.0 Introduction ... 1

1.1 Introduction to the report and report structure ... 1

1.2 Context: Background to and operation of the ESF Learning Networks ... 1

1.3 Evaluation aims and key evaluation questions ... 3

1.4 Theoretical framework for the evaluation ... 3

1.4.1 Theory of change / intervention logic ... 4

1.4.2 Evaluation question and indicator framework ... 8

1.5 Evaluation methodology ... 8

2.0 Objectives and relevance ... 13

2.1 Introduction ... 13

2.2 The Learning Networks’ thematic focus and objectives ... 13

2.3 Development of the Learning Networks’ themes and activities ... 16

2.4 Relevance of Learning Networks’ themes and activities to ESF and national priorities ... 17

2.4.1 Effects of the economic crisis ... 20

2.5 Complementarity between Learning Networks ... 21

2.6 Summary of key findings ... 21

3.0 Effectiveness and efficiency: processes and inputs ... 23

3.1 Introduction ... 23

3.2 Effectiveness of Learning Network reach and participation ... 23

3.2.1 Motivations for participation ... 25

3.3 Effectiveness of Learning Network organisation and leadership ... 25

3.4 Effectiveness and quality of Learning Network activities ... 26

3.5 Efficiency of delivery and use of resources ... 27

3.6 Summary of key findings ... 28

4.0 Effectiveness and efficiency: dissemination and outcomes ... 30

4.1 Introduction ... 30

4.2 Effectiveness of dissemination ... 30

4.3 Extent and nature of individual learning outcomes ... 33

4.4 Extent and nature of organisational learning and change ... 34

4.5 Extent and nature of policy transfer and learning ... 35

4.6 Additional outcomes including broader co-operation between partners and Calls for Proposals ... 37

4.7 Effectiveness in meeting Learning Network aims ... 37

4.7.1 Meeting the aims of individual Learning Networks ... 37

4.7.2 Meeting the aims of the Learning Networks as a whole ... 38

4.8 Sustainability of Learning Network activities and outcomes ... 39

4.9 Efficiency in generating outcomes and impacts ... 40

4.10 Summary of key findings ... 41

5.0 Support and monitoring ... 43

5.1 Introduction ... 43

5.2 The role of the Commission in supporting the Learning Networks ... 43

5.3 Support offered by the Learning Network contractor and their role... 44

5.4 Summary of key findings ... 45

6.0 Implications for the future of transnational co-operation through the ESF ... 46

6.1 Introduction ... 46

6.2 Stakeholder views on the key strengths, weaknesses and success factors relating to the Learning Networks ... 46

6.2.1 Success factors concerning Member State engagement ... 47

6.3 Key learning points from Learning Network activity ... 48

6.4 Implications for future policy and activity around transnational co-operation (virtual Policy Delphi) ... 48

6.5 Summary of key findings ... 52

7.0 Conclusion and recommendations ... 54

7.1 Concluding remarks and revisiting the theory of change ... 54

7.2 Summary answers to the evaluation questions ... 55

7.3 Recommendations ... 56

Annex One: Additional detail on the theoretical framework ... 1

Annex Two: Research tools ... 6

Annex Three: Learning Network case studies ... 37

Annex Four: Member state case studies ... 138

Abbreviations

ACB Administrative Capacity-Building Network (ESF Learning Network) AEIDL European Association for Information on Local Development

BFSE Better Future for the Social Economy Network (ESF Learning Network) COP Community of Practice on Partnerships (ESF Learning Network)

COPIE Community of Practice on Inclusive Entrepreneurship (ESF Learning Network)

CRF Common Reference Framework

DG EMPL Directorate-General for Employment, Social Affairs and Inclusion ENYE: European Network on Youth Employment

EMPOWERMENT Empowerment and Inclusion (ESF Learning Network)

EQ Evaluation Question

ESF European Social Fund

ExOCoP Ex-offender Network (ESF Learning Network)

GenderCoP Gender Mainstreaming Network (ESF Learning Network)

IMPART Increasing the participation of migrants and ethnic minorities in employment (ESF Learning Network)

MS Member State(s)

NGO Non-Governmental Organisation

OECD Organisation for Economic Cooperation and Development

OP Operational Programme

PES Public Employment Service

RBM Network on Results-based Management (ESF Learning Network)

SaviAV Social inclusion and vocational integration of Asylum seekers and Victims of human trafficking (ESF Learning Network)

WP Work Package

Executive summary

Introduction

This summary presents the findings of an evaluation aimed at Assessing the effectiveness, efficiency and relevance of the ESF Learning Networks. Findings are based on research undertaken by Ecorys between May 2013 and April 2014. The methodology for the evaluation combined desk research, key stakeholder interviews, Learning Network case studies involving in-depth assessment of selected networks, Member State case studies investigating the Learning Networks from the Member State perspective, an online survey with Learning Network participants and other key ESF stakeholders, and a ‘virtual Policy Delphi’

engaging a selection of high level stakeholders in considering the implications of the findings that emerged from the study.

Reflecting the structure of the evaluation Final Report, key findings relating to a series of aspects concerning the Learning Networks’ operation and outcomes are presented below. The summary concludes by addressing the key evaluation questions set in the study Terms of Reference (ToR).

Recommendations arising from the study are presented in the main body of the report at section 7.3.

Objectives and relevance

It is evident that the 13 Learning Networks assessed as part of the evaluation demonstrate significant breadth and scope in terms of their objectives. These objectives tended to focus broadly on one of three main areas (although there is some overlap): a thematic / policy focus, a concentration on ESF target groups, or operational concerns relevant to the delivery of the ESF. This breadth of focus was generally viewed as being a positive aspect of the overall Learning Network approach, though a minority view was expressed over the spread of thematic focus and objectives being too wide.

The establishment of the Learning Networks’ thematic focus and planned activities can be said to have been broadly effective in most cases. In part this reflects the fact that a number of the networks emerged from pre-existing work and collaboration which was felt to have provided a solid basis for the networks’

development. Commonly the development process involved selecting a broad theme and subsequently refining this through collaborative working. This was widely seen as a positive way to engage multiple perspectives and ensure that the development of the networks’ thematic focus was effective through being inclusive and democratic. Another aspect commonly cited as effective involved the use of a two tier organisational model, using a steering group to develop a network’s thematic focus and objectives supported by working groups charged with further developing and refining particular activities. This was cited as a significant success factor by a number of those involved in the networks when reflecting on the link between organisational structure and the overall effectiveness and outcomes of network activity.

The evidence collated also suggests that the processes used to develop activities were broadly effective in ensuring that they reflected and supported the objectives of the networks. However, it was not always evident that a structured approach, ensuring that activities linked to and would be likely to support objectives, had been followed in all cases. In the main, those involved with establishing the networks also felt that the right partners had been engaged in their development. However, there was some variation in the extent to which the development process was largely restricted to ESF Managing Authorities, or involved a wider selection of stakeholders.

It is clear that the objectives and themes selected by the Learning Networks have a high degree of

relevance to ESF objectives, along with other high level EU policy drivers such as Europe 2020. As might

be expected, a high degree of relevance was also apparent in respect of national priorities of participating

partners, particularly in the case of partners leading networks. This relevance was confirmed through consultations at the Member State level as being apparent both in respect of policy concerns and in relation to operational issues facing ESF MAs and other organisations. Evidence from the online survey undertaken similarly demonstrated that all the main network activities were considered to be relevant or very relevant on all policy levels (ESF, national and EU).

While there were some clear potential linkages between the thematic focus and activities of the different Learning Networks, it is clear that the extent of collaboration between them varied, but was overall limited.

In some cases this was related to the fact that co-ordination of, or involvement in, a network required significant inputs, hence reducing any time that might be available for collaboration. In others, those involved with a network felt that there would be limited value from engaging in such cross network activity.

Finally, the study also gave the opportunity to explore whether and how the economic crisis had affected the thematic priorities or activities developed. In general the main impacts reported were practical and logistical, particularly in terms of reduced ability to travel on the part of some partners from Member States facing significant austerity measures. Some of the network leads and members felt that other external factors (that is, change of government or elections) had had more of an impact than the crisis itself.

Effectiveness and efficiency: processes and inputs

Participation in the Learning Networks and taking the role of lead partner was unevenly distributed across Member States. A group of Member States were represented in networks six times or more while several countries participated in only one or no networks. While a more even spread of participation at the Member State level can be seen as desirable, clear explanatory factors for this unevenness were evident.

These included the lack of a specific requirement for Member State engagement in the networks and the differential level of resources available to ESF MAs to support their participation. Suggestions for encouraging a more even spread of participation included a greater promotional role on the part of the Commission in articulating the importance of transnational co-operation, more compulsion to co-operate for ESF MAs, and enhanced technical assistance to help address issues of limited resources or transnational expertise / experience.

In terms of participation at the level of specific organisations and individuals, it is evident that several networks did take a pro-active and inclusive approach by opening up their activities to NGOs, policy- makers, experts and practitioners. This spread of engagement can be identified as a key positive aspect of the networks in several cases. Equally, in general, the degree and balance of representation across the networks appeared to be appropriate and effective. However, the quality of the individual participants in the Learning Networks was more varied and it is clear that some networks experienced variations in the level of commitment by members. The evidence gathered also demonstrated that Member States and organisational / individual actors within them to engage in Learning Network activity due to a combination of individual interest, organisational interest and the desire to share learning or learn from others.

The available evidence also indicated that the effectiveness of organisational processes within Learning Networks varied significantly. In addition, these processes clearly represent an important factor in determining the success or otherwise of particular networks. Generally speaking, it seems that different organisational modus operandi seemed to work for different networks. In turn this indicates that the organisational structure required to effectively implement network activities needs to reflect the size, particular composition and focus of the networks.

In cases of effective practice, common factors included effective planning and leadership, clearly

delineated steering and working groups with specific and clear remits, clear lines of governance and

accountability and effective project management. Weaknesses in these organisational aspects appeared

to have been a key factor in particular networks not achieving their intended outcomes or doing so to a lesser extent than had been anticipated. In respect of organisational and process related challenges, work load intensity and the level of resource requirements (particularly for network leads) was clearly a key challenge. As noted by stakeholders, this indicates that an enhanced level of technical assistance would be beneficial in supporting network organisation in any future iterations of networks or similar transnational activity.

Despite the variations in organisational effectiveness which were apparent, on the balance of evidence it appears that the activities taken forward by the Learning Networks were implemented in a broadly effective fashion in the majority of cases. They were also generally effective in the sense of reflecting and helping to support the achievement of the networks’ aims. In particular, the importance of activities which were interactive, enabling knowledge exchange and mutual learning, were emphasised in this context. In cases where activities were evidently less effective or successful, a common reason for this appeared to be that the goals outlined initially were too ambitious given the time available and resource constraints faced by those involved.

Finally, judgements over the efficiency of the networks in terms of their implementation and use of available resources link closely to the above discussion over the effectiveness of organisation and leadership. In general, while the Learning Networks can be judged to have operated efficiently in terms of allocating resources to implement activities, some variations were apparent at the level of individual networks. However, it is certainly the case that many of the activities arranged by the networks were carried out on a limited budget which depended on notable in-kind resources being allocated by those involved in delivering them. This can be viewed as a key factor in supporting efficient implementation from the point of view of outcomes achieved against inputs in terms of resources used.

Effectiveness and efficiency: dissemination and outcomes

The dissemination approaches adopted by the Learning Networks proved to be mixed in terms of their effectiveness. In some cases dissemination products were effective in transmitting the results of network activity but this was not universally the case. Equally, while some dissemination strategies developed by offered a useful structure for sharing results, in respect of others evidence suggests that such strategies were either under-developed or inappropriately targeted. Where dissemination worked well, the use of partners as ‘ambassadors’ to share learning within Member States was widely seen as a significant success factor. Similarly, building links with key players at EU and Member State level also appeared to be important to ensure effective dissemination. In terms of specific mechanisms, ‘passive’ approaches such as printed materials or websites appeared less effective than more active forms of dissemination through events, meetings or on-going liaison with relevant stakeholders.

The clearest and most consistently positive outcomes from the Learning Networks emerged at the level of individual learning amongst those participating. Whilst this might be expected, the evidence gathered illustrated the degree to which the vast majority of network members felt that notable positive effects had resulted from their engagement. These included new contacts, enhanced individual relationships, and a deeper level of knowledge about particular thematic areas. In the minority of cases where individuals felt that their involvement with the Learning Networks had been less positive, this generally related to the focus and activities of the network concerned in the sense of these not meeting expectations. In many cases organisations also clearly gained from the knowledge exchange involved in the networks, along with the opportunity to develop partnership and cooperation with related actors across Member States.

On balance, positive examples of organisational learning and impacts outweighed cases where networks

were felt to have had less of an influence at the organisational level.

Evidence of the degree of policy learning and transfer at the organisational, Member State and EU levels was, however, mixed and perhaps more limited than stakeholders involved with the Learning Networks had anticipated or hoped for. In particular, the concept of direct policy transfer stemming from network activity was questioned by a number of Learning Network and Commission stakeholders in terms of the achievability of this objective. Common reasons offered for this were the long term and complex nature of policy development, difficulties in achieving consistent engagement amongst policy makers, and the lack of direct transferability of policies stemming from the different political, social and economic contexts within Member States. Partly as a result of these factors, where policy effects could be identified these were more in the form of policy influencing as opposed to direct policy transfer.

Within this, policy influence as a result of network activity was more evident in the sphere of the ESF itself, whether in terms of influence over particular OPs (particularly in the context of planning for the new programming period) or in respect of ESF guidance and regulations at the EU level. However, while concrete examples of policy transfer appear relatively limited, it is evident that more subtle effects concerning the transfer of policy ideas and options for operational adaptation occurred within the context of the networks. Moreover, such effects have the potential to influence policy and operational change over a longer timeframe. In terms of broader outcomes, in the majority of cases evidence suggests that the Learning Networks contributed to improving ESF transnational cooperation between Member States.

However, there was little evidence of network activity influencing Calls for Proposals, certainly not in the sense of directly leading to such calls.

As well as the broader outcomes noted, the evidence assessed indicates that the Learning Networks varied in the degree to which they met their specific stated objectives, but that in most instances the majority of or all objectives were met to a high degree. There was also a trend in the types of objectives that proved more difficult to achieve. Typically these related to the production of recommendations and influencing policy.

In terms of sustainability in respect of the Learning Networks this can be considered at several levels:

firstly, in relation to the sustaining of contacts or relationships; secondly, in terms of continuation of network activity or successor networks; and thirdly, in terms of the sustainability of effects and outcomes generated through Learning Network activity. While for the last of these levels it is perhaps too early to make a judgement, it seems likely that such effects will continue but might prove diffuse and difficult to identify as time passes. In respect of the other levels of sustainability noted, evidence was mixed but broadly positive in relation to the continuation of relationships and activity begun under the networks.

Finally, in respect of the efficiency of the Learning Networks, it appears that the majority of Learning Networks were broadly efficient when judged from the perspective of what they achieved relative to the resources available to them. Key factors underpinning this judgement relate to the fact that the networks benefitted from considerable ‘in-kind’ resources contributed by partners, had to implement their activities carefully given overall resource constraints, and in a number of cases produced outputs that would have cost significantly more had they been produced by consultants or other external contractors.

Support and monitoring

Perspectives on the support offered by the Commission to the Learning Networks by those leading or co-

ordinating the networks were divided fairly evenly between positive views and a feeling that the support

could have been better and / or could be improved in future. Despite this some very positive comments

on the support and its contribution to network functioning and outputs were offered. In cases where views

were less positive, the key issues raised concerned high levels of turnover in Commission staff and a

concern that the capacity available to support the networks was insufficient. Potential improvements

suggested by network leads included: greater assistance with identifying and working with external

experts, facilitators and/or external evaluators; enhancing the support and guidance provided on running effective networks; providing standardised tools for running and co-ordinating a network; assistance in developing links with relevant Commission units; and reducing reporting and administrative requirements.

Views on the role of AEIDL in monitoring the Learning Networks also varied. While a number of network leads and participants appreciated the supportive role that individuals from AEIDL had offered, noting that this went beyond their monitoring remit, in other cases views were less positive. The main issues raised concerned a perception that in some cases AEIDL staff did not possess the thematic knowledge that network participants felt was required for their role, that the quality of the monitoring function was variable and that some meetings and events were not organised effectively. Views on the network co-ordination meetings facilitated by AEIDL were similarly variable; while most stakeholders felt that the concept of the meetings was good, it was sometimes noted that the content and organisation of the meetings was less effective. Finally, stakeholders commonly felt that the role of AEIDL might have been clearer, particularly in terms of their remit and the balance between monitoring and support functions (whilst acknowledging that the latter element was not a formal part of their remit).

Implications for the future of transnational co-operation through the ESF

Stakeholders provided a range of views on the key strengths, weaknesses, and success factors emerging from the Learning Networks which can inform future Learning Network type activity and transnational co- operation more broadly. Key strengths cited tended to relate to the core purpose of the networks, in terms of transferring learning and sharing good practice, along with the quality of some of the networks’ outputs and perceived achievements. Weaknesses cited tended to relate to organisational and process issues in addition to a lack of impact in terms of directly influencing policy or policy transfer. Drawing on these views, a series of success factors relating to the networks can be summarised as follows:

 The importance of developing a clear and consistent thematic focus with clear and realistic objectives being developed to reflect these themes

 Producing a clear implementation plan which sets out the relative responsibilities of individual partners, steering groups and working groups

 Effective network composition in terms of ensuring that appropriate and committed partners (and an appropriate range of partners) are engaged in network activity

 The development of clear governance processes and lines of accountability

 Effective organisation and leadership on the part of those co-ordinating network activity

 A well developed dissemination strategy, supported by high quality tools and outputs which are aimed at specific audiences.

Examination of the effects of the Learning Networks at the Member State level also serves to highlight some key implications for how Member States might best engage in transnational co-operation. These include the need to ensure that Member States develop a clear strategy to guide their participation in such activity, the development of a supporting infrastructure able to transfer learning to relevant stakeholders in the domestic context as well as feeding in learning effectively to the European level, and ensuring consistency of participation as far as possible (both in respect of operational and policy actors).

Running a virtual Policy Delphi to explicitly consider the future of transnational co-operation served to

highlight several key considerations in this area. These include the importance of effective technical

assistance to support the engagement of Member States and actors within them in transnational

initiatives, the significant role the Commission can play in promoting transnational co-operation and its

value whilst actively guiding such co-operation in the context of the ESF, greater engagement of

Commission policy units, greater realism over the both the potential and limitations of transnational co-

operation on the part of both the Commission and those engaging in it, and the importance of Member States / ESF MAs acting as an effective conduit for transferring learning to domestic actors and back to the European level.

Summary answers to the evaluation questions

Responses to the evaluation questions set out for the evaluation in the study ToR, based on the analysis presented in preceding chapters, are summarised below. Where applicable, the reader is signposted to the specific sections of the main report from which the evidence to inform these responses can be found.

EQ1 - To which extent have Learning Networks’ activities corresponded to the overall aims and policy objectives of the ESF?

There is a clear relationship between the activities developed by the Learning Networks and the overall policy aims and objectives of the ESF. A high degree of relevance between network activities and these policy objectives and overall aims was apparent in the majority of cases (section 2.4).

EQ2 - To which extent, under what circumstances, from whom and by whom learning has taken place during the duration of the Learning Networks?

Learning deriving from the activities of the Learning Networks has taken place at a range of levels and amongst numerous actors during their duration. Whilst such learning is clearest and most apparent in respect of individual network participants (section 4.3), it is also widely apparent in respect of participating organisations, whether ESF MAs or other partner bodies (section 4.4). The ways in which learning took place varied between the networks, reflecting their different focus and the range of activities they undertook. Typically, however, peer reviews, study visits, events and conferences were cited as effective mediums through which to share practice and generate learning (section 3.4). There is also evidence of influence on policy learning, although evidence of direct policy transfer is limited (section 4.5).

EQ3 - To which extent the activities of the Learning Networks have contributed to the achievement of the Learning Networks’ objectives?

As reflected in the response to EQ.2, there was some variation in the specific activities which led to particular objectives being met or otherwise across the Learning Networks. In turn, this reflects the variation in the scope and focus of the networks themselves. Overall, however, a clear relationship between the activities networks engaged in, and the high level of achievement of the objectives they set themselves, was apparent in most cases (sections 3.4, 4.7). The key exceptions to this involved some networks failing to produce particular planned outputs and less evident achievement of objectives around influencing policy development (sections 4.5, 4.7).

EQ4 - What are the achievements of the calls for proposals organised in the context of the Learning Networks (where applicable)?

The assessment revealed that Calls for Proposals did not generally emerge from Learning Network activity. While some examples of activities influencing the design or development of Calls for Proposals in the context of Member State and regional ESF OPs were offered, the linkages between these and network activity were unclear. Overall, therefore, there was little evidence of network activity influencing Calls for Proposals, certainly not in the sense of directly leading to such calls (section 4.6).

EQ5 - To which extent have the dissemination activities reached relevant stakeholders?

The dissemination approaches adopted by the Learning Networks proved to be mixed in terms of their

effectiveness. In general, ESF MAs were reached to a greater extent than other actors, as were ministries

of labour though to a lesser degree. NGOs, representatives from other ministries, and other actors such

as employers and trade unions appeared to have been less well reached by dissemination activity

(section 4.2).

1.0 Introduction

1.1 Introduction to the report and report structure

This final report presents the findings of an evaluation aimed at Assessing the effectiveness, efficiency and relevance of the ESF Learning Networks. The report builds on two previous reports submitted as part of the evaluation, an Inception Report submitted in July 2013 and an Interim Report submitted in December 2013. The final report is based on research undertaken by Ecorys between May 2013 and April 2014.

The remainder of the final report, including annexes, is structured as follows:

 Chapter one details the context for the evaluation, along with the evaluation aims and methodology;

 Chapter two addresses questions relating to the objectives and relevance of the Learning Networks, in particular the relevance of the networks’ themes and activities to ESF and national priorities;

 Chapter three assesses the effectiveness and efficiency of the Learning Networks in relation to key processes around their operation, including network participation, organisation and activities;

 Chapter four assesses effectiveness and efficiency in terms of dissemination and outcomes from the Learning Networks, including organisational learning, and policy transfer;

 Chapter five examines the effectiveness of support and monitoring processes linked to the operation of the Learning Networks;

 Chapter six considers implications for the future of transnational co-operation through the ESF arising from the experience of the Learning Networks and the findings of the evaluation;

 Chapter seven concludes the report by addressing the key evaluation questions established for the study and providing recommendations on the basis of the evaluation findings.

► Annex one details the theoretical framework guiding the assessment through mapping evaluation questions to indicators and methodological tasks;

► Annex two presents the research tools used in the study;

► Annex three presents the Learning Network case studies which formed one element of the methodology;

► Annex four presents the Member State case studies that formed another element of the approach.

1.2 Context: Background to and operation of the ESF Learning Networks

To promote access to new ideas, innovative approaches and new skills the European Commission supports transnational co-operation between Member States of the European Union. One route to supporting such transnational co-operation involves channelling ESF funding to Member States and their regions to support transnational priorities in national and regional ESF Operational Programmes (OPs), the mechanism through which ESF objectives and activities are agreed with the Commission. In order to provide additional support and incentives to work transnationally, the Commission also undertook a number of actions in the 2007-2013 ESF programming period including the funding of transnational 'Learning Networks'. The Learning Networks are intended to facilitate and strengthen transnational exchange and cooperation between ESF managing and implementing bodies and strategic stakeholders.

The Learning Network approach draws on the experiences of the nine Thematic Focus Groups (TFGs)

implemented within the context of the ESF Community Initiatives ADAPT and EMPLOYMENT during the

1994-1999 programming period, and the two rounds of European Thematic Groups (ETGs) implemented within the five thematic pillars of EQUAL Community Initiative during the 2000-2006 programming period.

As part of the process of establishing the networks, ESF Managing Authorities (MAs) were invited to respond to restricted calls for proposals in order to obtain funding for the management of Learning Networks, including the provision of facilitators and any other professional expertise required for their successful management. Calls were launched in 2008 (VP/2008/018) and 2009 (VP/2009/012), and more recently in 2012 (VP/2012/005). As a result of the 2008 and 2009 calls for proposals, thirteen Learning Networks were established which are the subject of this evaluation. Each Learning Network covered a specific thematic or governance related area as follows:

 COPIE: Community of Practice on Inclusive Entrepreneurship

 BFSE: A Better Future for the Social Economy

 EMPOWERMENT: Empowerment and Inclusion

 ENYE: European Network on Youth Employment

 IMPART: Increasing the Participation of Migrants and Ethnic Minorities in the labour market

 ExOCoP: Ex-offender Community of Practice

 Partnership: Community of Practice on Partnerships in the ESF

 TNC: Transnational Cooperation in the ESF

 CoP RBM: Results Based Management in the ESF

 Gender CoP: Community of Practice on Gender Mainstreaming

 ESF AGE: ESF AGE

 SaviAV: Social inclusion and vocational integration of Asylum seekers and Victims of human trafficking

 ACB: Administrative Capacity Building - Facing the Challenge – How to effectively support public administration reforms by ESF funds.

The Learning Networks were each led by representatives of MAs from within Member States, although each adopted different management, implementation and operational approaches and undertook different activities related to their thematic focus and objectives. Common elements included the engagement of a set of Learning Network members or participants drawn from different participating Member States, the defining of a set of objectives for the network to work towards, and the identification and selection of a set of activities through which to meet these objectives. Further detail on the rationale, objectives, activities and intended results and impacts of the Learning Networks is provided in section 1.4 below as part of setting out a ‘theory of change’ relating to Learning Network activity.

In terms of monitoring and support for the networks, as part of its approach to transnational co-operation in 2009 the European Commission launched a tender for a contractor to provide monitoring services for transnational and innovative projects, and to synthesise and disseminate relevant results (VT/2008/090).

This tender was awarded to the European Association for Information on Local Development (AEIDL).

The contract included the provision of a monitoring and support role in respect of the ESF Learning Networks. As part of their role, AEIDL participated as an observer in all EU Learning Networks and provided regular reports to the Commission on their performance.

In order to support the work of the ESF Learning Networks, the European Commission sought to provide

support throughout their implementation. In particular, Policy Officers from DG EMPL were assigned to

specific Learning Networks in order to support their work in a variety of ways: attending certain meetings

or events, seeking to link activities or outputs with other relevant policies/stakeholders, and providing

inputs as appropriate. The Commission also set up "support groups" composed of policy officers from

relevant DGs. Finally, the Commission asked the monitoring contractor, AEIDL, to organise Coordination Meetings of the Learning Networks in order to share experiences on a regular basis.

1.3 Evaluation aims and key evaluation questions

The assessment of the Learning Networks is significant in determining not only their success to date in the current ESF programming period, but also in identifying recommendations and areas of improvement in the field of transnational co-operation for the 2014-2020 programming period. This balance of backward-looking evaluation and forward-looking assessment is reflected in the Terms of Reference (ToR) for the study. The ToR outlined three key aims:

1. To assess the effectiveness, efficiency and relevance of the Learning Networks.

2. To assess their [the Learning Networks] contribution to enhancing capacity building and mutual learning between ESF managing and implementing bodies and with strategic stakeholders.

3. To formulate recommendations on the basis of the outcomes of this evaluation, which support the discussion on the role of the Learning Networks in the next ESF programming period 2014-2020, especially in the area of promoting transnational co-operation.

The ToR also set out six key evaluation questions (EQs) as follows:

EQ1 - To which extent have Learning Networks’ activities corresponded to the overall aims and policy objectives of the ESF?

EQ2 - To which extent, under what circumstances, from whom and by whom learning has taken place during the duration of the Learning Networks?

EQ3 - To which extent the activities of the Learning Networks have contributed to the achievement of the Learning Networks’ objectives?

EQ4 - What are the achievements of the calls for proposals organised in the context of the Learning Networks (where applicable)?

EQ5 - To which extent have the dissemination activities reached relevant stakeholders?

EQ6 - What are the successful examples of mutual learning coming out of the Learning Networks activities and successful projects funded under the Call for Proposals?

1.4 Theoretical framework for the evaluation

At the outset of the work, a theoretical framework was developed to guide the study and ensure that the above evaluation aims and key evaluation questions were addressed. The core components of this framework were:

 The development of a ‘theory of change’ approach in respect of the Learning Networks through which to provide a structured means of assessing their operation and outcomes.

 Mapping of the key evaluation questions to sub-questions, indicators and methodological tasks to

ensure the evaluation aims were met and to guide the assessment process.

The overall approach to the assessment was informed by the recognition that multiple factors are likely to have influenced the relevance, effectiveness and efficiency of the ESF Learning Networks. In order to determine the contribution of the networks the assessment process therefore focused on:

1. Outlining the intervention logic / theory of change for the ESF Learning Networks to include learning processes and other outcomes.

2. Through this intervention logic, identifying effects we would expect to observe at each stage and determining the key success factors that explain how the desired effects have been brought about; if necessary, considering alternative explanations for any observed effects.

3. Undertaking case studies and an online survey to explore how the effects have been generated and that offer good practice examples for future expansion or replication.

4. Developing recommendations that draw on confirmed explanatory factors and key research findings.

1.4.1 Theory of change / intervention logic

In order to undertake a structured assessment of the Learning Networks it was first necessary to define

the intended rationale for the networks’ operation, along with their aims and objectives, inputs, activities,

intended results and intended impacts. This ‘theory of change’ or ‘intervention logic’ is outlined below and

presented in diagrammatic form in figure 1.1 overleaf.

Figure 1.1 Intervention logic / theory of change for the ESF Learning Networks

Rationale Aims and objectives Inputs Activities Results Impacts

Collaborating and sharing experiences across borders, whether national, regional or organizational, is an effective way to access new ideas, innovative approaches and

new skills

Use of Learning Networks as one mechanism by which transnational cooperation and exchanges of good practice and knowledge can

be facilitated, hence supporting the implementation, delivery

and outcomes of ESF

Support the objective of:

- improving the quality and efficiency of Structural Funds

programmes and their impact on employment, social inclusion and training

across the whole Union -contributing to policy development at EU level

through:

- exploiting opportunities for programme managers, strategic stakeholders, governmental departments in charge of policy supported

under the ESF and practitioners for learning from one another, and with

each other - capitalising good practice under the ESF Operational Programmes 2007-2013 (in particular from innovative action and transnational

cooperation)

Financial support through the ESF Technical Assistance

operational budget (maximum of €4 million under 2008 call for proposals

and €3.6 million under 2009 call)

Additional contributions through the Commission’s

role in facilitating good practice and acting as a catalyst for learning and

change

Additional (in-kind) Member State contributions (financial

and human resources)

Thirteen Learning Networks undertaking a range of

activities including:

- exchange events - peer reviews - learning seminars

- policy fora - the development of toolkits, communication platforms and technologies

- the transfer of competencies and experience through mechanisms such as training

and coaching

Programme level:

- Fine-tuning of arrangements promoting transnational exchange

and co-operation - Stakeholder and practitioner

networks at OP level - Launching of action plans to base

ESF support on agreed good practice - Introduction of common

(management) tools - Common approaches to monitoring,

evaluation or reporting Individual members:

- helping each other in solving problems at operational and strategic

levels

- providing stimuli to reflect on practice and improve it - applying common, tested, working

methods

- providing opportunities for staff exchange - gaining professional skills and

recognition Participating organisations:

- keeping up with developments across Europe - getting easy access to a pool of competencies to respond faster to

emerging policy needs;

- using common successfully tested tools

- developing and assuring professional competence - speeding up the use and integration

of good practice gained elsewhere - developing a common voice on the

issues at stake

Learning Networks making a contribution to:

- strengthening the capacity to innovate - modernising and adapting institutions to new social and

economic challenges - identifying and assessing

issues and solutions for reforms in policy and

delivery - improving the quality of

governance of public policies, programmes and

actions.

Rationale

The presumption behind the Learning Networks was that collaborating and sharing experiences across borders, whether national, regional or organisational, is an effective way to access new ideas, innovative approaches and new skills. In line with this the ESF Regulation No 1081/2006 (Art. 3) invited Member States and regions to support transnational cooperation in all policy areas identified for ESF interventions.

The development of ESF Learning Networks was intended to form one mechanism by which transnational cooperation and exchanges of good practice and knowledge can be facilitated.

Aims and objectives

The overall aims and objectives of the Learning Networks involved supporting the wider purpose of transnational cooperation under the ESF; that is, contributing to employment policy and delivering reforms through learning from other countries and regions by bringing together people, knowledge and practice, activities, networks and fora in the field of employment, social inclusion and training. ¹ As outlined in the 2006 EC Action Plan to support Transnational Cooperation at EU level and the 2008 Call for Proposals (VP/2008/018), the Learning Networks were intended to yield important multiplier effects to:

 Strengthen the capacity to innovate

 Modernise and adapt institutions to new social and economic challenges

 Identify and assess issues and solutions for reforms in policy and delivery

 Improve the quality of governance of public policies, programmes and actions. ²

The 2009 Call for Proposals (VP/2009/012) ³ defined the purpose of the ESF Learning Networks as supporting the objective of:

 Improving the quality and efficiency of Structural Funds programmes and their impact on employment, social inclusion and training across the whole Union

 Contributing to policy development at EU level.

These objectives were intended to be achieved through:

 Exploiting opportunities for programme managers, strategic stakeholders, governmental departments in charge of policy supported under the ESF and practitioners for learning from one another, and with each other; and

 Capitalising good practice under the ESF Operational Programmes 2007-2013 (in particular from innovative action and transnational cooperation).

Inputs

Financial support for the implementation of Learning Networks was through the ESF Technical Assistance operational budget, budget line 04.022000. It is not easy to fully quantify the totality of the financial inputs to the Learning Networks overall. However, as detailed in the Call for Proposals (VP/2008/018), a maximum of €4 million was allocated for commitments in 2008. The equivalent indicative amount for the 2009 call was €3.6 million.

1 European Commission, Learning for Change: Setting up learning networks under the ESF 2007-2013, Call for Proposals VP/2008/018

2 Ibid.

3 European Commission, Learning for Change: Setting up learning networks under the ESF 2007-2013, Call for

Proposals VP/2009/012

Activities

The inputs outlined above supported the range of activities implemented through the Learning Networks.

At the overall programme level thirteen Learning Networks were established in response to the calls for proposals (VP/2008/018 and VP/2009/012) in 2008 and 2009. Within this a range of specific activities were taken forward by individual Learning Networks. While these varied, there were also a number of common elements which characterised the activity of the networks. These included exchange events;

peer reviews; learning seminars; policy fora; the development of toolkits, communication platforms and technologies; and the transfer of competencies and experience through mechanisms such as training and coaching.

Results

Each of these activities was envisaged to “generate added value for the ESF programmes and the policies which the ESF supports, and for the people and organisations participating in its activities.” ⁴ These include, at the individual level in terms of professional development:

 Helping each other solve problems at operational and strategic levels

 Providing stimuli to reflect on practice and to improve it

 Applying common working methods tested in other countries or regions

 Providing opportunities for staff exchange

 Gaining professional skills and recognition. ⁵

Added value through the results of Learning Networks is expected at the level of participating institutions and organisations in terms of capacity building for managing ESF programmes effectively. This is anticipated to be evident through:

 Keeping up with developments across Europe in the policy or governance field in question

 Gaining a pool of competencies and experience which allows to respond faster to emerging policy needs

 Using common tools successfully tested across Europe

 Developing and assuring professional competence

 Speeding up the use and integration of good practice gained elsewhere in Europe

 Developing a common voice on the issues at stake. ⁶

Finally at the programme level:

 Mutual fine-tuning and orchestration of arrangements to promote transnational exchange and co- operation under ESF programmes, in order to develop synergies and complementarities

 Setting up networks of stakeholders and practitioners at OP level contributing to and taking up results of the network

 Launching action plans to base ESF support on commonly agreed good practice or approaches

 Introducing common (management) tools

 Common approaches to monitoring, evaluation or reporting on ESF activities in the fields of cooperation. ⁷

4 Ibid. Also detailed in the 2008 Call for Proposals.

5 Ibid.

6 Ibid.

7 Ibid.

Impacts

Beyond the more immediate and short term results from Learning Network activity, the intervention is intended to contribute to several impacts which reflect the aims and objectives of the Learning Networks and the Commission’s support for them. The impacts involved are intended to flow from the more immediate results of the programme and may only be evident in the longer term. These impacts are summarised in terms of the intention for the networks to contribute to:

 Strengthening the capacity to innovate

 Modernising and adapting institutions to new social and economic challenges

 Identifying and assessing issues and solutions for reforms in policy and delivery

 Improving the quality of governance of public policies, programmes and actions. ⁸

1.4.2 Evaluation question and indicator framework

The second core component of the theoretical approach to the assessment involved a process of developing sub-questions relating to the main evaluation questions detailed in section 1.3, then mapping these evaluation questions and sub-questions to a series of indicators that could be used to underpin the assessment. The full framework, which also maps the evaluation questions and indicators to the methodological tasks involved in the study, is included at Annex 1. This approach ensured that the evaluation remained focused on the key questions under consideration, whilst also informing the methodological approach outlined below and the design of the research tools included in Annex 2.

1.5 Evaluation methodology

Informed by the theoretical framework outlined above, the methodology for the evaluation involved a series of interlinked work packages (WP) as follows:

WP1: Kick-off meeting

A kick-off meeting for the evaluation took place at DG EMPL on 23 May 2013. The meeting focused on agreeing the study approach in terms of the theoretical framework, methodology, research tools, data sources, work organisation, day-to-day project management and client liaison.

WP2: Desk research

A programme of desk research was undertaken to examine available information about the Learning Networks. The analysis laid the foundation for the remaining work packages in terms of developing a deeper understanding of the networks on the part of the evaluation team, in addition to providing evidence through which help address the Evaluation Questions specified for the assessment. Specifically, the desk research involved a targeted and structured review of the following (where appropriate):

 Interim and final reports of the Learning Networks submitted to the Commission

 Monitoring reports prepared by AEIDL, including quarterly reports for each Learning Network produced during the duration of the grant agreements between the Commission and the networks

 Reports, evaluations, tools and other outputs produced by the Learning Networks

 Reports of the major events organised by the Learning Networks

8 Ibid.

 Policy literature regarding the ESF, Learning Networks, Europe 2020 and EU policy more generally

 Additional relevant literature, including that form academic sources, identified over the course of the assessment.

WP3: Evaluation scoping

An evaluation scoping phase was undertaken to develop research tools for the assessment of the Learning Networks. The scoping stage also involved the further design and development of the case studies which formed a core element of the methodological approach (see WP6 and WP8 below). The research tools included semi-structured topic guides for use in the key stakeholder interviews (WP5) and case studies, along with templates for analysing and reporting the results of the case studies.

In terms of informing the design of the case studies, the scoping phase involved the development of criteria with which to select a sample of Learning Networks for more in-depth assessment in WP6. The criteria used were as follows:

 Type: a balanced sample of Learning Networks focused on ‘governance’ or ‘thematic’ issues

 Theme: at least one network selected from each thematic cluster defined

 Geographical coverage: as far as possible, a selection of networks representing the inclusion of as many MS as possible, as well as diversity in ESF Managing Authority leads

 Longevity of Networks: as far as possible, a mix of Networks which started in the current ESF programming period and some that are continuations of earlier work (for example, EQUAL)

 Preliminary indications of degree of impact: examples of successful, but also less successful, practice identified through the desk research in WP2.

Application of the criteria resulted in agreement with DG EMPL that the following networks would be selected as case studies for WP6:

 BFSE: A Better Future for the Social Economy

 EMPOWERMENT: Empowerment and Inclusion

 ExOCoP: Ex-offender Community of Practice

 Gender CoP: Community of Practice on Gender Mainstreaming

 CoP RBM: Results Based Management in the ESF

 TNC: Transnational Cooperation in the ESF

 ENYE: European Network on Youth Employment.

The scoping work package also involved a selection process for the Member State case studies (WP8).

This process was based on three main criteria, as well as on insights gained through the stakeholder interviews (WP5) and desk research (WP2). The criteria used were as follows:

 Geographical/welfare regime balance: aimed at ensuring coverage, as far as possible, of Nordic, Anglo-Saxon, Continental, Southern European, Central and Eastern European Countries

 Degree of involvement in transnational cooperation: involving the selection of Member States based on the levels of engagement (using two sub-criteria: participation in Learning Networks and the extent of promotion of transnational projects under Member States’ own funding)

 Degree of centralisation of ESF management: aimed at ensuring a balance of countries where ESF is managed at national/state/regional level.

Application of the criteria resulted in the following Member States / OPs being selected in agreement with

DG EMPL: Spain, Czech Republic, Belgium (Flanders), and the United Kingdom (England and Gibraltar

OP). ⁹ It was also agreed to conduct interviews with representatives of the ESF MAs in Portugal and Greece, not as full Member State case studies but specifically to investigate reduced participation in the Learning Networks that were the subject of the evaluation.

WP4: Inception report

A draft Inception Report was submitted on 2nd July 2013 providing an updated study methodology based on the evaluation scoping phase, a progress update, evaluation work plan and initial findings from the desk research. An inception meeting to discuss the report took place at DG EMPL on 7 July 2013. After minor revisions, a revised report was submitted on 16 July 2013 and subsequently approved in July 2013.

WP5: Key stakeholder interviews

Between August and November 2013, in-depth interviews were conducted with nineteen key stakeholders with a strategic role in the design and operation of the ESF Learning Networks. The interviews were used to gather an initial strand of evidence as part of addressing the specified Evaluation Questions, to identify additional contacts and data sources for the ESF Learning Network case studies, and to contribute to selecting Member States for the Member State case studies. The interviews included consultations with lead representatives of the 13 Learning Networks under consideration, along with a selection of DG EMPL representatives with roles relating to the networks.

WP6: Learning Network case studies

On the basis of agreed criteria, seven Learning Networks were selected to be the subject of case studies (see WP3 above for those selected and the criteria used). The purpose of the case studies was to:

 Examine Learning Network members’ opinions regarding the quality and relevance of activities

 Assess the extent and impact of dissemination activities

 Determine the extent, nature and depth of learning benefits

 Identify the extent and nature of increase in personal networking and contacts

 Identify instances of organisational learning and change; for example, increased capacity, new/improved relationships, networking or co-operation with other relevant stakeholders

 Identify instances of policy influence and change to which the networks selected contributed.

Each case study involved several interlinked research tasks as follows:

 In-depth review of relevant documentation relating to the Learning Networks, including outputs produced by the networks, AEIDL monitoring reports, and other policy and contextual documentation relating to the thematic focus of the network in question

 Development of intervention logics for each network, drawing on the above review of documentation, through which to structure and frame the assessment

 Between six and ten in-depth interviews with stakeholders including, as appropriate, network leads and co-ordinators, AEIDL monitoring expert, European Commission support person, Learning Network members and experts used by the networks

 Analysis of the evidence emerging from the documents and interviews allowing a comparison of findings in light of the intervention logic

9 Given the way in which the ESF is managed and implemented in Belgium and the UK a decision was made, for

reasons of consistency and being able to better focus the case studies, to focus on Flanders in the Belgian context

and the England and Gibraltar OP in the United Kingdom context.

 Case study reporting, involving written summaries of around 10-15 pages each (the individual reports can be found at Annex three.

WP7: Interim report

An Interim Report, drawing on the above work packages, was produced and submitted to DG EMPL on 6 December 2013. The report included a progress update, revised work plan, and findings based on the research undertaken to that point. An interim meeting was held at DG EMPL on 12 December 2013 and a revised version of the report with minor amendments was submitted on 17 December 2013.

WP8: Member State case studies

On the basis of agreed criteria, four Member States were selected as case studies (see WP3 above for those selected and the criteria used). The Member State case studies were used to add to our understanding of the effectiveness of the Learning Networks by looking at how Member States engaged across a range of Networks and the impact of such engagement. The case studies were also used to draw conclusions relating to the process aspects of the Learning Networks in terms of:

 The most effective ways for Member States to engage with the networks

 The ways in which learning, policy transfer/adaptation, and the improvement of the quality of ESF management and capacity-building takes place.

The Member State case studies were undertaken through:

 Interviews with key individuals at the applicable Member State / regional / OP level, including those involved in the networks, those with a co-ordinating role with respect to the networks within the Member State, and those from organisations such as Government Ministries that had potentially benefited from their activities

 Reviewing relevant policy and operational documents (including those suggested by interviewees) relevant to any learning or other impacts in the Member State context;

 Analysis of evidence and subsequent reporting through written summaries of around 5-7 pages each (included at Annex four).

WP9: Online survey of Learning Network stakeholders

To add to the key stakeholder interviews and case studies, and further develop the evidence base for the assessment of the Learning Networks, an online survey was conducted with stakeholders connected to the Learning Networks using the web-based CheckMarket tool. The survey was carried out via an online questionnaire sent out to ESF Learning Network participants and other key stakeholders including Commission and ESF MA representatives and a small number of wider stakeholders with a link to the work of the networks such as social partners. Multiple channels were used to obtain the contact details of the respondents:

 For the ESF Learning Network members and related stakeholders, the participant lists in the AEIDL Learning Network monitoring and final reports were used. Contact details were updated through internet searches and, where necessary, by consulting the lead partners of the Networks (yielding 195 potential respondents in total).

 The European Commission sent out an invitation to the Technical Working Group of the ESF

Committee (yielding 62 potential respondents in total).