5 Evaluation methodology and data collection
5.2.4 Case studies
Case studies provide the opportunity for investigating more thoroughly a spe- cific theme, question or dilemma. Thereby, the case studies provide valuable insights and information that contribute substantially to the evaluation. How- ever, at the same we acknowledge that it is not straightforward to assess how the single case studies fulfil the criteria of relevance, effectiveness, consistency and complementarity, support and involvement, and sustainability, - and that it in particular is difficult to compare such fulfilment in between the case studies. In order to facilitate these assessments and comparisons we have developed a scoring system where each of the criteria for each of the case studies is scored on a scale from 1 to 4 - i.e. from low to high. The use of an even number of scores is adopted to force ourselves to assess whether or not the fulfilment is above or below average. Table 5-7 below shows the narrative descriptions of what the scores mean for each of the evaluation criteria.
Scoring of evaluation focus areas
Table 5-7 Scoring system for evaluating case studies
Relevance
4 Project addresses the real needs of European public health and the topic is high on the present political agenda or cross-country cooperation on the project topic is perceived to entail significant European added value
3 Project addresses the real needs of European public health and the topic has some policy attention
2 Project addresses the needs of European public health, but the topic is low on the present political agenda
1 Project addresses the real needs of European public health to a limited degree only
Effectiveness
4 Project objectives have been achieved and there is evidence that the interventions have improved the addressed health issue
3 Project objectives have been achieved and interventions are perceived to have improved the addressed health issue
2 Project objectives have been achieved but the effect on the addressed health issue is unclear
1 Project objectives have not be achieved
Consistency and complementarity
4 Project results fully support other EU programmes , national policies, and international initiatives within public health
3 Project focus is in line with the priorities in other EU programmes , national policies, and international ini- tiatives within public health
2 Project focus or results are only to a limited degree in line with EU programmes , national policies, and international initiatives within public health
1 Project focus or results are not in line with EU programmes , national policies, and international initiatives within public health
Support and involvement
4 Project organisation involves the right participants from EU Member States and international organisa- tions if relevant
3 Project organisation has satisfactory representation of central participants from some EU Member States and international organisations if relevant
2 Project organisation lacks to some degree representation of central participants from some EU Member States and possibly international organisations
1 Project organisation lacks to a high degree representation of central participants from some EU Member States and possibly international organisations
Sustainability
4 Project activities/results are sustained after the end of EC co-funding, e.g. by policy implementation, fol- low-up projects and by making results available on website or similar activities
3 Project activities/results are to some extent sustained after the end of EC co-funding by follow-up projects and by making results available on website or similar activities
2 Project activities/results are to a limited extent sustained after the end of EC co-funding by making results available on website or similar or similar activities
1 Project activities/results ended with the EC co-funding or shortly hereafter
The scores for each of the evaluation criteria are presented in the following chapters, accompanied with brief rationales for the scores given to the different case studies. These scores across the different evaluation criteria are then ana- lysed in Chapter 12 as part of the evaluation conclusions and recommendations. The evaluation comprises six case studies; two for each PHP strand (Table 5-8). The case study areas were selected by DG SANCO.
The case studies include a general assessment of the whole case study area and an in-depth study of a selection of PHP funded projects/activities made by in- dependent PHP experts. The selection is based on elements such as size and aim of the project as well as geographical scope and participation of the pro- jects. The approach to the case studies was an in-depth desk study and inter- views on specific projects funded by the PHP. According to relevance, inter- views were conducted with persons such as project coordinators, public ad- ministrations, experts in the field, and at least one independent expert in the field of the given case study area. The interviews were performed as face-to- face interviews, focus group interviews, and/or phone interviews. The docu- ment screening included project documents and other relevant documents in the field of the case study area related to the EU, national and international levels. Below is a description of the case study areas and the projects selected for the case studies. The selected projects have been approved by DG SANCO. Six case study areas
Table 5-8 Case study areas per strand and selected projects/activities
PHP Strand Case study area Selected projects/activities
Health information
Comparable Euro- pean information
Closing the Gap – Reducing Premature Mortality. Baseline for Monitoring Health Evolution Following Enlargement
Better Statistics for Better Health for Pregnant Women and Their Babies: European Health Reports
Creation & support of knowledge man- agement networks
European Surveillance of Congenital Anomalies (Phase 3) Rare Diseases Portal
Health threats
Organs European Living donation and public Health JACIE - Joint Accreditation Committee ISCT EBMT
Chemical threats
The Public Health Response to Chemical Incident Emergencies
Mass casualties and Health-care following the release of toxic chemicals or radioactive materials - MASH
Health de- terminants
HIV/AIDS
European Centre AIDS & Mobility A&M
European Network for Transnational AIDS/STI Prevention among Migrant Prostitutes
Addictions - drugs European Network on Drugs and Infections Prevention in Prison Democracy, Cities & Drugs II
Health information
The case study areas chosen under the health information strand are "Compara- ble European information" and "Creation and support of knowledge manage- ment networks".
Case study area 1: Comparable European information
The following two projects were selected for in-depth study in the case study area "Comparable European Information":
• "Closing the gap - reducing premature mortality. Baseline for monitoring health evolution following enlargement" (2003121). The project aimed at closing the gap in premature, preventable morbidity, disability and mortal- ity between the "old" and the "new" Member States as well as applicant countries by creating a baseline for monitoring evolution of preventable, premature mortality risk factors following enlargement. Special regard was given to improving the health of working population and diminishing ine- qualities in access to health. The project was the only project submitted (in 2003) by an applicant country. The project was completed in 2008, follow- ing an extension from 36 to 41 months, and is therefore suitable to provide Case study area 1:
Comparable Euro- pean information
evidence on the issue of sustainability. The EC contribution was EUR 584,580.
• "Better statistics for better health for pregnant women and their babies: European health reports" (2007114). The project objective was to produce and disseminate a European Perinatal Health Report based on data col- lected in 2006 in all EU countries, which includes policy-relevant analyses of maternal and child health outcomes, care provision, inequalities and mi- grant health. The project aimed also to develop an Action Plan for Sustain- able Perinatal Health Reporting with recommendations about the mission, structure, operation and partners of an information network. The project is an example of a relatively short (duration of 18 months) and focused pro- ject building on past investment in health information (EURO-
PERISTAT). The EC contribution was EUR 149,987.
Case study area 2: Creation and support of knowledge management networks
The following two projects were selected for in-depth study in the case study area "Creation and support of knowledge management networks":
• "European surveillance of congenital anomalities (phase 3)" (2003219). The project objective is to provide epidemiologic information on congenit- al anomalies in Europe. Furthermore, the project aimed to co-ordinate the establishment of new registries throughout Europe collecting comparable, standardised data; to co-ordinate the detection and response to clusters and early warning of teratogenic exposures; to evaluate the effectiveness of primary prevention, in particular folic acid supplementation; to assess the impact of developments in prenatal screening; and to provide an informa- tion and resource centre and ready collaborative research network to ad- dress the causes and prevention of congenital anomalies and the treatment, care and outcome of affected. The project provides a good example of a very large set of associated partners and thus insights into the establish- ment of cross-country partnerships. The high institutional profile of the partners involved also provides an opportunity to explore the role of a European project vis-à-vis national and regional authorities. Furthermore, despite being the third project in a series, it promised to be innovative and develop new knowledge. The EC contribution was EUR 812,074.
• "Rare diseases portal" (2006119). This project is not particularly innova- tive in terms of contents, but it promised to exploit the potential of ICTs (Information and Communications Technology) to make the information on Orphanet available to a broader public. It is also interesting as it dove- tails with other priorities highlighted by DG SANCO, such as harnessing the potential of e-health to provide peer-reviewed information and estab- lishing databases of European reference centres, especially in high-cost ar- eas such as the treatment of rare diseases. The EC contribution was EUR 960,000.
Case study area 2: Creation and support of knowledge man- agement networks
Health threats
The case study areas chosen under the health threat strand are "Organs" and "Chemical threats".
Case study area 3: Organs
The following two projects were selected for in-depth study in the case study area "Organs":
• "European living donation and public health" (2006211). This project aimed to reach consensus on European common legal and ethical standards regarding protection and registration practices related to living organ do- nors in order to guarantee the health and safety of these donors. This is an important but also a controversial field and may set an example for other consensus efforts at the European level. The EC contribution was EUR 524,893.
• "JACIE - Joint Accreditation Committee ISCT EBMT" (2003208). The aims of the project were to provide vital impetus to the JACIE16 Pro- gramme and ensure its integral role in standard setting, inspection and ac- creditation for health institutions and facilities involved in haematopoietic stem cell collection, processing and transplantation in Europe. The project contributed to a special, but transferable field of European added-value by showing how to reach European standards by means of a centralized ad- ministration, an online IT system, and training courses. It explicitly aimed at long-term results of accreditation and standards. Furthermore, it aimed to include eight new applicant countries and four more Member States. The EC contribution was relatively small with EUR 167,526, raising inter- esting questions about effectiveness.
Case study area 4: Chemical threats
The following two projects were selected for in-depth study in the case study area "Chemical threats":
• "The public health response to chemical incident emergencies (CIE Tool- kit)" (2007205). The aim of this project was to facilitate the rapid and ef- fective response to acute chemical incidents by providing a source of rele- vant material. The project is a specific predecessor project in the same field and is focused on a clearly defined outcome; a toolkit and a manual for training on chemical incident emergencies, thus providing a good opportu-
16 JACIE is a non-profit body established in 1998 for the purposes of assessment and ac-
creditation in the field of haematopoietic stem cell (HSC) transplantation. JACIE's primary aim is to promote high quality patient care and laboratory performance in haematopoietic stem cell collection, processing and transplantation centres through an internationally rec- ognised system of accreditation.
Case study area 3: Organs
Case study area 4: Chemical threats
nity to analyse the preconditions of cross-country dissemination of PHP re- sults. The EC contribution was EUR 697,431.
• "Mass-casualties and Health care following the release of toxic chemicals or radioactive materials (MASH)" (2007209). This project had a wider fo- cus, including emergencies with radioactive material. Thus, the objective of MASH was to improve competence and capability to deal with patients exposed to toxic chemicals or to radioactive materials. The project zooms in on the primary care process, and organisational measures are related to direct healthcare provision. The EC contribution was EUR 799,967.
Health determinants
The case study areas chosen under the health determinant strand are "HIV/AIDS" and "Addiction - drugs".
Case study area 5: HIV/AIDS
The following two projects were selected for in-depth study in the case study area "HIV/AIDS":
• "European centre AIDS and mobility (A&M)" (2003303). The general aim of the project was to develop and exchange solutions to handle specific is- sues relating to the vulnerability of mobile and migrant populations with a specific focus on young people to HIV/AIDS. The project had 25 partner countries and a wide composition of the different target groups (experts and stakeholders from GOs and NGOs, mobile (young) migrants). The project lifetime ended in 2006. These characteristics of the project provide the opportunity to study the European added value to the lead partner country and the partner countries, the involvement of the different target groups and the sustainability of the project outcomes. The EU Contribution of the PHP was EUR 1,559,334.
• "European network for transnational AIDS/STI prevention among migrant prostitutes (TAMPEP)" (2004320). The overall purpose of this project was to further develop the models of good practice and tools to support the planning and implementation of coordinated and comprehensive health promotion and social care services for migrant sex workers and trafficked women in all EU countries, through an ongoing pan-European cooperation and through expanding the network up to 24 partner countries by including the new EU member countries. The end of the project lifetime was in 2006. Thus, it possible to study the added value of the project to the lead partner and the partner countries and to evaluate the sustainability of the outcomes. The EC contribution was EUR 595,776.
Case study area 5: HIV/AIDS
Case study area 6: Addiction - drugs
The following two projects were selected for in-depth study in the case study area "Addiction - drugs":
• "European network on drugs and infections prevention in prison" (2003308). The objectives of the project were to collect, compare and widely distribute data and information on infectious diseases, drug use and its consequences and related prevention activities in prisons in the EU Member States; to develop with all involved partners common and effec- tive epidemiological and sociological research tools, in order to monitor the epidemiology of drug related health threats and evaluate prevention approaches; to promote exchange of experience and information in the above area in the Member States; to promote and sustain the implementa- tion of effective harm reduction and abstinence oriented programmes (in the context of current national legislation); and to formulate recommenda- tions for primary and secondary prevention of infectious diseases and other drug related health and social problems. The project involved the devel- opment and maintenance of a European-wide network (24 Member States), combining the activities of three existing networks by demonstrating the European reality in the everyday life in the field of prevention of infections diseases and drug use in prisons. The project lifetime ended in 2007. These characteristics of the project provide the opportunity to make an in-depth evaluation of the European added value to the lead partner country and partner countries and of the sustainability of the project outcomes. The EC contribution was EUR 1,895,223.
• "Democracy, cities and drugs II" (2007306). The aim of the project was to help support EU cities develop local, partnership based drug policies in- volving the relevant stakeholders – local authorities, health services crimi- nal justice services, communities, including visible minority ones, and drug service users. The project was based on three pillars: a) EU wide ex- perimental network of more than 20 partner cities or regions, b) four na- tional networks of EU Member States involving 24 cities, and c) five the- matic working groups. The project was a follow up project of "Democracy, Cities & Drugs I” (2005– 2007). The characteristics of the project provide an opportunity to make an in-depth evaluation of the European added value to the lead partner country, the partner countries and cities and the sustain- ability of the project outcomes, including reflections on the sustainability of the outcomes of project "Democracy, Cities & Drugs I". The EC contri- bution was EUR 900,000.
In relation to each case study, the co-ordinator of each project as well as experts in the field was interviewed (see Table 5-9). Furthermore, the National Focal Points from each of the host countries were contacted (see Table 5- ).
Case study area 6: Addiction - drugs
Interview with co- ordinators, experts and some national Focal Points
Table 5-9 Interviewees in relation to case studies PHP
Strand
Case study Area
Selected activities Interviewees Health in-
formation
Comparable European information
"Closing the gap - reducing premature mortality. Baseline for monitoring health evaluation following enlarge- ment"
Project Country Coordinator, Tit Albreht, Slovenia.
Head of Laboratory of General Epidemiology, Carlo La Vecchia. Project Co-ordinator, Marta Manzuk, Poland.
Project Leader, Witold Zatonski, Poland.
"Better statistics for better health for pregnant woman and their babies: European health report"
EURO-PERISTAT Project Leader and original Project Leader, Gerard Breart, France.
Marina Cuttini, Epidemiology Uni.
Independent expert, Viviana Mangiaterra, WHO. Report Coordinator, Ashna Mohangoo, The Netherlands. EURO-PERISTAT Scientific Coordinator, Jennifer Zeitlin, France. EURO-PERISTAT Project Coordinator, Meagan Zimbeck, France. Creation & support of knowledge management networks "European Surveillance of Congenital Anomalies (Phase 3)"
Full Member Registry Leader and Co-Chair of the Coding & Classifica- tion Committee, Ingeborg Barisic, Zagreb.
Expert Member for Drugs Surveillance Working Group, Maurizio Clementi, Italy.
EUROCAT Project Leader, Helen Dolk, UK.
Independent expert, Pierpaolo Mastroiacovo, Director of WHO World Craniofacial Anomalies Registry, Italy.
Applicant Member Registry Leader, Ivan Zatsepin, Belarus. "Rare disease portal" Director of Research, Ségolène Ayme, France.
Jean-Jacques Cassiman (member of research team “Human Mutations and Polymorphisms Section”), Belgium.
Scientific Director, Bruno Dallapiccola, Italy.
Independent expert, Petra Wilson, Senior Director of the European Healthcare Team.
Health threats
Organs "European Living Donation and public health"
Project coordinator, Dr. Assumpta Ricart, Barcelona Independent expert Prof. Dr. h.c. Eckhard Nagel, Germany