Desk study

The desk study is based on available programme documents, including e.g. programme decision5, the White Paper "Together for Health"6, annual work plans7, the interim evaluation8 and report from the Court of Auditors9. Project documents, including project abstracts, final reports and documentation from project websites, are included in the desk study performed as a part of the case studies.

Furthermore, the evaluation draws on the results of a portfolio analysis con- ducted by COWI10 concerning the coverage of PHP objectives and priorities by activities funded.

5.2.2 E-survey

In all, 1,242 respondents were invited to participate in the e-survey. Only exter- nal stakeholders and beneficiaries received an invitation to participate in the e- survey. The recipients of the e-survey are further grouped according to their role, and the different groups received targeted questions (see Table 5-3). The initial sample of respondents of the survey thus comprises all beneficiaries of the PHP (proposals and direct grants) and the numerous stakeholders involved in the PHP (e.g. programme committee members, working parties, forums, etc.). This means, that the sample is assumed to consist of the complete popula- tion of potential respondents. The stakeholders were initially divided into three groups: beneficiaries, other stakeholders (representing the six groups mentioned in Table 5-2), and a group representing stakeholders, who were identified as both beneficiaries and other stakeholders.

In this report, beneficiaries are defined as stakeholders, who received funding from the PHP, while the group of other stakeholders covers stakeholders, who participated in the PHP, but who did not receive funding.

5 _{Decision No 1786/2002/EC of the European Parliament and of the Council of 23 Septem-}

ber 2002 adopting a programme of Community action in the field of public health (2003- 2008).

6 _{Communication from the Commission to the Council, the European Parliament, the Eco-}

nomic and Social Committee of the Regions on the Health Strategy of the European Com- munity. White Paper - Together for Health: A Strategic Approach for the EU 2008-2013.

7 _{Work plan 2003-2007}

8 _{Oortwijn, W., ling, T., Mathijssen J., Lankhuizen, M., Scoggins, A., Stolk, C. and Cave J.}

Interim Evaluation of the Public Health Programme 2003-2008

9 _{The European Union's Public Health Programme (2003-07): An Effective Way to Im-}

prove Health?

10 _{COWI. Portfolio analysis and evaluation of the health project mapping 2003-2009 exer-}

cise - final report

Programme and pro- ject documents

Portfolio analysis

Number of invited participants

Table 5-3 Overview of e-survey

Stakeholder group e-survey question Number of questions Evaluation criteria

International organisations 7-11, 15, 22, 32-34, 38, 40 13 Effectiveness, Sup- port/Involvement Public administrations 7-11, 15, 22, 31, 32-34, 35,

36, 37, 38, 40

16 (17) Effectiveness, Sup- port/Involvement Interest groups 7-11, 15, 22, 32-34, 38, 40 13 Effectiveness, Sup- port/Involvement Economic opera-

tors/private bodies

7-11, 15, 22, 32-34, 38, 40 13 Effectiveness, Sup- port/Involvement

Experts 7-11, 15, 22, 32-34, 38, 40 13 Effectiveness, Sup-

port/Involvement Other body involved in the

PHP (e.g. university) 7-11, 15, 22, 32-34, 38, 40 13 Beneficiaries 4-6, 12-14, 16, 17, 18-20, 23-30, 32-35, 36, 39, 39a, 41 24 (27) Effectiveness, Sup- port/Involvement, Sustain- ability

Note: The numbering of the questions corresponds with the numbering of the e-survey questions in appendix 3.

Some of the statistical data for this report were collected through an online questionnaire in the period from 6 May to 7 June 2010. All tables and figures representing the results of the e-survey contain a note indicating data source. E- mail addresses of possible respondents were provided by DG SANCO, and all identified respondents received an invitation to fill in the questionnaire. Non- responders received up to two reminders. The first reminder was sent approxi- mately 1.5 weeks after the initial invitation, and the second reminder was sent approximately one week later.

All questions to both beneficiaries and other stakeholders were collected in one questionnaire, but the respondents only answered questions relevant to their affiliation (beneficiaries or other stakeholders). The questionnaire including a schematic overview of the three respondent groups and the questions belonging to them is presented in appendix 3. The online questionnaire was dynamically programmed, meaning that respondents' answers determined the subsequent questions. The main advantage of this approach is that respondents do not have to "find their way" through the questionnaire and are only presented with rele- vant questions.

The number of respondents in each group is shown in Table 5-4. Of the 1,242 invitations, approximately 130 e-mails11 were immediately rejected because of faulty addresses, corresponding to 10.5 per cent of respondents. It is assumed that this error percentage is identical in all respondent groups, and thus 10.5 per cent of respondents in each group are assumed not to have received the e-mail. This gives the net frequency as presented in the table. These frequencies are used below to calculate net response rates. The groups 'beneficiaries' (291) and 'beneficiaries and other stakeholders' (13) are funded respondents; whereas the group 'other stakeholders' (938) are non-funded respondents.

Nine respondents received the invitation in several e-mail accounts. It seems reasonable to assume that other respondents experienced a similar problem. As the contact information of beneficiaries and other stakeholders did not contain the names of all respondents, this was inevitable. Furthermore, seven respon- dents (all from the group of other stakeholders) did not feel it relevant to an- swer the questionnaire, primarily due to lack of knowledge of the PHP.

Table 5-4 E-survey: number of respondents

Gross frequency Net frequency

Beneficiaries 291 260

Other stakeholders 938 840

Beneficiaries and other stakeholders 13 12

Total 1242 1112

304 funded respondents12 were invited to participate. Of these, 93 responded, which is a gross response rate of 30.6 per cent corresponding to a net response rate of 34.2 per cent13 (see Table 5-5). 951 not-funded respondents14 were in- vited to participate. Of these, 236 responded, which is a gross response rate of 24.8 per cent corresponding to a net response rate of 27.7 per cent15.

11 _{Since the e-survey of tasks1 and 2 was executed as an integrated questionnaire, a total of}

1330 invitations were sent to respondents with questions concerning task 1, task 2 or both tasks. Of these 1330 invitations, 139 emails were immediately rejected, which corresponds to 10.5 per cent of the total amount of emails. It is assumed that this error percentage is identical in all respondent groups as well as respondents to tasks 1 and 2 respectively. Of the 1,242 invitations sent to respondents to task 1, an estimated 130 emails were rejected (10.5 per cent of 1,242).

12 _{291 + 13 = 304}

13 _{Net respondents presented with beneficiaries' questions are 260 + 12 = 272.} 14 _{938 + 13 = 951}

15 _{Net respondents presented with beneficiaries' questions are 840+12=852.}

Number of respon- dents

Table 5-5 E-survey: response rates

Gross response rate Net response rate

Beneficiaries 30.6% 34.2%

Other stakeholders 24.8% 27.7%

The population of the survey is unknown in the sense that the distribution of stakeholders (funded as well as non-funded) is not known. For instance, the distribution of non-funded stakeholders among the different stakeholder groups as presented in Table 5-3 (international organisation, public administration etc.) is not known for the entire population of non-funded stakeholders. This applies to all other distributions of stakeholders, e.g. home country and target group of the activity.

The unknown population is a source of uncertainty in terms of the representa- tiveness of the collected data, since it is not possible to determine whether the collected data are representative of the complete population. Further, it is not possible to weigh the results, since weights would have to be based on the real distribution of the population. Bias because of the unknown population can oc- cur if one of the respondent groups is overrepresented and dominates the out- come of the aggregated responses. Since it is not possible to determine the exis- tence of such bias in the e-survey, it is assumed in the presentation of the re- sults that the collected data set is representative.