Chapter 9: Quality and Evaluation
3. Evaluation methodologies
Questionnaire:
The questionnaire is a commonly used methodology. Questionnaires may follow courses or conferences, accompany newsletters or other project products or feature as ‘pop-ups’ on a project website. However, the return rates from questionnaires may be extremely low.
Involvement: Please circle in column 1 the level of involvement you should have (according to your agreement with the co-ordinator) in
each p activity below and circle in column 2 the level you actually have.
Scale: 0 = none, 1 = very low, 2 = low, 3 = moderate, 4 = high, 5 = very high.
Overview of typical network activities 1: Should have 2: Actually have
Collection of relevant materials 0 1 2 3 4 5 0 1 2 3 4 5
Evaluation / assessment of relevant materials 0 1 2 3 4 5 0 1 2 3 4 5 Production / writing of reference materials 0 1 2 3 4 5 0 1 2 3 4 5
Doing research in this field 0 1 2 3 4 5 0 1 2 3 4 5
Piloting / testing methods / material 0 1 2 3 4 5 0 1 2 3 4 5
Organising conferences 0 1 2 3 4 5 0 1 2 3 4 5
Organising training days 0 1 2 3 4 5 0 1 2 3 4 5
Being a speaker, giving presentations 0 1 2 3 4 5 0 1 2 3 4 5 Sharing experience within the network partnership 0 1 2 3 4 5 0 1 2 3 4 5
Communicating within the network 0 1 2 3 4 5 0 1 2 3 4 5
Working with specific target groups 0 1 2 3 4 5 0 1 2 3 4 5 Training of network actors (partners, members, target groups) 0 1 2 3 4 5 0 1 2 3 4 5 Passing on information in your own institution 0 1 2 3 4 5 0 1 2 3 4 5 Providing support to other projects in this thematic field 0 1 2 3 4 5 0 1 2 3 4 5 Creating visibility of the network beyond its participants 0 1 2 3 4 5 0 1 2 3 4 5 Awareness raising, campaigns in the field 0 1 2 3 4 5 0 1 2 3 4 5 Representing interests and advocacy 0 1 2 3 4 5 0 1 2 3 4 5
Contacting policy makers 0 1 2 3 4 5 0 1 2 3 4 5
Dissemination / valorisation 0 1 2 3 4 5 0 1 2 3 4 5
Policy development 0 1 2 3 4 5 0 1 2 3 4 5
Validation, recognition, integration of innovation into existing systems 0 1 2 3 4 5 0 1 2 3 4 5
Curriculum development 0 1 2 3 4 5 0 1 2 3 4 5
Creation of a European added value 0 1 2 3 4 5 0 1 2 3 4 5 Calling upon your own (existing) networks 0 1 2 3 4 5 0 1 2 3 4 5 Developing and extending the network 0 1 2 3 4 5 0 1 2 3 4 5 Interacting with other projects and networks 0 1 2 3 4 5 0 1 2 3 4 5
Generating new projects 0 1 2 3 4 5 0 1 2 3 4 5
Mainstreaming: integrating outcomes in regular curricula: 0 1 2 3 4 5 0 1 2 3 4 5
Creating a network culture 0 1 2 3 4 5 0 1 2 3 4 5
Box 6: Exemplar self-evaluation sheet: Degree of involvement in network activities (From: Bienzle/Gelabert/Jütte/Kolyva/Meyer/Tilkin (2007), The Art of Networking, p. 103)
Quality
and
Evaluation
Chapter
9
ways which encourage reflection and problem solving. One very important ground rule is for members to avoid simply telling their own stories as a ready means of providing a solution. In an international project it is most unlikely that working circum- stances are the same or even similar so it is important for group members to ask questions as part of the procedure.
This process can be summarised in the GROW model (Box 7).
GROW model (or process):
This is a technique for problem solving or goal setting which is used extensively in the business arena but has many applica- tions elsewhere. The model can also be used by any group who are all working on the same problem or goal. No one person can be identified as the originator but Graham Alexander, Alan Fine and Sir John Whitmore who are all well known in the world of coaching, made significant contributions.
http://en.wikipedia.org/wiki/GROW_model, March 2010
Review meetings:
Review meetings may be used to monitor the progress of the project but also to examine the findings from formative evalu- ation work (see Section 1). Review meetings may be timetabled to coincide with important project milestones such as reporting deadlines and may be face-to-face or virtual. To make them a success they should:
Most Significant Change (MSC):
This is a form of participatory monitoring and evaluation that can be planned to occur throughout the project to provide data on both impact and outcomes. It involves the collection of ‘stories’ about significant changes brought about by the proj- ect. These stories will usually come from structured interviews with members of the target groups who are asked the same questions across the project partnership. When the stories have been collected they are analysed by an independent panel whose members identify the most significant factors in the stories. MSC has become a well used evaluative methodology in international projects.
See: www.mande.co.uk/docs/MSCGuide.pdf
Peer Mentoring:
There may be occasions in the life cycle of a project where Peer Mentoring has a place as an evaluative strategy to explore issues and challenges facing the project more probably in terms of the project’s processes rather than its outputs. It works well in bringing together workers who are comparatively isolated (either face-to-face or virtually) to explore issues and challenges. It is therefore part of the self-evaluation process based on Action Learning:
Action Learning:
‘…is a process underpinning the belief in individual potential: a way of learning from our actions, and from what happens to us, and around us, by taking the time to question, under- stand and reflect, to gain insights, and consider how to act in future.’
Krystynia Weinstein (1995): Action Learning – A Practical Guide for Managers
Peer Mentoring works best if it follows a defined structure: Project workers agree to meet collaboratively on a regular
■ ■
basis
The group agrees ground rules which include complete open-
■ ■
ness and mutual trust
Members take turns in presenting their individual situations
■ ■
complete with highlights, challenges and problems Action points are discussed and agreed
■ ■
The overall quality of the project is improved
■ ■
An important part of the process is the way in which the discus- sion is structured so that groups (focussing on each individual member in turn) give advice and offer options for action in
The GROW model
G Goal What is it that we are aiming to achieve in this aspect of our work? R Reality What is the reality of the situation? What feed- back have we received so far about the pro- ject? What are the issues and concerns identi- fied through the evaluation?
O
Obstacles and Options
What are the obstacles to change? What are the recom- mendations? Are they useful?
What are the resources we have available? W What, when, whom? What exactly are we going to do differently? Who is going to do it? When? Box 7: The GROW model
Quality
and
Evaluation
Chapter
9
The important difference from a tightly structured event is that participants create and manage their own agenda around a number of themes of strategic importance to the project. In terms of evaluation this again implies flexibility and resourcefulness on the part of the project co-ordinator plus a familiarity with a number of evaluation tools and techniques.