Whose interests are being served?

Crucially, the question of whose interests are being served in any particular evaluation study needs to be seen as a political issue:

"Evaluation research should be understood as inherently political...Whilst most commentators recognise that evaluation operates within political constraints, we go further and suggest that evaluation itself is socially constructed and politically articulated."

(Taylor and Balloch, 2005: 1)

Whilst in many instances evaluations are primarily intended to meet the needs of their funders, and therefore decisions around purpose and even methodology will

55 be largely dictated in advance, there are no such external constraints in the case of this particular study. Hence, in making decisions about the approach that this study will take, it is important to consider whose interests may be served. As Greene argues, evaluators have no choice but to adopt some form of political position:

"social program evaluators are inevitably on somebody's side and not on somebody else's side. The sides chosen by evaluators are most importantly expressed in whose questions are addressed and, therefore, what criteria are used to make judgements about program quality." (Greene, 1997: 25)

Clearly, these questions of whose interests are being served in any particular evaluation study overlaps significantly with the previous discussion about the purposes of evaluation, since different people may have different interests in terms of formative, summative or generalisable findings. As Weiss (1998: 29) suggests,

"Expectations for the evaluation generally vary with a person's position in the system", with policy-makers being more likely to have an interest in summative findings, whilst practitioners may have more concern for formative aspects of evaluation. Whilst, as suggested above, the distinctions between these different forms of evaluation may be quite blurred in practice (and indeed, there is

considerable dubiety with regard to who ‘makes’ policy), there is nevertheless an important question with regard to whose interests are represented in the design and implementation of any evaluation study. Moreover, this goes beyond the putative policy-maker/practitioner divide to include those individuals who are most affected by any particular policy or programme:

"Evaluation should not privilege one set of beliefs over others. It should not take seriously only the questions and concerns of study sponsors rather than those of staff and clients whose lives may be even more affected by their experiences in the program." (Weiss, 1998: 100)

Thus the question of whose interests are being served through an evaluation is intimately connected to the issue question of who should actually be involved in

56 the evaluation process, since the suggestion is that direct involvement in

evaluation enables people to shape the questions asked and influence how the findings are used.

Although the situation in practice is inevitably more complicated, it is useful to characterise participation in evaluation along a spectrum, from one extreme at which the researcher controls the entire study, through to the other at which all aspects are developed by a wide group of stakeholders. Underpinning positions along this spectrum are different perspectives on the need for ‘objectivity’, versus the value of hearing the voices of those involved in a policy or programme. Thus Scriven (1997) outlines the archetypal version of the emphasis on objectivity, even advising against interviews with staff unless absolutely necessary for data

collection, on the basis that any personal interaction will introduce 'bias'. At the other end of the spectrum, Balloch and Taylor argue that:

"no evaluation of any worth can afford to neglect the views of the central actors, be they young people, residents of a neighbourhood renewal area, parents or others. These groups should take

precedence over practitioners and the evaluators themselves."

(Balloch and Taylor, 2005: 250)

Hence the suggestion is that hearing the voices of those who are the targets or recipients of a policy is essential to understand the processes involved and the reasons for their responses. Indeed, advocates of ‘user-led’ research take this one stage further, maintaining that there is a basic incompatibility between the

objectivity and distancing that researchers such as Scriven would promote, and complete understanding of how a policy works (Beresford and Evans, 1999).

This debate between objectivity and participation also filters through into the utilisation of evaluation findings. On the one hand, there is the view that ‘objective’

and ‘scientific’ discourse may be privileged by some decision makers (Humphries, 1997), or as Patton critically expresses it:

"while I believe that the paradigms debate has lost its acerbic edge among most evaluators, many users of evaluation – practitioners,

57 policy-makers, program managers and funders – remain mired in the

simplistic worldview that statistical results (hard data) are more scientific and valid than qualitative case studies (soft data)." (Patton, 1997: 267)

On the other hand, a number of authors suggest that the involvement of both practitioners and affected members of the public in an evaluation may be practically important when it comes to utilisation, since they will have some commitment to seeing the research findings employed to influence future policy (Clarke and Dawson, 1999: 18; Gregory, 2000: 180; Weiss, 1998: 100), as indeed will many researchers.

These debates about who participates in an evaluation and the interests it serves have a particular resonance for this study for three reasons. Firstly, the subject of the research is community participation, so it might seem methodologically and politically incongruous, to say the least, if the research approach did not involve some degree of participation. Secondly, starting from my own political positioning and the original objectives for the project, outlined in the previous chapter, I would suggest that communities which become ‘subjects’ of the research are entitled to expect direct benefits from the process and therefore need a participative role within the project. Indeed, even ignoring the political considerations, it could be practically difficult to study community participation in detail without involving participants in the process, since community activists are likely to have

expectations of participation in anything that relates to them (Beresford, 2005).

And thirdly, since the implementation of community participation policy is

inherently shaped by the actions and interactions of communities and public sector agencies on the ground, the utilisation of findings needs to operate well beyond the narrow circle of official policy-makers.

These considerations highlight the importance of clarity regarding who might potentially benefit from this study and what forms of knowledge are likely to be important to them, in order to inform the choice of methodology. As outlined in the previous chapter, the admittedly ambitious research aims are to attempt to provide useful findings about ‘what works’ in community participation policy and practice which may be useful for communities, local practitioners, and local and national

58 policy-makers, as well as contributing to evaluation and community participation scholarship. Clearly for communities and practitioners directly involved in the study there is likely to be value in formative feedback through the process as well as summative findings of impact, to the extent that these are possible within the timescale and resources of the project. Alongside this, the aim to develop more widely usable findings requires an emphasis on generalisable knowledge which may be of use to all the different audiences.

As the previous section argued, the suggestion from Blamey and Mackenzie (2007) that it may be possible and potentially fruitful to combine Theories of Change and Realist Evaluation approaches in one study, may offer an approach with the potential to meet all of these goals. Clearly there is a risk in this approach, since no evaluation can hope to answer every question and satisfy every

interested party (Weiss, 1998: 33; Gambone, 1998), but it seems like a risk worth taking in order to meet as many goals as possible and to test the methodological waters. Taking this suggestion forward, it is important to consider in a little more detail how these methodologies relate to the debates about participation and how they might serve the multiple interests targeted by this particular study.

As noted earlier, there is a significant difference between these two theory-based approaches in terms of participation, since ToC evaluators emphasise the

importance of collaboration in developing theories of change (Anderson, 2005;

Connell and Kubisch, 1998), whilst Pawson and Tilley argue that the theories in Realist Evaluation should primarily come from the evaluator and her expert knowledge, with policy makers, practitioners and (possibly) service users being interviewed only to confirm, falsify or refine those theories (Pawson and Tilley, 1997: 159). For Gregory, this emphasis on the expert position of the evaluator places limitations on RE approaches:

"Consequently, the knowledge produced by Realist Evaluation is critically restricted as a result of the methodology's failure to embrace the range of knowledge held by people except as represented in the literature and/or as interpreted by practitioners/researchers."

(Gregory, 2000: 192)

59 Notably, however, Pawson and Tilley (1997: 160-2) do recognise the importance of the ‘knowledgeability of the social actor’ and highlight the ways in which

subjects of social programmes, practitioners and policy makers may each bring particular knowledge of contexts, mechanisms and outcomes, which will not be immediately available to the evaluator. And whilst Pawson and Tilley (1997: 65) emphasise the importance of getting ‘beneath’ the appearance of things to identify underlying mechanisms, their consequent assumption that the evaluator is best placed to do this seems to be based largely on the rather questionable assumption that evaluators bring a different level of expertise from their experience of other evaluations and knowledge of other studies. Moreover, Pawson and Tilley’s objections to participative methods, based on their ‘division of expertise’ between subjects, practitioners and evaluators start to break down in the context of

studying community participation. Whilst community members may be conceived of as subjects from one perspective, they are also practitioners in participation processes and often evaluators of their own projects as well.

Hence, it is reasonable to argue that the notion of combining ToC and RE

methodology need not be undermined by these differing attitudes to participation.

Rather, it seems sensible to suggest that both methodologies may be applicable with different levels of participation in different situations, and that the level and form of participation needs to be determined by the political considerations of whose interests are being served. In the instance of this particular study, it should be possible to combine a significant degree of participation with both approaches, using ToC methodology to provide formative feedback and combining it with RE methodology to develop the more summative findings.

Moreover, the combination of approaches has the potential to provide knowledge in different forms to meet the needs of different audiences. As Weiss (1998: 68) suggests, the ‘stories’ provided by Theories of Change approaches may provide accessible summative findings for a range of people, from policy-makers to community members, using the power of the narrative form to convey important learning. Whilst a Realist Evaluation element to the study may offer middle-range theory which can be more readily generalised to other contexts (Blamey and Mackenzie, 2007).

60 In addition, the combination of the two methodologies may help to manage some of the challenges relating to power differentials within evaluation. A number of ToC practitioners have pointed to the challenges in practice of dealing with different and often incompatible theories, which may be articulated by individuals or groups with different levels of power in the process (Connell and Kubisch, 1998: 20;

Granger, 1998: 223), although Mason and Barnes suggest that the process of identifying different theories may in itself be a strength of the approach, and that a Theories of Change evaluation can and should consider different theories:

"working with difference is an important and challenging aspect...Seeking agreement at the expense of clarifying the contributions that different approaches and understandings of complex social problems can make to developing creative solutions is unlikely to be the best way forward." (Mason and Barnes, 2007:

161)

In this respect, combining ToC methodology with RE could potentially be

productive, since the examination of particular mechanisms within a broad theory of change may help to adjudicate between different perspectives.

Thus it is argued here that a combination of ToC and RE approaches may be able to offer an element of formative evaluation which is of direct benefit to the

communities involved, well-developed theories of change which can play a

summative role for a range of stakeholders, and Realist CMO configurations which can enhance the generalisability of the ToC findings, as well has helping to

manage conflicting theories.