Methodological, technical and other issues encountered

From a variety of issues encountered, I describe five methodological and three technical issues, as visualised in Table 5.16 (overleaf):

189

Table 5.16: Methodological and technical issues encountered

5.4.1.1 Methodological issues

The first methodological issue faced concerns the foci of the research. For example, while the initial literature reviewed appears to indicate an exploration focused on the construction of the evaluation process, the advance of the literature search combined with a pilot study confirmed the need to focus on ‘practice’ as the main case, rather than ‘process’. This also meant a redefinition of the literature review and the general approach to data collection and analysis. A second methodological issue was the decision of positioning this thesis, primarily in the field of evaluation, rather than in the field of DE. This decision became clear during the analysis, where two bodies of data were emerging as distinct, one was the EP, as the phenomenon under investigation and the other the domain of DE, as a research setting, serving as an interesting context to

190

observe EP over time (Patton, 2002). As a PhD thesis has specific limitations, I knew that I had to narrow my focus once more and, in reflecting about the initial motivation to conduct research, I knew that I wanted to position this study within the EP domain (again, a matter of ‘researcher loyalty’, described in section 5.2.4). This decision made sense, as I have been connected to evaluation as a practitioner and would like to pursue my career in this field.

A third methodological issue was about the difference of conducting and researching evaluation. This issue arose during the data collection when I felt the constant need to reassure participants that I was neither judging their competence nor their organisation’s performance, as the aim of my study was to research how evaluation has been practised within a DE domain. So I was researching EP and not conducting an evaluation of the YP. To overcome this challenge, before starting each interview, I read an introduction of the research to the participants, reminding them about this difference.

A fourth methodological issue was the apparent limitation that arose from the use of a SPT lens that focuses on an individual, rather than in a collective practice. Due to ethical reasons I analysed EP from a group’s perspective (embedded cases) for two reasons: the first concerns the case study design regarding the presentation of the composite profiles of participants (described in section 5.3.5), which allowed the reader to make connections with their own setting (Simons, 2009). Moreover, I believe this intention was achievable from a recollection of accounts, in which I drew attention to some aspects rather than others. However, the reader holds the freedom to pinpoint a particular nuance, without disclosing the individuals’ anonymity. As such, the thesis is interested

191

in the collective practice of evaluation within an organisational setting, which was accomplished through a SPT view of evaluation, from each group, and as a whole. The second reason to justify my decision for analysing practices collectively, relates to the connection between a DSP analytical lens with the literature on communities of practice (CoP). In this respect, previous research on CoP suggests that the way individuals interact through practices needs to be explored in the context of narratives, both personal and collective (Wenger, 1998). So even though DSP has been mainly used to explore individuals’ practices (Shove et al., 2012), my research considers how individuals practise evaluation collectively and how that practice is negotiated and perceived across different groups. Besides, the richness of this data set was the combination of personal stories collected over time, to produce an in-depth collective account of how participants practised evaluation and how it changes over time. This account was organised according to each participant’s professional affiliation, because literature on EP (Saunders et al., 2011) has recognised the need to advance the current knowledge about each professional groups’ challenges in evaluation.

Lastly, a fifth methodological issue arose with the selection of the analytical framework employed. As I subscribed to the view of evaluation as a social practice, the application of a SPT framework was decided from an early stage; however, the encounter with the specific strand of the DSP framework (Shove et

al., 2012) occurred later in the research journey, specifically, at the end of data

collection. Before this encounter, I had considered the use of RUFDATA (Saunders, 2000) as a potential framework, but because it represents a reflexive tool to plan evaluation, there was the risk of deductively imposing it on the data,

192

ending up checking how well NDEC was using it or not. In other words, my research aim was to describe what practitioners did when evaluating the YP - and how that practice changed over time, so for this reason a DSP framework seemed appropriate to conduct this research study’s analysis.

5.4.1.2 Technical issues

Various technical issues occurred throughout the research journey, but for the purpose of the thesis I outline four. The first issue encountered was some participants’ manipulation of the research interview. As they were asked to identify relevant episodes in the YP evaluation, one participant had prepared in advance written hand-outs for the first interview. This detail felt uncomfortable, as noted in the following extract from my learning journal:

Extract 5.1: Extract from my learning journal, December 2012

This situation was manifested as manipulation from a research participant, but also produced further reflection about my bias, especially after the transcription of other interviews, where participants mentioned additional issues regarding this specific participant, as the following example illuminates: “They came in

and, sometimes, I think they would come with their own agenda” (Sam). As a

researcher, after reading Sam’s transcript, I noticed that I felt the same about this

During most of the interview the interviewee picked up mainly their ideas, already written in the handout given to me, at the beginning. So, I constantly had to prompt them, sometimes successfully, other times not so much. They seemed very protective about what they had written so in a sense very well-aligned with the hard-evidence paranoia that they seemed keen to defend. In the second interview, although papers have not been given, they did refer back to those, by explicitly saying, “I think I already said that Joana. I even wrote it down on the papers given to you”.

193

‘manipulative’ participant – who seemed to have their own agenda. From that moment onwards, I strictly monitored my bias, particularly in relation to this participant (Simons, 2009). This made me develop a profound reflection in the remaining journey.

A second technical issue occurred during the first phase of interviews when I asked coordinators and practitioners to fill in a form with various evaluation activities. Funders were excluded because I assumed that these activities were not part of their role (see section 5.3.7; composite profile of the research participants). On the one hand, data on funders’ activities was collected from other documentary sources of evidence alongside their interview transcripts; on the other, it may well hide some distortion, and for that reason I approached it as a technical issue.

A third technical issue encountered was the realisation that I have not been able to analyse data from NDEC’s partner in the same depth as I did with them, because, whilst the longitudinal dimension was secured in the NDEC case, their partner’s participants were not available for a second interview. Thus, some of the data related to the partner was collected through other peoples’ narratives (Weiss, 1995). Ethically speaking, I decided not to analyse that part of data in- depth, despite its coding and categorisation, because some turned out to be potentially sensitive regarding the confidentiality of participants. For the purpose of the thesis, part of this data was only used as background information (Patton, 2002), and sensitive evidence is in a confidential appendix (only made available for examination purposes).

194

A fourth and final technical issue was the realisation of the different role of writing and speaking about my research. An illustration was when I presented my research in a postgraduate conference held in Lancaster University, and the audience asked me at various times to clarify the meaning of DE as a sector, and not even once did they mention EP. In attempting to bring the audience back to my research focus - EP - I realised that the way I was speaking about my research was considerably different from the way I had been writing about it. Peer feedback and support was vital to align speaking and writing, and was made through the adaptation of my regular writing retreats, by incorporating a half hour ‘talking space’ to develop presentational skills (see section 9.7; writing retreat as a daily practice). Having summarised the main issues encountered, the reflective space next describes two stories from the fieldwork.

5.4.1.3 Reflective space

Extract 5.2: Extract from my learning journal, October 2012

During one interview the telephone in the office rang and I focused my attention on the interviewee, assuming that they wouldn’t pick it up but, surprisingly, they ran upstairs, saying that there was nobody else in the office. On their return, I managed to get the interview’s pace back, when the telephone rang again. Luckily, this time the interviewee said, “I think I will leave this one”, and I ironically thought to myself: ‘that is very kind of you’. After this situation, I reflected on the expectations researchers have, which requires that the world around us should stop to support our research process. This episode made me reflect about the different meaning attached to the interview, by this participant and myself. For me, it was absolutely fundamental that the interview was not interrupted; I had even put a note on the door advising that audio-recording was in progress. For this participant, the real life kept going and the telephone had to be picked, the office has to keep running, and if they managed to be interviewed at the same time, splendid.

195

Extract 5.3: Extract from my learning journal, January 2013