Methodological tensions in conducting policy analysis

In this section, I discuss some of the methodological tensions that can arise when conducting policy analysis. The discussion here deals predominantly with interviewing policy-makers as a method for policy analysis, but a number of points raised hold relevance for the area of policy analysis more widely.

The first methodological tension that I discuss is the selection of an overall approach for policy analysis. Humes (1997) distinguishes between an ‘inside’ and an ‘outside’ approach but suggests that they are best viewed as opposite ends of a continuum rather than distinct categories. An ‘inside’ approach involves relying on the insights of those who have been intimately involved in the policy planning and development and this is the approach used in this research. He suggests that such an approach contends that only individuals who have been actively engaged in the process can fully explain any decisions made, shifts of emphasis, changes in direction and re-definitions of aims, and that this insight is required for the researcher to fully understand the policy process. Furthermore, Humes warns that an ‘inside’ approach is “indispensable”, and to “neglect it is to run the risk of missing important clues or misinterpreting events” (Humes, 1997, p. 21). For this reason, I decided to ensure that my research would involve those actors considered to be on the ‘inside’, but remained cautious about the extent to which this approach allowed me to gain a ‘full understanding’ of the process.

Despite these advantages, there are difficulties associated with an ‘inside’ approach, which suggest that it may not yield the type of information that the researcher is seeking. While the interview might be the most suitable method for gathering data from key policy actors (Duke, 2002), it is often the case that such individuals are constrained by their professional position. This can impact on the validity of the data obtained, but Humes (1997) suggests that the most likely consequence of this is ‘un-illuminating’ data that in which responses have been carefully edited. Furthermore, it is often the case that policy actors are well practiced at answering interview questions, and revert to communicating the ‘official line’

on various issues (Welch, Marschan-Piekkara, Penttinen & Tahvanainen, 2016). Even if policy actors are not ‘constrained’ by their position, it might be difficult for them to step back and evaluate the process within its wider context. However, this might not necessarily be a problem, depending on how their data is used. Participant insights from the ‘inside’

can be useful for getting a picture of how the policy actors perceive the process from within and can be triangulated with data gathered through other means. While each of

62

these scenarios have implications for data validity, it is important to remember that they also raise some ethical issues that the researcher has to grapple with. Gathering data from individuals in important roles can lead to a sense of vulnerability amongst participants. It is possible that any data obtained will be politically sensitive and this was certainly the case within this research.

It might be assumed that the issues outline above are less problematic if the interviewee is no longer in post, and if the policy being analysed is somewhat historic. However, my research sought to analyse a policy which was very much in the ‘development’ stage and had not yet been implemented; as a result, each actor that I wanted to interview could still be considered as being ‘in post’. I discuss the implications of this towards the end of this chapter.

An ‘outside’ approach (Humes, 1997), on the other hand, starts from a different set of assumptions. It assumes that if a researcher is to remain critical, they must not be drawn into the political process that they are attempting to analyse, or in other words, they must not be drawn into their ‘assumptive worlds’ (McPherson & Raab, 1988). Another assumption of the ‘outside’ approach is that key actors in the policy community are likely to provide a distorted account of what is actually happening. Traditional sources of data obtained when employing an ‘outside’ approach are documentary sources such as official reports, minutes and press releases. These, of course, are limited by the fact they have been written for those outside of the policy processes. Nevertheless, Humes (1997) states that there is a tendency for researchers to undervalue informal documentary sources, which can be used to illuminate connections between people and institutions, allowing the researcher to develop an overall view of policy networks. In my critical research, as well as conducting interviews with policy actors, I analysed a range of documents, including official minutes for meetings of the NPG and sub-groups, in order to gain a broader understanding of the wider policy process. I discuss my use of document analysis in more detail further in this chapter. In this sense, it could be said that my research drew on elements of an ‘inside’ and an ‘outside’ approach.

The second methodological tension for discussion concerns networks and relationships in Scottish education. One key characteristic of Scotland’s education policy community is that it is particularly small and tightly networked. As discussed in Chapter 1, this creates opportunities for intimate relationships between key actors, with many central figures

63

knowing each other on a personal basis (Humes, 1997). This raises a number of challenges for researchers, particularly around anonymity and the nature of data obtained. However, a well-connected network can also enhance data collection as it increases the number of opportunities to connect with potential participants.

Grek (2011) argues that the concept of ‘close proximity’ is also characteristic of the research-policy relationship in Scottish education, with long standing conversation between elite academic and policy elite communities existing as an established practice.

On one hand this can be viewed positive feature as it should help to create the right context for research informed policy. On the other hand, it can work to restrict researchers, and limit the extent to which they can be critical of educational reform.

A critical view on this relationship is offered by Humes (1986) who suggests that Scottish research in the field of education policy has lost its ‘freedom’, describing negotiations with Her Majesty’s Inspectorate of Education (HMIE) about a piece of research that resulted in him being told he could “write what [he] liked but that [he] should not forget that Scotland is a small country” (Humes, 1986, p. 173). It can therefore be suggested that the relationship between policy makers and academic researchers has become somewhat tense, particularly in regard to the government’s double role of a ‘user’ and ‘commissioner’ of research. Grek (2011, p. 235) concludes “researchers are very aware of their need not to offend powerful funders and gatekeepers”. Although this research was not funded by the Scottish Government, similar concerns around not wishing to ‘offend’ arose during the collection, analysis and presentation of this research.

The final methodological tensions that I would like to introduce here centres on the way in which the researcher might be perceived by their respondent in an interview situation and the impact that this may or may not have on the data obtained. In their paper ‘Sex, Lies and Audiotape: Interviewing the Education Policy Elite’, Ozga and Gewirtz (1994) discuss, from a feminist perspective, some of the difficulties that can arise when researching education policy as female researchers. They draw on their experience of conducting research with policy elites who had all previously worked in the field of education policy, in the context of central or local government. All of their informants were male, bar one female director of education. In their paper, Ozga and Gewirtz (1994) reflect on the perception that their gender impacted on the research and their relationships with their informants. They argue that the central issue here was one of self-presentation, which is

64

perhaps better described as a gender issue. They state that their gender assisted them in gaining access to their participants, as they believe that their informants perceived them as

“unthreatening and relatively unimportant” (p. 132). They believe that they were viewed as women in very stereotypical ways, some of which they endorsed due to feeling a degree of familiarity, and as a result, acted to confirm this view. One of these stereotypical roles was that of the ‘attentive listener’. They believed that their informants regarded them as receptive and caring because of their gender, and they decided to act to confirm this stereotype because it was productive for the project. Ozga and Gewirtz (1994) were well aware that they were being patronised by their informants, and in accepting this, that they were compromising themselves on the grounds of feminism. However, they perceived it as a small price to pay in exchange for obtaining deep and meaningful data.

At a deeper level, is the issue of power relations, which can also be swayed by perceptions of youth and inexperience. Given the longstanding relationship between academic and policy elite communities, Grek (2011) states that the position of a ‘junior’ researcher is of great methodological interest, and she lists several points to be aware of as an early career researcher. She warns that gaining access to participants is the first hurdle to overcome, but that “with the ‘right’ institutional affiliation and the ‘right’ cover letter authorised from an established academic... a request for an interview is usually accepted” (p.237). If access is granted, Grek (2011) suggests that the novel researcher can ‘perform’ their role in one of two ways: the role of ‘outsider’ or the role of the confident researcher. She states that the role of the ‘outsider’ is often adopted by young female researchers, and that such an encounter is often rendered ‘harmless’ or non-threatening by the participant. In other words, the researcher is not considered important enough to constitute a threat (Dexter, 1964). In a similar way to the gender effect discussed earlier, such a perception may be advantageous to the research with regard to the nature and depth of data on offer.

Furthermore, Grek (2011) suggests that, when a researcher plays the role of inexperienced

‘outsider’, interviewees often appear confident and self-assured when providing answers, and can be very quick to deflect questions which they consider to be inappropriate.

Although there is a danger of the researcher being ‘drawn in’ by such a performance, Grek (2011) argues that it is easily detected and can be an interesting part of the analysis. Being a female researcher, at an early stage of my career, I can familiarise with many of the issues described above.

65

Ozga and Gewirtz (1994) also warn of the danger of the researcher being ‘drawn in’ when conducting interviews with “polished and experienced policy practitioners” (p.121). They felt that the interview data they received in their Policy Elites project could be characterised by “the self-conscious presentation of the ‘public servant’” (p. 131), rather than an accurate presentation of the individual. Indeed, Grek (2011) acknowledges that conducting research in this area can elicit responses from interviewees which are often well rehearsed and at times, platitudinous. Because of this, she stresses that it is important to understand interview responses as part of a story – “as a construction of events and relationships that has a particular plot and follows certain conventions in the roles taken by actors” (Grek, 2011, p. 239). I found this approach particularly useful within my thesis, as it allowed me to take a step back and consider interview data as reflective of the way in which this individual operated within the policy space, as a representative of an institutional actor. I explain this approach in greater detail further in the chapter where I discuss the process of data analysis, and also in the final section.