Validity and generalizability

There has been an ‘extensive unpicking’ of the ideas of validity and generalizability (the latter being synonymous with external validity or replicability) within qualitative and ethnographic research (Laurence, 2005: 130). These notions are now widely assumed to be more appropriately linked to positivist research which can support a priori hypotheses and result in quantifiable ‘hard’ data, as opposed to the less tangible, ‘messier’ results yielded from qualitative research in a ‘real-life’, human and social context.

The traditional positivist view of what constitutes internally valid or objective research corresponds to the accuracy of the selected methods in seeking out results and depicting the phenomena under enquiry – that is, to the ability of the methods to provide proof that supports the hypothesis as either correct or incorrect. This view of validity is plainly problematic within the constructivist paradigm of my study, where meaning is derived from process and in which there can be no definitively ‘correct’ or ‘incorrect’ findings, nor absolute proof. It is as a result of the poor fit of the positivist view of validity with the nature of qualitative research that the qualitative research community has sought alternative ways of conceptualizing and ensuring both internal and external validity.

Historically, some qualitative researchers have dismissed the concept of validity as unrelated and therefore not useful to qualitative endeavour, even going so far as to claim any pursuit and preoccupation with validity within qualitative research as nonsensical (Wolcott, 1994). Others have acknowledged its necessity in order to

ensure rigour within qualitative work and therefore to ensure that such work and its findings are seriously considered and valued by the research community as a whole and beyond (Yin, 1994). However, many who agree that validity is important in both the execution and reputation of qualitative research also agree that a reconceptualization is useful in extricating qualitative research from the bounds of traditional, positivist notions of validity while still supporting and ensuring rigour within qualitative research studies. These reconceptualizations include supplementing new terms and criteria including ‘plausibility’, ‘credibility, ‘coherence’, ‘intention’ and ‘authenticity’ (Hammersley, 1992 and Guba and Lincoln, 1998). Guba and Lincoln also suggest terms such as ‘confirmability’, ‘dependability’ and ‘transferability’ (1985) all of which I found fitting in relation to the development of supportive partner relationships in my own research, along with the term, ‘trustworthiness’ (Guba and Lincoln, 1998).

By adopting a constructivist approach, I was able to approach my study more empathically, putting to positive use the negative experience of being researched that I had previously endured. Denzin and Lincoln propose that ‘verisimilitude, emotionality, personal responsibility, an ethic of caring, political praxis, multivoiced texts, and dialogues with subjects’ can provide the qualitative researcher with suitable substitutes for the positivist notion of hard and fast validity (Denzin and Lincoln, 1998: 10). These alternatives allow for the taking into account issues of power, related to who stands to gain or to lose by research, whose voices are heard or ignored and whose interests are being served by the research.

Given my less-than-positive experience of being researched, I approached my own research with the importance of internal validity at the forefront of my mind. As I had now learned, continually questioning my approaches, the effect that I as researcher might be having upon the feelings, actions and responses of those participating in the study and the way in which I was interpreting the research findings was of paramount importance if the study was to be reliable in terms of its findings.

Gallagher points to the concept of internal validity or objectivity as a ‘pretense’ and a form of ‘male bias’, pervading theory and research and something that can never really be fully achieved. She proposes that by acknowledging this and abandoning the

pretence of objectivity, feminist researchers can begin to ‘break down power differentials in the research process’ (Gallagher 2008: 68).

Another way of maintaining internal validity was offered to me through the bricolage or multimethod approach earlier described. This enabled numerous perspectives of what was unfolding within the study and provided scope for the triangulation of data. In order to maintain internal validity I kept thorough, detailed research notes and formulated questions to ask the teachers on a weekly basis about their views of the study as we worked together, to ensure that their perceptions of what was unfolding either matched, or did not differ considerably from my own. In this way I sought to prioritise the ‘polyvocality’ of Gallagher’s methodological approach previously cited (Gallagher 2008: 71).

My ongoing commitment to ensuring that my research findings were reliable was not only in response to some of the weaknesses in terms of validity that I noted within the Music Potential study, but also a consequence of a strong interest in being able ultimately to generalize outward from the case study research findings.

Threats to validity

Guba and Lincoln (1998) suggest that threats to the internal validity of qualitative study include reactivity, researcher biases and respondent biases. Reactivity pertains to the ways in which research participants may modify their behaviour and actions in the presence of a researcher and while being observed. If one is to uphold the ethical ideal, discussed in the previous section, of being honest and clear with research participants about what will be observed and when, then reactivity is likely to occur naturally, albeit subconsciously, and this must be taken into account and acknowledged if research is to be trustworthy. The teachers and children participating in my research study were well aware of my presence and interest at all times and as a result, may well have succumbed to what Gillham refers to as ‘the observer effect25’ (Gillham 2000: 47). I was acutely aware of the observer effect as a result of personal experience of modifying my own behaviour and ways of working in the presence of the field researchers during the Music Potential study (see above). This strengthened

25 _{Coined by Gillham (2000) ‘the observer effect’ denotes the effect that the presence of any observer}

my own awareness and recognition of reactivity when it occurred, and this recognition in turn has served subsequently in consolidating the findings and the maintaining the trustworthiness of the study as a whole.

On the subject of researcher bias and the ‘observer effect’ Gillham echoes Gallagher’s point that there is no way around the subject of researcher bias and the limitations of one’s own objectivity other than to acknowledge them as problematic and bear the issues in mind. He advises:

In real-world research as we have mentioned before, the researcher is the research instrument, and any instrument used makes some contribution. You have to make a consistent effort to observe yourself and the effects you might be having. You can also ask members of the group or institution if they think that what happens when you

are there is characteristic. A conscious attempt at rigour can usually lead to a

reasonable judgment: we can expect no more. (Gillham, 2000: 47)

The issue of researcher bias is highly pertinent in terms of this study as a result of my extensive professional familiarity with the phenomena under investigation, my involvement and contribution to the earlier Music Potential study, and the unavoidable effects of my own personal philosophical and ethical beliefs in relation to the field as declared throughout this chapter and thesis. Taking these factors into account, I cannot reasonably claim to be unbiased in relation to the research, and inevitably entered it with preconceptions about primary music teaching. However, by maintaining an on-going and rigorous self-awareness alongside a reflexive approach (Ahern, 1999: 408), I have sought to minimize the effects of my existing and potential biases on the research project.

Another area requiring continued awareness in relation to trustworthiness was

respondent bias. Integral to the study were the relationships that I was building with

the teachers and the children, and I was well aware that with a friendly relationship comes a duty to be supportive and positive. This sort of relationship also requires honesty, but that honesty often proves difficult to adhere to, should the truth be something the other party may not want to hear. Many find it much easier in such a situation to act the part of the ‘good bunny’ (Robson, 2002) rather than disappoint a friend or colleague. Reflecting upon this, I have been aware of the possibility that the positive responses I received during the course of the study might have been affected

by respondent bias. I could have reduced the risk of this by asking an additional researcher to conduct some of the observations and interviews, but arguably this would not have been conducive to the development of the relationships between the co-researchers and might have adversely affected my ability to gain the emic perspective so central to my methodology.

Generalization or external validity

My hope on embarking upon the project was that it might be used in order to enact wider change within the teaching of music within primary schools. While this may at first appear grandiose, Ward Schofield points out that qualitative researchers within the field of education often intend their work to be more widely assimilated in order to affect educational and social change:

This desire to have one’s work be broadly useful is no doubt often stimulated by concern over the state of education in our country today. (Ward Schofield, 1996: 204) Eisner (1997) concurs that broad usefulness of a specific research study for the purpose of wider improvement is a common intention held by researchers. He says of his own educational research:

My aim is to expand the ways in which we think about inquiry in education, and to broaden our views about what it means to ‘know’. But my ultimate aim goes beyond these: it is to contribute to the improvement of education. For me, the ultimate test of a set of educational ideas is the degree to which it illuminates and positively influences the educational experience of those who live and work in our schools. (Eisner, 1997: 2)

This citation reflects my own feelings and describes the impetus that compelled me to undertake my research. However, I am aware of the potential limitations of one single study, bounded as it was within one educational setting, to have wider influence and impact on affecting change in terms of teacher attitudes toward music teaching and within the field of primary music education more widely. Yin (1994) addresses criticism aimed at single or small-scale case study such as my own by arguing that contrasting case study with survey research, as in his view many critics are implicitly doing, is futile. He states:

The external validity problem has been a major barrier in doing case studies. Critics typically state that single cases offer a poor basis for generalizing. However, such critics are implicitly contrasting the situation to survey research, in which a sample (if selected correctly) readily generalizes to a larger universe. This analogy to samples and universes is incorrect when dealing with case studies. Survey research relies on statistical generalization, whereas case studies (as with experiments) rely on analytical generalization. In analytical generalization, the investigator is striving to generalize a particular set of results to some broader theory. (Yin, [1994] 2003: 37) Guba and Lincoln, however, point to the difficulty of generalization within qualitative research, imposed by its integral, contextual elements:

It is virtually impossible to imagine any human behaviour that is not heavily mediated by the context in which it occurs. One can easily conclude that generalizations that are intended to be context free will have little that is useful to say about human behaviour. (Guba and Lincoln 1981: 62)

They advise qualitative researchers instead to replace the concept of generalization with the idea of ‘fittingness’, judging the ‘match’ of existing studies to other areas of research in which they might be interested. As Ward Schofield puts it:

Much of the attention given to the issue of generalizability in recent years on the part of qualitative researchers has focused on redefining the concept in a way that is useful and meaningful for those engaged in qualitative work. A consensus appears to be emerging that in qualitative research, generalizability is best thought of as a matter of the ‘fit’ between the situation studied and others to which one might be interested in applying the concepts and conclusions of that study. (Ward Schofield, 1996: 221) This idea of ‘fit’ corresponds with Michael Bassey’s concept of ‘fuzzy generalisation’ or ‘best estimate of trustworthiness’ (Bassey, 2001) through which predictive statements may be made from the analysis of qualitative data, drawing on the findings combined with the researcher’s own ‘professional tacit and explicit knowledge’, their ‘experience and reading’ of the phenomena or subject under investigation (Bassey, 2001: 1).

Ward-Schofield also argues that a better and more attainable approach to generalizability in qualitative research would be to consider ‘what is, what may be, and what could be’ (in Hammersley, 1996: 221), echoing Yin’s notion of a ‘broader theory’ in place of generalization (1994) and Bassey’s ‘fuzzy generalization’ (2001). The potential generalizability of my study will be further discussed in Chapter Seven.

The next section of this chapter considers the methods of this study, designated above and now examines more closely the two central methods.