6 What counts as ‘research’ in science education, and what makes research ‘good’?
6.4 What features of a research study make it ‘good’ research?
Section 6.2 above has identified some of the characteristics of an enquiry which are seen as important in deciding if it can be called ‘research’ — its defining characteristics. e same features were also referred to by many interviewees and focus group participants when discussing the
quality of research studies. A particular characteristic of a study might, for example, make some respondents unwilling to regard it as research — whereas others classified the study as ‘research’ but saw the same characteristic as important in judging its quality. Ideas about the features of research which ensured its quality arose in many interviews. ey were also explored directly in one focus group task, where participants were asked their views on specific examples of published research, presented to them as short vignettes.
It is not surprising that the boundary between what is research, and what is good research is also blurred. e term ‘research’ carries an evaluative, as well as a descriptive, connotation. So classifying an enquiry as ‘research’ oen carries an implicit positive evaluation. ‘Research’ becomes synonymous with ‘systematic’ or ‘sound’ enquiry.
Several focus group participants referred to methodological factors when deciding how convincing a piece of research was. Of these, the issue raised most frequently was the scale of the study, and the extent to which the choice of sample or setting could be regarded as representative.
T: A class of five, it’s not enough. I: So what would convince you?
T: Well you’d want to see something similar done in a variety of different schools with, you know, normal schools … [secondary focus group 1, paras. 139-141] It’s not a convincing piece of research when you have such a small sample. Especially a hand picked sample, presumably. And not necessarily with your ordinary, average teacher either …
[secondary focus group 1, para. 386] … a case-study of 5 students, it’s too small a sample really. Five students in a less able science class and they did it once. It would depend which way the wind was blowing, I would imagine, what results [you got]. You know, they did it this morning when there was a howling gale, the kids would be climbing up the walls. If they did it in July when it’s nice sunny day, you might get a completely different result. As a piece of research I felt that was very wishy washy in quality.
[secondary focus group 2, paras. 208-212]
Such teachers do not seem to recognise the point made by Darling-Hammond (2002) that medical research, for instance, ‘typically uses small sample experimental research as the basis for establishing the possibilities of effects and uses large correlational studies as rough indicators of possible
relationships that require further examination’ and that ‘researchers recognise that mixed methods of research serve complementary purposes’ (p. 15).
Some teachers did acknowledge that quantitative data from small scale studies can have some value, but placed greater trust in findings from larger scale studies:
For me to rate and value … research I’ve got to be absolutely convinced of the basic reliability, the validity, so sample size is important though that is not the be all and end all … even some of the anecdotes can be absolutely gob smacking …. but for the most part the educational research I trust a great deal is … when that is done with large sample sizes, when it’s done across
the world... [Simon, S2, para. 54]
Several of the extracts above also raise the issue of the transferability of the findings to other settings, a point to which we will return in section 7. Another influence on the thinking of several respondents was their background as science graduates. Some refereed explicitly to this as backing for their views.
I’d want to see more evidence that that actually has an impact. Four schools to me is, as a scientist, it seems ridiculously small. I mean we’re teaching about sample size and four doesn’t do it for me, I’m afraid. [secondary focus group 3, para. 331]
One Head of Science interviewed saw his colleagues’ science background as a significant influence on their views of educational research. Scientific research was taken as the ‘model’ for all research, and social science research then seen as of lower quality.
I’ve got a couple of teachers here, both in their forties, […] they’ve done research extensively in chemistry and have done a lot of time in University in chemistry, not in education as such. And they see research as a scientific thing. ey don’t see education research as a scientific process, which it is, it’s got to be, the way it’s got to be put together. But it isn’t necessarily perceived by
them as such. [Ken, S2, para. 224]
e tendency to privilege an idealised model of scientific research when judging reports of other people’s research might lead us to expect that teachers would use such a similar model to evaluate changes in their own practice. Yet, when asked about this later in the interview, very few teachers appeared to use such demanding methodological standards or criteria. Comments like those above may, therefore, reflect a rhetoric used to discuss research carried out by others, and tell us little about the methods practitioners actually use to assure themselves of the good sense of following one course of action rather than another.