Reliability and validity issues

Commenting on quality criteria for assessing mixed method research, Brannen (2005) suggest that universal agreement seems to have been reached that quality concepts developed for quantitative research such as generalizability, validity, reliability and replicability cannot nor ought not be applied to qualitative research (Spencer, Lewis, and Dillon, 2003). Rather, drawing upon Lincoln and Guba (1985), broadly equivalent concepts can be found that apply to qualitative research. For example,

• credibility/trustworthiness: internal validity; • fittingness: external validity

However, Brannen (2005) poses the question, “in doing mixed methods research how far do we work with these separate criteria or do we develop new specific or convergent criteria for mixed method research?” Brannen suggests that the criteria are likely to depend upon the dominance of the qualitative or quantitative method and type of data analysis used within the research study. Thus if the qualitative component is dominant, then it may be more appropriate to use the criteria by which such research is judged and similarly when the quantitative component dominates, although a further consideration is how far the different results are integrated in the overall analysis.

Brannen (2005) admits that currently the solution is less obvious or satisfactory where both qualitative and quantitative components are equally significant. Bespoke or convergent criteria may be required here. As Tashakkori and Teddlie (2003b) suggest, a new nomenclature could be created; they suggest the term ‘inference quality’ as a substitute for validity/trustworthiness in order to convey the quality of the conclusions that can be drawn from a mixed methods study. In this study, I have discussed the criteria for both quantitative and qualitative research because I considered the two methods as equally significant.

Regarding the quantitative method, Cohen, Manion and Morrison (2003) argue that it is impossible for research to be 100 per cent valid; that is the optimism of perfection. Quantitative research possesses a measure of standard error which is inbuilt and which has to be acknowledged. In quantitative data validity might be improved through careful sampling, appropriate instrumentation and appropriate statistical

treatments of the data. The sampling technique I used in selecting the sample and the time expended in developing the questionnaire have been outlined.

Robson (2002) points out that the validity and reliability of questionnaire data depend to a considerable extent on the technical proficiency of the researcher. In this study I also sought advice and suggestions from experts during the designing of instrument. A great deal of technical proficiency was therefore employed. Consequently, more enduring processes were adopted in designing and piloting the questionnaire as already outlined at Sections 4.5.3 & 4.5.4. Also, Robson suggests that the validity issue concerns the response rate, the higher the response rate the better. In this case the response rate was very high, 92%, as explained at Section 4.5.5. In terms of reliability, since I presented all the respondents with the same standardized questions, which had been carefully worded and piloted; it was possible to obtain high reliability of responses.

Additionally, as stated at the section for analyses of data, I used factor analysis for the main aspect of the questionnaire used to address the research question. According to Field (2005) if factor analysis is used to validate a questionnaire, it is useful to check the reliability of the scale. In practice, the simplest way to do this is to use split-half reliability. However, Field argues that, the problem with this method is that there are several ways in which a set of data can be split into two and so the result could be a product of the way the data were split. To overcome this problem, Cronbach in 1951 came up with a measure that is loosely equivalent to splitting data into two in every possible way and computing the correlation coefficient for each split. The average of these values is equivalent to Cronbach’s alpha, α, which is the most common measure

of scale reliability. In this study the Cronbach’s alpha, α was used to check the reliability of the whole scale as well as the three-factors or themes (see detail in Chapter 5).

In relation to the qualitative data, Cohen, Manion and Morrison (2003) state that validity might be addressed through the honesty, depth, richness and scope of the data achieved, the participants approached, the extent of triangulation and the disinterestedness or objectivity of the researcher. In this study, teachers’ transcriptions were typed and given back to them to read through to check the content. I also checked the tapes with children, put special marks for the purpose of identification, check the translated transcription with a colleague in the Department of Ghanaian Languages, at the University of Education in Winneba, Ghana; and used notes made during the fieldwork to authenticate information. I also involved three disinterested colleagues in extracting themes from the transcription, two of whom were also involved in the pupils’ transcription. I used verbatim quotations; I also adopted the constant comparative methods which involved examination and re-examination of the data to discover the inherent themes and use of information from other sources to refine the themes.

Another advantage is the use of different sources within the qualitative methods such as: teachers, lower attaining pupils, documents and observations or triangulation (Cohen and Manion, 1985), or ‘gathering account’ (Elliot and Adelman, 1976) of continuous assessment from four different points of view. As Denzin (1989) suggests, “by combining multiple observers, theories, methods, and data sources, researchers can hope to overcome the intrinsic bias that comes from single-methods, observer,

and single-theory studies” (p. 307). However, Patton (2002) clarifies the notion that the purpose of triangulation is to test for consistency rather than to achieve the same result using different data sources or inquiry approaches. Concerning inclusive practices, Clarke, Dyson and Millward (1995) suggest that the inclusion of the experiences of pupils (lower attainers) who are currently excluded, not only reflects internal consistence but also the authenticity (validity) of the voices that are heard.

I also participated in a seminar in Ghana on April 7, 2005 involving 16 academic staff of the Department of Special Education, University of Education, Winneba. The rationale of the seminar was to discuss the methodology and methods of data collection with a group of academics who were familiar with the Ghanaian educational system. I spoke for 20 minutes and spent 15 minutes answering questions. The major issue that emerged at the seminar concerned the rationale for using the mixed methods design and whether the design was adopted for the purpose of triangulation. I explained that the choice of design and the range of methods for data collection were driven by the aim of the study and the research questions. I explained that while it was possible to use questionnaire to address research question one, the other questions involved specific classroom arrangements and experiences of lower attaining which required the use of in-depth interviews, observations and documentary analysis.

I was asked whether the sampling technique would enable me to generalize the findings of the study. As explained earlier, the sampling technique was rather influenced by the aim of the study. I adopted non-probability samples, which according to Robson (2002) cannot be used to make statistical inferences. However, it

may still be possible to say something sensible about the population from non- probability samples. The ultimate purpose of this study was to provide a basis for action, methods of assessing pupils and using the information to support pupils in the classroom. While policy-makers, planners and administrators want generalizations and rules, classroom practitioners are not helped by generalizations which apply to the whole. Nonetheless, the findings are relevant for both policy-makers and classroom teachers.

Further, I was asked to explain the concept lower attaining children and how I sustained the interest of such children during the focus groups’ interviews, since these children were normally quiet and withdrawn in the presence of adults. With respect to

lower attaining children, I explained that the group encompasses pupils who record

lower attainments in all or many school subjects as reflected in the continuous assessment records (see figure 4.1). Since the lecturers were conversant with the continuous assessment records they understood the explanation I offered.

The seminar provided a unique opportunity to talk about the research study to people who are familiar with the Ghanaian education system. The seminar was beneficial and I would recommend that the Department continued to organise such seminars to enable the staff discuss and debate issues relating to their practices.

4.8 Negotiating access

A letter from my supervisors (Appendix 8) was sent to the Director General of the Ghana Education Service about my intention to involve schools in the Agona and Affutu districts of the Central region in the study. Further, letters were sent to the two

District Directors of Education at Agona and Affutu in November 2003 asking for permission to do the study in schools in their districts. Telephone conversations were held as follow up to ensure the Directors acted on the letters. I was asked to meet them on arrival in Ghana from England (Appendix 8A). On arrival I phoned the District Directors to arrange meetings with them to explain the purpose and significance of the research study. I met the two District Directors of Education personally and they granted me permission to do the research.

Apart from verbally granting me access to the schools, the District Director of Agona was unable to write an introductory letter for me to take to the schools. This was because she was on national assignment and had to leave for Accra soon after that meeting. She however delegated her representative (officer-in of examinations) to accompany me to the schools I had selected in the sub-district and introduce me to the teachers. The officer accompanied me to the schools and introduced me to head teachers. After the introduction I explained the purpose of the study and requested a meeting with only the teachers I wanted to complete the questionnaire.

Thus apart from the Directors I also sought permission from head teachers and teachers at different stages. As Scott (1997) explains, gaining access to research settings involves far more than simply being granted permission to begin research. It is a continuous series of negotiations and re-negotiations, with different personnel at different levels within the organisation school.

Further, I also wrote to the head teachers to ask them to inform the teachers about the interviews (Appendix 8B). However, because I did not make provision for names on

the questionnaire, for the sake of anonymity and confidentiality, I sent the letter to the schools and met teachers who had volunteered to do the interviews. I then negotiated with each teacher a convenient time for the interview.