Chapter Three
3.7 Reliability and Validity
“Valid research is ‘plausible, credible, trustworthy, and, therefore, defendable’” (Johnson & Christensen, 2000, cited in Johnson & Turner, 2003, p. 300). It must be noted here that although the statistical data does have a degree of reliability and generalisability, that is not necessarily the case for the qualitative data. It is more important that this data have validity which has been achieved through several means such as the triangulation of information, checking details with participants, using “rich description”, making the researcher’s own position clear, and presenting and discussing data that does not ‘fit’ with the identified themes (Cresswell, 2002, p. 196). The variety of information is important as “the more sources of data you have the more likely it is your findings will be credible” (Hendricks, 2006, p. 73).
Johnson and Turner (2003) note, there are three main validity types that were originally developed for qualitative research:
1. “descriptive validity, refers to the factual accuracy of an account as reported by a researcher”;
2. “interpretive validity, refers to the degree to which the researcher accurately portrays the participants’’ meanings about what is being studied”; and
3.
“theoretical validity, refers to the degree to which a theoretical explanation developed by the researcher fits the data” (p. 300).Each of these validity types is equally appropriate in a mixed methods approach to research. A further two important validity types were also identified by Johnson and Turner (2003):
Internal validity is traditionally defined as what may be called ‘causal validity’ or one’s justification in making a causal inference
from one’s data … External validity is traditionally defined as the degree to which one can generalise a research finding to other people, places, settings, and times. (p. 301)
These two aspects, however, focus more on the overall validity of the study than the internal validity of the data collected. All types of validity need to be carefully considered in mixed methods research.
Each step in the analysis was carefully documented to ensure that the process had and maintained reliability. Gratton and Jones (2003) support this by saying that “many researchers reporting qualitative research do not make it clear how the analysis has taken place” (p. 217), so by documenting each step the thought process and analysis steps become clear, eliminating potentially invalid findings.
A further ethical consideration with research is that it contributes to the body of knowledge in a particular field and that the research has adequacy. Research must have clear research goals that are attainable through the proposed research design; it must contribute to the current body of knowledge; and should be overseen by supervisors with appropriate qualifications (Massey University, 2009). This study met all these criteria.
3.8
Access
Access to the institution was negotiated with the Board of Trustees (BoTs). From initial discussion with the senior management of my secondary school this access was not likely to pose any problems, apart from the ethical issues associated with my employment and position at the school. I am currently a member of the middle management of the secondary school and so to alleviate this potential ‘power’ issue a volunteer process was employed for the recruitment of participants. All staff were involved in the change process on a whole school basis, so no bias of and for the volunteers resulted. To gain access I formally wrote to the institution outlining my plans, hopes and aspirations, and requesting an opportunity todiscuss the matter further with the BoT (Appendix 9.1). Their acceptance of my request was quickly forthcoming.
3.9
Research Participants
Prior to undertaking this study and actually working with participants I had to ensure that all potential ethical issues were considered. This is particularly so when one is working within a school environment. With this research several aspects needed careful thought including the consent process and protection of rights. For AR, initial general consent was needed prior to commencement. This was granted by the institution. A second consent was needed once the actual processes involved have been established and the research specified. Follow‐up consents were necessary for interviews and group discussions, held later in the research cycles. The consent process had to be free from coercion and participants had to have a clear idea of the purpose of the research, what their role would be and how the data would be used. They also were made aware that they have the right to withdraw at any point (Massey University, 2009). Given that I was working with colleagues, it was necessary to provide reassurance that neither participation nor non‐participation, whichever option was chosen, would have a negative impact on them. Privacy, confidentiality and preservation of anonymity were assured as far as was possible within the bounds of the law. The research did not involve deception. The participants were also informed that all raw data collected will only be accessible to me and my supervisor/s, again ensuring privacy and confidentiality. All this information was made clear to participants prior to their signing their consent forms. This thesis does not identify individuals involved, or the school where the research took place. It must be noted at this point that the professional relationships amongst the research volunteer group, myself and senior management were enhanced by the ethical considerations taken, thereby proving the benefit of utilising such systems to protect the rights of individuals and groups.
The sample for this research comprised teachers from a secondary school. The initial survey was randomly distributed among this sample and was anonymous in nature. One hundred surveys forms were delivered to each teacher at the school. Sixty four were returned, however, one of these was blank. This type of survey is termed cross‐sectional as the population from which it is drawn is essentially pre‐ determined (Fraenkel & Wallen, 2008). Information was collected at one point in time and gave perceptions and opinions of the whole staff at that point. The sample for the AR consisted of volunteers and the expectation was that sample size would be between five and ten participants. It was hoped that teachers with a variety of experiences, subject expertise and year level focus would participate. This range would allow for a much broader and diverse set of beliefs, experiences and understandings that would lead to robust and vigorous debate/professional discussion. It was noted in planning this research that AR “is nearly always a demanding experience” with “contending dimensions” (Miles & Huberman, 1994, p. 34) that it is possible to feel overwhelmed with the process and its analysis. By limiting the group to a manageable number it was hoped that some of these issues would be overcome. This proved to be the case. The final group consisted of eight teachers coming from subject areas which included Food Technology, Media Studies, Science, Social Sciences and Computer Studies. The group’s composition included senior management, middle management, a Specialist Classroom Teacher (SCT), and classroom teachers. Ages and teaching experience also varied. The group did not include any year one or two teachers as their high workload precluded them from taking on the additional requirements of this research process. All teachers, both those in the volunteer group and the remaining staff were, however, involved in the whole school processes, as required by senior management.
Selection of the volunteer group was not, however, a straight‐forward process. I first had to inform the whole staff of the research I wished to undertake and then ask for volunteers to help with the AR group component of the research. It was made clear at this point that there would be no special favours for those who did
volunteer and that the research would be on top of existing workloads. I asked that those who were interested in participating email me so I had a record of potential names. Several teachers came and asked further questions prior to making their decision. Once I had a group of possible participants I provided all with an information sheet (Appendix 9.3) and allowed them a period of time to read the information and assess the implications individually.
The information sheet (Appendix 9.3) outlined the purpose of the research and the expectations for those who volunteered. Once teachers had volunteered for the group, they were each given an Individual Permission Form (Appendix 9.4), a Group Permission Form (Appendix 9.5) and a Confidentiality Agreement (Appendix 9.6). All these documents were part of the requirements to gain ethical approval (Massey University, 2009), and helped to ensure the participants knew not only what was expected of them, but also their rights within the research process. All this information was given out prior to a school holiday so that teachers had time to make an informed and considered decision. The result of allowing this time was that several who had initially agreed to participate pulled out. Those still interested signed the consent forms and the study began in earnest.
Due to the small sample size, the results will have limited generalisability but as Carr and Kemmis (1986, p. 193) state “action research reminds the practitioner that he or she is, in some small way, changing the world”. The findings of this research can, and do, inform the practice of the institution in improving their PMS.
Initially I had ten respondents who were interested, but work commitments and changes in people’s lives meant that two had to withdraw. While this was disappointing at the time, it was also completely understandable. On reflection, it was probably beneficial as the final group numbered eight which was much more
manageable. The group comprised a wide mix of people with a variety of teaching
experiences and from different curriculum areas. The group comprised:
Number of years teaching ranging from 3 to 10+ years
Curriculum areas including food technology, ICT, business studies, science,
social sciences, and media studies
Positions within the school ranging from teacher to senior management.
There was also a variation in levels of confidence – most teachers were comfortable with their practice, but one was very unsure and had chosen to take part in the group to gain support for changes in her practice. This became clear in individual discussions with her and comments made during the group sessions. Please note: no names are used throughout this thesis to protect participant identity. Each member of the group was assigned a letter on a random basis and will be referred to by that letter in the report and discussion of the findings that follow.