Data Collection and Analysis

Likert type questions were utilised for the on-line questionnaire as these are designed to measure attitudes or opinions (Bowling, 1997; Burns and Grove, 1997) and, according to McLeod (2008), this is the most widely used rating scale for measuring attitude as it allows for degrees of opinion, or no opinion, though there is a potential risk of compromise due to respondents giving replies that are affected by ‘social desirability’ bias. As discussed earlier in this chapter, this inside research by a ‘fellow headteacher’ is believed to have strengthened this study, eliminated bias and encouraged honest responses.

Opinions differ as to whether Likert type responses are ordinal or interval. Several authors (Keller 2008; Cohen et al. 2011; Boone and Boone 2012) argue that they indicate a rank order of priority rather than a measured progression and that they require non-parametric analysis as there is no assumption that the sample population is normally distributed. Others (Bryman and Cramer 2005; Pell 2005; Kinnear and Gray 2010; Norman 2010) suggest that, with sociological variables such as attitudes, Likert data can be treated as interval data and that it can sometimes be appropriate to use parametric analysis. Jamieson (2004:1217) when discussing the dichotomy of views in this regard suggests that “no statement is made about an assumption of interval status for Likert data, and no argument made in support”. This resonates with my own view that the gaps between strongly agree/agree/undecided/disagree/strongly disagree in a questionnaire eliciting perceptions of headteachers would not be equal intervals and, accordingly, these have been regarded as ordinal necessitating non-parametric analysis. Rowntree (2000) recommends the use of the chi-square test when dealing with categories and investigating whether there is a significant difference between samples in proportions rather than means as “this compares the frequency with which we’d expect certain observations to occur, if chance only were operating, with the frequency that actually occurred” (p187). He describes it as “one of the most widely used tests in social statistics” (p150). He opines that such non-parametric techniques are essential when dealing with category-variables and may in other cases be advisable when we cannot be sure that the parent population is normally distributed

The suitability of chi-square tests for analysing percentages is also highlighted by McMillan and Schumacher (2006:488) “Chi-square (X2) is a statistical procedure that is used as an inferential statistic with nominal data, such as frequency counts, and ordinal data such as percentages and proportions.” Similarly, Coles and McGrath (2010) recommend that Likert scales can be analysed by plotting percentage responses and that items can then be compared using chi-squared which compares actual and expected responses. Kinnear and Gray (2010) suggest that chi-square tests can be used to establish issues or significant factors which might be identified between items. Boone and Boone (2012) note that descriptive statistics recommended for ordinal scale items include the chi-square measure of association. The utilisation of chi-square tests as a

suitable means of analysing the percentages of respondents to the questionnaire for this research is, therefore, supported by the above views.

Appendix B contains the overall responses to the questionnaire and Appendix C provides an example of a returned questionnaire. The data was then converted into contingency tables that demonstrated how the 5 sub-groups responded to each of the Likert-style questions – these are provided in Appendix D. This enabled the application of chi-square tests to ascertain whether there were significant differences between the sub-groups with regard to observed and expected results utilising the formula:

• X2 _{= ∑ (O – E)}2 , E

• where the degrees of freedom (r - 1)(c – 1) are greater than 1 • where E = row total x column total

grand total

• where the expected numbers were no less than 5 as this would render the formula for X2 _{invalid (Rees, 2000)}

A specialist survey company was utilised to submit the on-line questionnaire and it provided the facility to collate, cross-tabulate and filter the responses with access by password. The software would only accept one response from any single computer. Responses could be monitored individually, in total and by group and were recorded as percentages - these quantitative figures are summarised and presented as findings in Chapter 4 (p92-107).

Two open-ended questions were included in the survey to enable respondents to express their own priorities:

• Which aspects of leading your school give you the greatest pleasure? • Which aspects of leading your school give you the greatest challenge?

Respondents’ answers to these questions were listed individually (but anonymously) on the responses and presented as coded categories in Chapter 4 (p108). This qualitative data, as well as the data from interviews, necessitated a different approach to collection and analysis as discussed below.

Three necessary components for qualitative data analysis are suggested by Miles and Huberman (1994); data reduction (involving selecting, focusing, simplifying, abstracting and transferring data), data display (providing an organised compressed assembly of information that provides conclusion drawing) and conclusion drawing (ensuring that meanings emerging from the data can be tested for plausibility, sturdiness and confirmability). However, they warn that ‘qualitative analyses can be evocative, illuminating, masterful – and wrong - the story, well told as it is, may not fit the data’ (p. 247).

Realism research, note Sobh and Perry (2005), unlike constructivism or critical theory, is not interested in every detail – only those perceptions related to the external reality. They stress the need for all observations to have explanations and representative quotations. Charmaz (2003) notes that coding starts the chain of theoretical development and for this EdD thesis, the introduction of the two open-ended questions in the questionnaire enabled respondents to introduce new aspects for consideration and so prevent sole dominance of the researcher’s suggestions and any pre-conceptions. Template analysis (King 2004) produces lists of codes representing identified themes. However, various descriptive terms (codes, categories, concepts, themes, and key points) are all used by different authors (for example; Goetz and Le Compte 1984, Berg 2001, Patton 2002, Allan 2003) to designate ways of extracting and sorting qualitative data. Although Strauss and Corbin (1998) suggest analysing data word by word, they did so from the perspective of grounded theory. Others (for example, Glaser 1992) suggest coding by key points rather than by individual words. Some authors recommend preliminary analysis of data as soon as possible after commencing interviews (Delamot 1992, Miles and Huberman 1994) while others suggest delaying to obtain more of a feel for the whole (Hitchcock and Hughes 1995, Fielding and Thomas 2001).

An initial ‘start list’ of coding categories, is recommended by Miles and Huberman (1994) which can be modified as more data are produced. They regard this ‘start list’ as the midway point between deductive and inductive approaches with the benefit of both. However, Sobh and Perry (2005) from a realism perspective suggest that only those perceptions relevant to the external reality are worth pursuing and so codes for reducing data are usually generated from a conceptual framework so that one can ‘leapfrog’ the

first level codes normally associated with qualitative research. They point out that there may be some missed patterns as a consequence but suggest that these can be picked up during open questions during interviews and recommend that the last question to interviewees should be whether they wish to add any further data. They suggest that the second stage of a research project can aim at verifying the conceptual framework in the first stage by using the same interview protocol across all cases. In this thesis, the initial questions in the survey centred around the particular characteristics of voluntary aided schools (the impact on workload caused by admissions, staff employment, RE curriculum, etc.) but the open ended questions, as noted above, enabled respondents to raise other matters and the subsequent interviews enabled interviewees to elaborate and verify the questionnaire findings as well as raise other issues.

Three approaches to qualitative content analysis are noted by Hsieh and Shannon (2005); conventional (where coding categories are derived from raw data), directed (where initial codes start with theory or relevant research findings, and summative (where coding starts with the counting of words). The approach in this EdD thesis accords with the second ‘directed’ approach in that the responses to the two open-ended questions in the survey were utilised to form the questions for the semi-structured interview questions – thus providing the initial themes/codes for the qualitative analysis. Miles and Huberman (1994) distinguish between descriptive codes (what the respondent is saying), interpretive codes (what the analyst thinks is implied by the respondent) and inferential codes (in which broader patterns can be identified). They note that the use of matrices for individual cases can disclose patterns. For this thesis, the responses of the interviewees were coded from the initial question themes into broader topics to see whether other patterns became evident.

It should be noted that several authors advocate the use of counting the frequency of responses in qualitative analysis (Goetz and LeCompte 1984, Robson 1993, Miles and Huberman 1994, Silverman 2010, Cohen et al. 2011) in order to establish relativity and patterns. However, caution is urged as meaningful statistics cannot be derived from these and the main focus of qualitative data should be on descriptive narrative. From a realism perspective, Sobh and Perry (2005), in addition to suggesting that data displays can show numerical frequency of empirical experiences, suggest three further guidelines which have been adopted when reporting the findings in Chapter 5;

• The importance of explanations for observations focusing on contingencies, structure and mechanism

• The importance of frequent representative quotations in support of explanations with links to respondents

• The fact that data analysis software is not essential for realism research as this emphasises relationships, connections and creativity and computer software may lead to a decrease in sensitivity about these