Classifying and categorising schools: using data responsibly

Schools are diverse in so many ways that using a simple classification was difficult. Some classifications were more readily available in the data sets while some potentially

interesting differences had to be derived from other sources. This section describes the issues that had to be considered in deciding on classification criteria, the data to be used to classify schools using the Newsam intake performance classification and the formula used.

The methodological problem was how best to classify the different types of school to investigate this issue. It was decided to manipulate the data in different ways to see whether significant patterns emerged that could be further investigated for confirmation and clarification of the findings using other research methods. This was a flexible design research strategy where ‘the central aim is to generate theory from data collected during the study’ (Robson, 2002, p.90). The advantage of using census data was that any

differences that exist are variations in the full population not apparent differences that might result from sampling errors.

128

The variety of school classifications were identified in chapter 2 (page 50). There were two immediate issues because of the diversity of schools in the English education system and plethora of ways of classifying them. Firstly which of the classification(s) should be used and secondly how easy would it be to obtain the data. Two of the possible classifications are described below with a discussion of possible associated research issues and

implications.

a) Provision: State and independent schools.

This classification appears to be a relatively straightforward binary classification:

Is the school funded by the state? Yes or No? One implication of this classification is that there can be differences in the curriculum provided by the school. State-funded schools have to follow government regulations and legislation while independent schools do not. However this has become more complex in recent years where the funding of some state schools (City Technology Colleges and Academies) has been removed from local authority control. Instead funding comes directly from the government and may be topped up by private sponsors. These schools are also partly removed from government regulations so they have more freedom to decide on the curriculum taught.

This classification also illustrates that ‘facts’ in the real world are not value free. People’s perceptions of independent schools vary enormously and there is considerable debate about the quality and nature of the education they provide. These differing viewpoints might influence the research questions or respondents interactions with the researcher.

129 b) Type of school

The English education system has a large number of different types of school with multi- layered categories (such as 11-16, Roman Catholic, Voluntary Aided, Mixed,

Comprehensive, Technology school) although most official data uses one classification largely based on governance and selection (ToE). In the NPD this has led to twenty to thirty categories of school of which only seven were mainstream schools (table 3.7). The features outlined in table 3.7 contributed to the diversity of schools because they might result in different aims and approaches to education.

Table 3.7 Similarities and differences between types of school

Governance Funding Owned Admissions Employer Curriculum

Community LA LA LA LA National

Foundation LA Governors Governors Governors National Voluntary

Aided LA Charitable Foundation Governors Governors National Voluntary

Controlled LA Charitable Foundation LA LA National Independent Private

fees Charitable Foundation Governors Governors Governors City

Technology Colleges

Govern

ment Charitable Foundation Governors Governors Governors / National Academies Govern

ment Charitable Foundation Governors Governors Governors / National

Table 3.7 does not capture all the diversity within these groups of schools, some of which is the result of the admissions policy of the school (table 3.8).

130

Table 3.8 Admissions policy for different types of school

Type of School Admission policy

Comprehensive All ability

Selective Highest achievers

Modern All ability (not most high achievers)

Independent Varied

City Technology College All ability

Academy All ability

Admission policy was also unsatisfactory because it did not reflect the diverse nature of the actual cohorts in schools which the literature review had suggested was likely to be segregated by residential area and social class. An alternative classification that more accurately reflected the actual attainment of the school cohort was the eight school categories proposed by Newsam (2003). The initial study had suggested the attainment of individuals and the cohort was worth investigating because there appeared to be a

relationship between attainment and geography entries. Unfortunately none of the data categorised schools in this way so it was decided to do in an alternative way. The initial ideas were to use baseline data collected by schools about entry performance or outcome data such as GCSE performance (5+ level 2 grades or mean points score).

Originally it was thought that it would be difficult to collect intake attainment data

because although it was available within schools it was not published so it was decided to use KS4 data instead. GCSE performance data were less problematic to obtain because there are nationally published performance tables that provide performance outcomes at age 16 for every state school. This measure has a strong correlation with entry

131

be to weight schools by their value added (VA) or contextual value added (CVA) scores (DCSF, 2007). However the accuracy and value of these scores is disputed (Leckie and Goldstein, 2009; Gorard, 2009b). The complexity of the calculations used and the margin of error mean it is difficult to be certain whether the school effect is ‘real’ or ‘statistical’.

“Is the variation in school outcomes unexplained by student background just the messy stuff left over by the process of analysis? Or is it large enough, robust enough, robust and invariant enough over time, to be accounted a school ‘effect’?” (Gorard, 2009b: p.2).

There are a number of other caveats about the quality of KS4 performance data. Firstly while the data is the result of a national census and can be deemed to be reliable because it is the outcome of a ‘high stakes’ examination process that is externally administered (by awarding bodies) and monitored (by QCA until 2010 and Ofqual from 2010) there are inevitably some elements of unreliability within the data. Some of these relate to the methods used to compile the data set, some to the impact of using the outcomes of GCSE examinations as a measure of accountability on the behaviour of school managers and some to the underlying principles and processes of GCSE assessment which are beyond the remit of this research.

Secondly a decision had to be made about the performance measure to be used within the data set as there were several alternatives. Average point scores KS4 meanG (2003) and average pts score (2007) were chosen because they gave a continuous measure of overall student performance at Year 11. The more commonly used reported measure of school performance: the proportion of students achieving 5+A*-C (level 2) grades was a

132

categorical (binary) variable that just reported: Achieves level 2: True or False. In contrast average point scores were capable of distinguishing between the relative quality of performance. The capped point score (the eight best scores achieved by a student) was not used because it was a more limited reflection of the curriculum followed by some students. The point score was derived from categorical data where the grade obtained in an examination is given a point score. Since there are eight grades at GCSE level this allowed relatively small differences in performance to be identified, although some variation in performance (eg the range of scores within a grade) was lost. The point score was based upon the overall performance calculated from student performance in all approved qualifications entered at age 16.

Table 3.9 Quartile share of the cohort KS4 mean GCSE

performance

Mainstream Schools Top 25% 2nd 25% 3rd 25% Bottom 25% Total Total Students 2003 % of Mainstream Cohort 2003 (24.9%) 151228 159041 (26.1%) 154224 (25.3%) (23.7%) 143925 608418 (100%) Total Students 2007 % of Mainstream Cohort 2007 (25.7%) 166871 165968 (25.6%) 164771 (25.4%) (23.2%) 150679 648289 (100%) (Source, NPD, 2003, 2007)

Having identified a measure of performance that could be used it was then necessary to use this measure to assign each individual student to one of four performance quartiles. The quartile boundaries were calculated and each student was assigned to one of the four performance groups (Table 3.9). The shares of the cohort are slightly different because all

133

students with the same performance score were assigned to the same group and students in special schools, who were largely in the bottom 25% of performance, were removed. The numbers of students in each quartile group in each school was calculated using a pivot table. A number of different formulae were used to more objectively and quickly classify the nearly 4000 schools in the NPD. Initially the formulae were tested by comparing the results with those already obtained for the three Local Authorities and decisions (table 3.10) were made about boundaries.

Table 3.10 Classifying schools by intake performance (Newsam)

Selective More than 75% of students in the top 25% of the cohort performance

Comprehensive plus More than 60% of students in the top 50% of cohort performance

Comprehensive Between 40% and 60% of the students in the top 50% of _performance Comprehensive More than 60% but less than 75% of students in the _{bottom 50% of performance} Secondary modern More than 75% of students in the bottom 50% of

performance

The underlying principle was that a truly ‘comprehensive’ school would have

approximately the same number of pupils in each performance quartile whereas selective or secondary modern schools would have large numbers of students of similar attainment. The ‘comprehensive’ group was the most problematic group since this implied that there was equal numbers of students in each of the four performance groups. In reality very few schools had this even profile of performance so a decision had to be made about an

134

acceptable range from the mean. A judgement was made that comprehensive schools would have no more than 60% of students in either half of the performance cohort.

Only around 30% of schools nationally in any one year had a cohort that was

comprehensive (table 3.11) based upon performance at KS4 while nearly 40% of schools had a cohort skewed towards high performers (selective and comprehensive plus). About 31-32% of schools had a cohort skewed to the bottom end of performance. This confirms the assertion above that in reality there are few schools in the country that are truly comprehensive in terms of their students’ performance, especially when it is remembered that the definition of a comprehensive school (table 3.10) allowed some fluctuations between the quartile groups.

Table 3.11 Schools assigned to Newsam school types: 2003 and 2007 Mainstream

Schools England Schools 2003 Number of Percentage of Schools 2003 Schools 2007 Number of Percentage of Schools 2007

Selective 433 11.0 417 10.5 Comprehensive Plus 1104 27.9 1048 26.4 Comprehensive 1135 28.7 1267 32.0 Comprehensive Minus 806 20.4 890 22.5 Secondary Modern 473 12.0 341 8.6 Total 3951 100.0 3963 100.0 (Source: NPD, 2003, 2007)

135

Although this performance segregation is well recognised and diversity is encouraged by politicians there is still sometimes a tendency to talk about ‘comprehensive’ schools as if they were uniform in their student characteristics. What must also be remembered is that this classification is based on performance at KS4 not on intake. This means that any school effects are reflected in the quartile performance of each student. Interpretation of data must be done in a responsible and informed manner (Gorard, 2001).

Table 3.12 Schools assigned to Newsam school types: 2003 and 2007 2007 2003 Sel ec tiv e Co m pre hen siv e Plu s Co m pre he ns ive Co m pre he ns ive M in us Se co nd ary M od ern Clo se d af te r 2 00 3 Gran d T otal Selective _(76%) 330 _(21%) 90 _(0%) 1 _(0%) 0 _(0%) 1 _(3%) 11 _(100%) 433 Comprehensive Plus (6%) 71 (68%) 748 (21%) 230 (1%) 12 (1%) 14 (3%) 29 (100%) 1104 Comprehensive _(0%) 2 _(13%) 143 _(66%) 753 _(18%) 206 _(1%) 16 _(1%) 15 _(100%) 1135 Comprehensive Minus (0%) 2 (2%) 16 (28%) 224 (55%) 443 (12%) 97 (3%) 24 (100%) 806 Secondary Modern (0%) 2 (0%) 6 (9%) 41 (41%) 195 (37%) 176 (11%) 53 (100%) 473 Opened after 2003 (6%) 10 (31%) 48 (16%) 25 (24%) 37 (24%) 37 NA (100%) 157 Total Schools 2003 (10%) 417 (26%) 1051 (31%) 1274 (22%) 893 (8%) 341 (3%) 132 (100%) 4108

Although the overall numbers of schools in each category were approximately the same in 2003 and 2007, the performance of individual school cohorts varied so schools were not necessarily assigned to the same Newsam group. The 2003 and 2007 cohorts were compared to identify changes (table 3.12). The yellow shading identifies where there is

136

the greatest commonality between school types in the two years. This suggests about two thirds of schools stayed in the same Newsam group and therefore broadly had the same cohort performance between the two years. However there were differences between different school types with this number ranging from 37% for secondary modern schools and 76% for selective schools. There are many different points that could be made about using this data responsibly but two will be focused on here, the use of percentages and the interpretation of results.

Some of the problems with using percentages can be illustrated using the data from table 3.12. The total number of selective schools fell a little from 433 to 417, a fall of 16 (a fall of nearly 4%) between 2003 and 2007. The number of selective schools that had a slightly less selective intake in 2007 was 90 (21%) while the number of comprehensive plus schools that became more selective was 71 (6%). Although the numbers are similar the percentages are very different because the cohort sizes are different (433 selective; 1104 comprehensive plus in 2003). Because the denominator in the comprehensive plus group is larger the percentage change is much lower. Percentages need to be used responsibly so within this study segregation ratios have often been used to try to allow for cohort size (Gorard, 2001, p. 65-71). There are a number of different indices that have been used over time and some dispute over what they show (Allen and Vignoles, 2007; Gorard, 2009b) but the important point for this study is that where they have been used it is in an attempt to try to make a more realistic assessment of differences between groups.

137

The second issue about responsible use of data is in the interpretation of the results. Again using data from table 3.12 secondary modern schools showed the most change, partly because of closures (11%) and movement (41%) to the comprehensive minus group. Comprehensive minus schools also showed greater change as 28% of schools moved into the comprehensive group. These changes could be interpreted in different ways. The first might be to say that this shows that the spread of attainment in many secondary modern schools has broadened so the school group has become less segregated. This is further illustrated by the reduction in the number of schools in this category from 473 in 2003 to 341 in 2007. Another interpretation might be to say that half the secondary modern schools in 2007 had failed to change the performance of their cohort from 2003. A third might be to say that secondary modern schools had been more successful in raising the attainment of their students. The differences in these statements illustrate the way that the same data can be interpreted in different ways (Gorard, 2001). These interpretations may be to make a point that supports the case being made or they may just be slightly different phrasing that actually communicates a different message to the reader because of the use of either positive or negative language. Using data in a careful and responsible manner therefore requires an understanding of the methods used, the limitations of the underlying data and a responsible approach to interpretation of data.

138