Spearman correlation coefficients and their significance are summarized in Table 4.11. As for the WJEC populations, a positive correlation between the SEG 1994 population's GCSE science grades occurs significantly (P <0.001) for biology and chemistry, biology and physics, and chemistry and physics. Obtaining a high grade on one science is apparently predictive of obtaining a high grade in another science for both examining groups. One interpretation is that even though WJEC and SEG have different science examinations, these evoke interactions with their populations that produce positively skewed grade distributions. It is tempting to speculate from a constructivist view that this is because the same types of skills are required and / or the same types of items (questions) are present in the different groups’ science examinations.
Table 4.11 Spearman Correlation Coefficients Between Biology, Chemistry and Physics Grades - SEG
Chemistry Physics Biology 1994 1995 0.76 0.64 0.75 0.59 Mean 0.70 0.67 Chemistry 1994 1995 0.79 0.65 Mean 0.72 N for 1994 = 1001
1995 = 1759 for biology and chemistry, 1758 for physics pairings All correlation coefficient values are significant at the 0.1% level
Again as for the WJEC populations, the chemistry and physics grades are the most positively
correlated of the three subject combinations, followed by biology and chemistry grades which in turn are slightly more positively correlated than the biology and physics pairing. The same holds for the 1995 population but the biology and chemistry pairing’s value is closer to that for chemistry and physics than that for biology and physics. The similarity in the order of the subjects’ positive correlation values suggests that there might be factors influencing students’ performances in these subjects or in the marking / grading processes that transcend examining groups. From a cognitive constructivists view, one would expect this trend if the skills required by the chemistry and physics examinations are more similar than those required by physics and biology or chemistry and biology.
Kappa analysis
Both the 1994 and 1995 primary datasets required amendment to secure matched grades for kappa calculations. This procedure reduced the 1994 primary dataset from 1001 to 998 students and similarly the 1995 population from 1761 to 1758. Table 4.12 shows the resulting kappa values.
Table 4.12 Kappa Values for Biology, Chemistry and Physics - SEG Biology Chemistry Physics 1994 0.305 0.280 1995 0.323 0.282 Mean 0.314 0.281 Chemistry 1994 0.263 1995 0.297 Mean 0.280
All Kappa values are significant to 0.1%
As for all four WJEC populations, there is ‘fair’ (Landis and Koch, 1977) agreement between students’ achieved grades for all of the science subject pairings for both the 1994 and 1995 SEG populations. The biology and chemistry pairing has the highest level of agreement for both SEG populations. The lowest level of agreement is shown by chemistry and physics in 1994 and by biology and physics in 1995. However, there are only small differences in the kappa values for all of the different subject pairings and in particular for the physics pairings with biology and chemistry. Overall, the mean kappa values indicate that it is more likely for students to obtain the same grade in biology and chemistry than in the other science subject pairings in the 1994 and 1995 examinations under scrutiny. This is in contrast to the WJEC findings where most agreement was shown most often for the physics / chemistry pairing. Other than all science subjects showing ‘fair’ agreement between students’ grades there are no other patterns in kappa values that hold across the two examining groups. Overall, the analyses
indicates that there is only a ‘fair’ (ibid.) chance of obtaining identical grades in the science subjects at Triple Award GCSE. The notion of subject ‘gradeness’, in the sense that if a student obtains a
particular grade in one science subject it is predictive of the same grade in another, is not strongly supported by the findings across both WJEC and SEG. Arguably the findings support the view that students’ interactions with examination artefacts vary in their nature from student to student and thus population to population. One would not expect identical grades to be obtained by the same population across different subjects even if these have similar skill demands as there is still the potential for
I 149
differences in interaction with items (questions), for example in what skills the students actually use and / or the contexts used. The differences in the kappa values for WJEC and SEG, and in particular for the physics and chemistry pairing, also illustrate the sensitivity of the meaning attributed to comparability. The comparability outcomes can vary for the same subjects for the same periods of time according to the source of data even when the same statistical treatment is used.
Descriptive statistics
Appendix 6 shows the frequency, percentage, cumulative percentage, means and standard deviations for the students achieving each GCSE grade in biology, chemistry and physics. The bar charts in Figure 4.10 illustrate some of this information. The standard deviation values, although similar, reveal a slightly larger distribution of the 1994 population’s achieved grades for physics (1.47) than chemistry (1.45), with the smallest distribution being for biology (1.37). This order replicates that for the degree of severity of grading from most to least severe based on achieved means. The cumulative percentages indicate a significantly lower attainment of grades A and B in physics than in biology and chemistry. All of the achieved science subject grades are positively skewed, with physics being the least positively skewed, followed by chemistry then biology.
The standard deviation values for the 1995 population show a significantly larger distribution of the achieved grades for chemistry (1.49), than for either biology (0.90) or physics (0.85). This agrees with the order of degree of grading severity (most to least severe) identified from the achieved means. The pattern of SEG’s standard deviation values for biology and physics decreasing from 1994 to 1995 reflects the same trend for all of the science subjects for WJEC from 1994 to 1995. This again highlights the examination of new syllabuses by examining groups in 1995 being associated with changes in patterns of assessment outcomes. Chemistry’s position as the most severely graded science subject is not reflected in a lower percentage of grade A*s (biology has proportionally more) but rather in fewer grade Bs and more of the lower grade C and unclassified. If students failed to achieve grades A* - C in this particular SEG examination (high option, 1995), then the policy was to award them grade U and none of the grades D - G inclusive (SEG, 1996). Chemistry appears to have a
Figure 4.6 SEG GCSE Biology, Chemistry and Physics Grade Distributions 1994 ■ S D = 1 . 3 7 □ S D = 1 . 4 5 □ S D = 1 . 4 7 GCSE grade
disproportionate number of students who fall into this category and therefore appear to have been entered for an inappropriate examination. Here the technical approach illuminates the importance of teachers’ decisions on students’ tier entry for students’ achievements and the need to explore this with teachers.
4.8.2 Are there relationships between students' performances in biology, chemistry,
physics GCSE examinations and their average GCSE grade scores?
Graphical analysis
The average of the average grade scores for each grade (A*-U) in each of the SEG 1994
population's biology, chemistry and physics results was produced from the associated primary data set. The same treatment was conducted on the 1995 population for its associated grades A* to C and grade U. Figure 4.7 illustrates the analysis outcomes with line charts (graphs). The gradients of the lines in the line chart give an indication of the relative difficulty of the subjects biology, chemistry and physics for the study's populations with the generalization that the steeper the gradient and / or the higher up the y axis the line appears, the 'easier' (less severely graded) the subject.
There is no consistency in the science subjects’ slope difference in the 1994 graph making it difficult to comment on the subjects’ relative severity of grading, which in my view even if one supports the technical approach to comparability, limits the usefulness of this treatment in technical comparability investigations. There is an overall tendency for physics to be the most 'difficult' (severely graded) science subject, which concurs with the finding from the subject-pair analyses. Overall, the 1994 population has done rather less well in the science subjects than in their GCSEs as a whole. For example, students achieving grades C, D, E or F in the study's GCSE science examinations have average GCSE grade scores which are respectively better. For these students, the associated science subjects would appear to have been 'harder' than most of their other GCSE subjects as a whole. The science subjects’ line slopes are more consistently related to each other for the 1995 population than for the 1994 population. Overall, physics is 'easier' (less severely graded) than either biology or chemistry for the 1995 population under scrutiny. Chemistry is overall 'harder' (more severely graded) than biology. These findings concur with those from the subject-pair analyses. As for the 1994
Figure 4.7 G raphical Analysis: SEG 1994 and 1995