6.3.1 The Data
Raw data for the study are the final marks awarded by the examinations board in the courses considered. These marks, from 19 modules, include those of the 99 1Y students and the 1638 remaining students from the same years. Each of these marks includes an exam result and any coursework submitted; if students have retaken a module their highest mark is used. These data include nearly fifteen thousand exam results and give a firm statistical basis on which to draw conclusions. Modules in the second and third years of the degree programme are included to determine whether any follow-on effects are present.
Table 6.1 shows the number of data points used in the analysis of each year and level, i.e.
the number of marks recorded by students at each level. These results, 6867 for the first year, 5085 for the second year and 1383 for the third year were distributed over six, six and seven modules, respectively, in each of the three years.
Academic Level
Table 6.1: Data points for analysis by level and year
Each of the 18 years’ data were contained in separate spreadsheets that listed student ID, module code and final mark. Thus a student who gained a mark in more than one module was listed more than once in the same list. To arrange the data into a usable form, all 18 spreadsheets were concatenated into a single sheet and a macro written (see Listing B.1) that put the marks for each student on a single row arranged so that the marks for each module were in a single column. Results were initially ordered by student ID, then module code and final mark, all in ascending order. Thus, for modules which students have re-sat examinations or retaken the module, the higher mark was used for analysis. One module initially included for analysis, MSM3P04, was taken by no 1Y students and so was left out.
To these results were added the results of the 1Aa class test, a 50 minute examination taken halfway through the first semester. These data, from 1498 students who had a mark recorded, were also given their own column. Finally the gender of the student and whether or not they
were a 1Y student were given columns of their own. The resulting table recorded the results of 1737 students across all years, and was copied into an SPSS data set for analysis.
6.3.2 The Method
Standard statistical methods were used in the study; in particular Lomax (2001) was used as a guide for the methods themselves (Chapter 12 covers multivariate regression analysis) while Field (2005) informed the use of SPSS. Whilst we are nominally interested in attainment, our primary concern is progress; Plewis (1997, pg. 24) states that:
“
The distinction between progress and attainment is an important one in educa-tional research. Progress is a dynamic concept needing longitudinal data for its measurement, attainment is a static concept needing only cross-sectional data.As our dataset comprises results from individual modules, it is cross-sectional, though we argue, given that for the majority of students we have more than an single year of results, our study is still dynamic to a degree.
Initial work used independent t-tests to determine whether 1Y students performed better than their peers to a statistically significant degree (5% or under). Tables A.1 and A.2 in the Appendix show the results of the t-tests on eleven of the modules considered, using data up to 2009/2010. In most modules, however, the data failed Levene’s Test for homogeneity of variances, and so we cannot base solid conclusions on these results. Furthermore, the mean of 1Y students’ scores on the 1Aa class test was 12.85% higher than their peers. (None of the 1Y
students had completed the module before taking the class test, though roughly a fifth would have taken the module in their first semester and so had some 1Y experience.)
As t-test proved inappropriate for our dataset, it was decided to perform multivariate regression analysis to predict students’ scores on other modules. Multivariate analysis us-ing SPSS allows us to determine, with reasonable margin of error, the contribution that 1Y participation makes to other modules. The decision to use the 1Aa class test data for regres-sion analysis of first year modules was made because complete transcripts of A-Level results for students were not available. This being the case, the only other variable that could be used would be a student’s A-Level grade. When the overwhelming majority of students in the dataset recorded an A in mathematics this does not give sufficient distinction between students for reasonable coefficients of determination. Overall correlation between the class test mark and students’ mean marks in first and second year modules was reasonably high with a coefficient of determination of 0.36. The 1Aa class test data contributes 10% to each student’s 1Aa mark, with the remaining 90% coming from the end of year examination (70%) and continuous assessment (20%). Given that the contribution of the 1Aa class test mark to the 1Aa module mark is relatively small, it was decided that in the particular case of the 1Aa module regression, we would still be able to draw conclusions on the result of the regression.
Thus we have the equation:
Module mark =β0+β1CT +β21Y. (6.1)
In Equation 6.1, β1is the unstandardised coefficient of the class test mark (the Class Test
mark being a percentage) and β2is the unstandardised coefficient of the 1Y variable. In this way the contribution 1Y makes to a student’s performance in a module is isolated from other factors that influence a student’s accomplishment. The 1Y variable is 0 for students who did not take 1Y and 1 for students who did take 1Y; this gives us the added advantage that the value of β2is the increase in mark gained by being a 1Y student.
With such models we have two hypotheses that must be rejected in order for positive conclusions to be drawn. The first, the null hypothesis, states that the model itself does not predict the outcome variable, i.e. H0∶β1= … =βi = 0. To reject this hypothesis we calculate the F statistic, given by the equation
F = R2/m
(1−R2)/(n−m−1), (6.2)
where R2is the coefficient of multiple determination, m is the number of predictors and n is the sample size. The F statistic is then usually compared to a critical value to determine whether it is significant at 5%, though SPSS gives the exact significance when performing regression analysis. If this hypothesis is dismissed, the model is a good predictor of the outcome variable, and one of the βi may be statistically significantly different to zero.
The second, alternative, hypothesis applies to each individual coefficient and states that it is not statistically significantly different to zero. This test statistic t is given by the parameter estimate, i.e. the unstandardised coefficient, divided by its standard error
t= βi
s(βi). (6.3)
The calculation of the standard error of βi can be found in Lomax (2001, pg. 244). The t-statistic is also usually compared to a critical value but once again SPSS gives its exact significance.
For each module on which regression analysis was performed, conclusions could only be drawn if both the F- and t-statistics were suitably significant, and so the two questions that essentially had to be answered were Is the model a good predictor of the module mark? and Does the 1Y coefficient contribute a significant amount?