Quantitative analysis

5.1.1.1 Data

An overview of the results appears in Table 16. The full results appear in appendix 8.5.

Average

When considered by ranking, the postgraduate students are entirely in ranks 1-4 of scores awarded (there was a large number of rankings shared between 2 or more candidates), with an average overall mark of 5.71 and no individual category awarded below a 5 (of a maximum mark of 6). Among these students the strongest skill was academic language use (5.85 average) and the weakest coherence and cohesion (5.57 average), the other two categories scoring 5.71 average each.

Among the undergraduate students the average mark was a whole band lower at 4.7. As with the postgraduate students, academic language use was the strongest category (5.1 average).

Notably, coherence and cohesion scored below the overall undergraduate average at 4.5 (slightly over a band lower than the postgraduates), and the weakest category was engagement with sources (4.1 average, slightly over a band and a half lower than the postgraduate students).

University grades are listed in appendix 8.6, and averages in Table 16. The number of grades given varied significantly: two of the postgraduate students reported one grade only (the overall grade for their non-modular qualification) while some had as many as 11 separate grades. All of the undergraduate students were enrolled in modular assessment and therefore had between 2 and 8 grades for the year. Taking both cohorts together, the standard deviation (not including the two students with one grade) was 4.35 and the average number of grades reported was 4.95. In terms of the undergraduate versus the postgraduate averages, I note that the postgraduate grade average is over ten marks higher than the undergraduate. This is not surprising, as to some extent a good undergraduate score is necessary to enrol on a postgraduate degree and therefore those on postgraduate courses would have been at the higher end of their undergraduate cohorts. Additionally, it is worth mentioning that the pass mark on postgraduate courses is often higher than for undergraduate, at 50% rather than 40%.

The sample’s grades did not cover a wide range of abilities: they were almost entirely 2.1 and above i.e. the more successful students in a cohort, with no students receiving a 2.2 and one

demonstrated: grades are from a range of universities and from a range of examiners. For reasons discussed in section 2.2, this is an unavoidable issue when using opportunity sampling.

Self- and tutor assessments are listed in appendices 8.7 and.8.8.

The self- and tutor assessment results were considered first as raw scores. However, as no standardisation took place in the use of the scales, there was little variation in mean score between candidates (with an average of 4.13, standard deviation of 0.77 for self -assessment and an average of 4.40 and standard deviation of 0.74 for tutors), despite dif fering grades and ALT results. Self- and tutor assessment results were then normalised to focus the analysis on variation in scores.

It was only possible to get tutor assessments for a small number of the candidates (n = 5).

This was for a few reasons: first, many of the postgraduate students were no longer in contact with their tutors, or were part of large cohorts where the tutor would have limited knowledge of a particular student’s performance. Second, a few tutors had concerns over confidentiality which could not be allayed by discussing the consent given by participants. Finally, due to time constraints the second half of results collection took place at the same time as the nationwide university staff strike, meaning that some participants’ tutors di d not have the time to provide an assessment. The small number of tutor results available means that the tutor assessment results reported below must be extremely tentative.

5.1.1.2 Analysis

Correlations were calculated as follows: for overall correlations, grades, overall tutor assessment and overall self-assessment were correlated with the overall CAEALT score. For each of the four separate categories (argument, coherence and cohesion, academic language use and engagement with sources), university grade s and the subskills relating to each category in the tutor and self-assessments were correlated with the CAEALT score in each category.

5.1.1.3 Grades to ALT

Figure 3: Grades to CAEALT, overall 40

45 50 55 60 65 70 75 80 85 90

3 3.5 4 4.5 5 5.5 6

Grades

ALT overall

Grades to ALT: overall

Figure 4: Grades to CAEALT, argument

Figure 5: Grades to CAEALT, Academic language use 40

Grades to ALT: Academic language use

Figure 6: Grades to CAEALT, coherence and cohesion

Figure 7: Grades to CAEALT, engagement with sources 40

Grades to ALT: Coherence and cohesion

40

Grades to ALT: Engagement with sources

Figures 3-7 show the correlation between university grades and CAEALT results (overall and by each marking criterion). As discussed in section 3.2.5.1, a correlation coefficient of over 0.2 is considered acceptable when analysing admissions tests. This study will follow the conventional thresholds of a p-value of 0.05 and effect size of 0.50.

Correlation coefficient (Kendall’s Tau, τ)

P-value Effect size (Cohen’s d)

Overall 0.518 0.04 8.42

Argument 0.344 0.074 8.4

Coherence &

cohesion

0.576 0.003 8.43

Academic language use

0.235 0.233 8.39

Engagement with sources

0.467 0.014 8.42

Table 17: Grades to CAEALT results

Three of the five measures show a significant correlation (p < 0.05) with university grades: the overall CAEALT mark, coherence and cohesion, engagement with sources. Argument has a p -value slightly higher than usually accepted, but the correlation coefficient and effect size suggest that this category should be tentatively included as approaching significance.

Academic language use is the weakest of these measures and the correlation with the grades here is not significant: a larger study is necessary to determine if this is a possible measure of academic literacy.

5.1.1.4 Self- and tutor assessment to ALT

Figure 8: Self-assessment to CAEALT, overall

Figure 9: tutor assessment to CAEALT, overall 3

3.5 4 4.5 5 5.5 6

3.00 3.20 3.40 3.60 3.80 4.00 4.20 4.40 4.60 4.80 5.00

ALT overall score

Self-assessment overall

ALT to self-assessment: overall

3.5 4 4.5 5 5.5 6

3.00 3.20 3.40 3.60 3.80 4.00 4.20 4.40 4.60 4.80 5.00

ALT overall score

Tutor assessment overall

ALT to tutor assessment: overall

Self-assessment (n=18) Tutor assessment (n=5)

Table 18: CAEALT results to self- and tutor assessment

Figure 8 and Figure 9 show the relationship between CAEALT results and overall self-/tutor assessment. Correlation coefficients and p-values overall and for each criterion are in Table 18. Neither self- nor tutor assessment showed any significant correlation with the CAEALT scores. Further discussion of this lack of correlation will take place in the discussion section; as previously stated, the number of tutor assessments available is extremely small (n = 5) and therefore no meaningful conclusions can be drawn from this data.

Correlations were also calculated between self-assessment and university grades and between tutor assessment and grades, both for the overall results and for each criterion.

However, no further correlations of interest were found.