Analysis of trends between applicant data and programming grade performance

Methods

The quantitative analysis in this section investigates the relationships between L4 grade performance and applicant profile data such as gender, DOB, Local Education Authority (LEA) catchment, nationality, ethnicity, and degree course. As these attributes are known in advance of undergraduate entrance, discovery of relationships would enable early intervention. The use of notched boxplot analysis is continued in this study to visualise and determine statistical significance between different categories of data. Pearson’s correlation analysis is also applied to continuous variables such as age and grade to determine their relationship.

This data was obtained for the identical sample of academic years 2013 – 16 analysed in Section 4.1. All non-numerical data was quantised for correlational analysis. Student age, at time of commencing study, was calculated from DOB. Other attributes such as gender, degree course, nationality, ethnic background and LEA data were assigned numbered categories (non-weighted). Students applied from a wide range of geographical regions in the UK. London boroughs were, however, grouped together for comparison with Buckinghamshire and surrounding regions.

As previously stated in Section 4.1, international differences meant that some comparisons could not be performed due to insufficient or incompatible data. The LEA catchment data is a UK metric and does not exist in Sri Lanka. Term addresses were recorded but not locations of previous education. In addition, every student enrolled at SAITM was Sri Lankan, thus a comparison of performance differences between nationality groups was not possible with the Sri Lankan dataset. All SAITM students were enrolled on the same degree course - BSc Hons Computing (Interactive Media) – also nullifying a comparison of performance differences.

The programming performance of men and women

Notched boxplot analysis (Figure 4.4) demonstrates that the median L4 programming grade was greater for L4 Bucks men (62%) than for women (56%). This sample contains a disproportionately high number of men to women, which is reflective of findings reported in Section 2.1 (Carter & Jenkins 2001; Universities UK 2015: 23). Whilst some women are prone to withdrawing from computing related courses (Lewis et al. 2006: 6; Margolis & Fisher 2002: 80), Figure 4.4

demonstrates that all women attempted L4 programming assessments. In contrast, some men did not submit assignments (they were given ‘0’ for a non-submission).

4-112 L6 Sri Lankan women (70%) achieved a higher average programming grade than their L6 male peers (62%) (Figure 4.4). Notched boxplot analysis illustrates that L6 Sri Lankan women (70%) had a significantly higher average Programming Grade than L6 Bucks women (43%) (Chambers et al. 1983: 62). L6 Sri Lankan women (70%) also had a higher median grade than L4 women (56%) (although not regarded to be significant). However, median grade performance amongst men is more consistent. Figure 4.4 also illustrates a wider range of grades for L4 men and women compared to L6 men and women. Grade distribution of Bucks L4 programming courses (Bucks 2015a; Bucks 2015b; Bucks 2016a; Bucks 2016b) is consistent with bimodal distributions commonly reported in literature (Robins 2010: 4). Whereas L6 grade distributions tend to be lower, as low achieving L4 and L5 students generally do not progress to L6.

Figure 4.4. Notched boxplot comparisons of programming grades of L4 & L6 Bucks and L6 Sri Lankan students. Notes: (1) n for L6 Sri Lankan categories is: Female, 12, Male, 50; n for L4 Bucks categories is: Female, 15, Male, 105; and n for L6 Bucks categories is Female, 8, Male, 35.

4-113 The usefulness of student age as a predictor of programming success

Figure 4.5illustrates the age distribution of Bucks L4 students. Most students are aged between 18 and 21 (102 out of 120; 85%), and these vastly outnumber mature students (older than 21) (18 out of 120; 15%). This ratio is consistent with reports of rising applications from young students, and declining mature student applications (Department for Education 2016; Independent Commission on Fees 2015: 17).

Pearson’s Correlation analysis revealed a slightly negative but significant correlation (r = -0.202, p = 0.027, n = 120) between student age and programming performance. This suggests that younger students achieve marginally higher grades than mature students.

Figure 4.5. Age distribution of L4 Bucks students. Notes: (1) n = 120.

4-114 Figure 4.6illustrates the age distribution of L6 Sri Lankan students. In addition to entering two years later than L4 students, Sri Lankan students are required to complete a year of bridging work in preparation for undertaking L6 courses. Consistent with the distribution of L4 Bucks entrants, the number of younger L6 Sri Lankan learners vastly outnumbers the population of mature learners. Interestingly, a few students are aged between 18 and 19, which is younger than the expected age of L6 entrance in the Sri Lankan cohort. However, there are exceptional circumstances in which some Sri Lankan students start compulsory education one or two years earlier than their peers (No author, n.d.), which may explain this pattern.

Pearson’s Correlation analysis reveals a positive but weak and non-significant relationship (r = 0.200, p = 0.119, n = 62) between student age and L6 programming grade. This suggests that some mature students perform marginally better than younger students. Despite the weak correlation, this positive relationship contrasts the negative relationship with Bucks data (r = -0.202, p = 0.027, n = 120).

Figure 4.6. Age distribution of L6 Sri Lankan students. Notes: (1) n = 64.

4-115 LEA, Nationality, Ethnic Background

Figure 4.7illustrates the distribution of programming grades across LEA (Local Education Authority) boundaries - the catchment area where students completed their previous study. Some geographical regions represented by one student were omitted from analysis. Whilst larger sample sizes are desirable, regions featuring a small number of students were included for comparison with more populous areas. The largest population of students applied from the county of Buckinghamshire (in which Bucks New University is located). Train and bus links also enable London students to

commute, which might explain why it is the second largest sample. However, notched boxplot analysis illustrates that the average grade for London students (48%) is lower than that for Buckinghamshire students (73%) (Figure 4.7).

The median grade was significantly lower for Slough (40%) and Milton Keynes (26%) students than for most regions. Further investigation revealed that students from these two regions were previously taught by the researcher. Factors such as low confidence, poor time management, and lack of integration with peers are thought to influence performance more than LEA.

Figure 4.7. Notched boxplot comparisons of programming grade across locations of previous study. Notes: (1) n for categories is: Bucks, 38; Slough, 6; Oxfordshire, 5; Windsor, 3; Milton Keynes, 4; Hertfordshire, 3; London, 19; Essex, 5 & Kent, 3.

4-116 Most students in this sample were born in the UK, and Figure 4.8 illustrates minimal difference between the median programming grade of UK (62%), European (70%) and Asian (66%) students. However, African students (median of 30%) scored significantly lower grades (Chambers et al. 1983: 62) than the other three nationalities. Despite this comparison being significantly different, the sample size of five African students is too small to conclude that all African students are vulnerable in programming courses. However, it is notable for further investigation that three of these five students are mature students and it is known that such students have difficulty integrating with traditional HE environments; consistent with literature (Gordon 2016: 6).

Figure 4.8. Notched boxplot comparisons of programming grades and nationality categories. Notes: (1) n for categories is: UK, 96; Other European countries, 10; Asia, 9 & Africa, 5.

4-117 Although most students are white, there are sizeable populations of ‘black and ethnic minority’ (BME) students. This is consistent with national statistics which report that undergraduate computer science courses often attract higher than average numbers of BME students (CPHC 2012; Gordon 2016: 6). Notched boxplot analysis in Figure 4.9 indicates that white students (72%) achieved a higher median programming grade than Asian students (56%). This also corroborates national attainment reports (CPHC 2012: 9; HEFCE 2014: 3). White students, however, did achieve a broader range of programming grades compared with Asian students (Figure 4.9).

The modest number of Chinese students (77%) and students of ‘Other Mixed Background’ (87%) achieved the highest median grades of all ethnic categories. This would appear to support western stereotypes regarding Chinese attitudes towards work (Phillips 2015) although this does not account for individual variance.

Figure 4.9. Notched boxplot comparisons for programming grade across ethnicity categories. Notes: (1) n for categories is: White, 64; Asian or Asian British, 21; Other Asian Background, 10; Black or Black British, 13; Other Black Background, 2; Chinese, 2; Other mixed background, 3 & Not given, 5.

4-118 Comparison of Degree course (choice of)

Each of Bucks’ six degree courses emphasise a particular aspect of computing but all students are required to complete the introductory programming course in their first semester. Independent Games Production was introduced in the 2015/16 academic year, which may explain the smaller number of students for their sample. However, the Software Engineering degree course was being delivered when the 2013/14 and 2014/15 samples were collected.

Notched boxplot analysis in Figure 4.10illustrates consistent grade ranges and median scores across most degree course programmes. None of the Independent Games Production group failed the course. However, this finding may reflect one cohort of students (2015/16) rather than the course itself. Figure 4.10illustrates that the median grade for the Software Engineering students (42%) is considerably lower than the other courses too. However, further investigation revealed that three students (of nine in the sample) had already been identified to have performed poorly for reasons unrelated to LEA, and may therefore also distort the typical performance of Software Engineers.

Figure 4.10. Notched boxplot comparisons between programming grade and course categories. Notes: (1) n for categories is: Foundation Computing, 16; Computing (BSc), 38; Computing & Web (BSc), 20; Software Engineering (BSc), 12; Games Production (BSc), 25 & Independent Games Production (BA), 9.

4-119

Summary of Section 4.2

This section discovered that the range of L4 programming grades for men and women is significantly greater than that for L6 men and women (Figure 4.4). This greater range reflects the bimodal

distribution reported in literature (Robins 2010); the population of those who fail is as great as those who achieve over 70%. Fewer students fail at L6 due to the requirement to pass L4 and L5, and the optional L6 programming courses tend to attract the most motivated and able students. Figure 4.4 illustrates that the average grade for L6 Sri Lankan men is consistent with the grade for L6 Bucks men; however, L6 Sri Lankan women achieved a significantly higher grade than L6 Bucks women. However, this is likely to be due to individual circumstances that caused L6 Bucks women to underperform, and exceptional L6 Sri Lankan women who outperformed their male counterparts. There are significantly greater populations of young students than mature students in both the L4 Bucks dataset and the L6 Sri Lankan cohorts (Figure 4.5 & Figure 4.6). This reflects the difficulty of recruiting and retaining mature students (Gordon 2016: 6; Independent Commission on Fees 2015: 17). Although Pearson’s correlation analysis demonstrates a weak correlation between age and grade for both cohorts, the correlation was positive for L4 cohorts and negative for L6 cohorts. A possible explanation is that more mature students retook L4 courses than for L6 courses. However, these correlations are too weak to consider age as an indicator of programming grade.

Comparison of LEA attributes identified that Slough students achieved significantly lower grades than other regions (Figure 4.7). However, further investigation discovered that course engagement factors offer more explanatory value for poor attainment than the region of prior study. Notched boxplot analysis also identified that African students achieved significantly lower grades than their UK, Asian and European peers (Figure 4.8). However, this sample of African students comprised mostly mature learners previously identified to perform poorly for various reasons. Figure 4.9 illustrates that BME students are often outperformed by white students, which corroborates national reports (Gordon 2016).

Finally, notched boxplot analysis (Figure 4.10) demonstrates consistent performance across most degree programmes, except for Software Engineering. But as previously established, the Software Engineering sample features mature students and other young students identified for their poor performance. It is notable that Games Development students achieved the highest median grade. Section 5.3 goes on to explore this trend further and attributes their performance to behaviours representative of social learning theories (Lave & Wenger 1991; Wenger 1998); consistency, working discipline and belongingness (Thomas 2012a).

4-120

4.3 Chapter 4 Conclusion

This chapter set out to investigate whether L4 programming performance differences existed between A-level and BTEC entrants, in addition to evaluating the usefulness of typical applicant data for predicting L4 performance. This chapter addresses Objective 3 “Evaluate the usefulness of prior

qualifications for preparing students to learn computer programming in a higher education establishment” as stated in Section 1.5.

Section 4.1 did identify patterns of performance unique to A-levels and BTEC entrants. However, A- level students in general (both A-level students who took ICT or Computing, and A-level students who did not) achieved higher L4 programming grades than their BTEC peers. Although small sample sizes do limit the validity of findings related to A-level Computing and A-level ICT, the total number of A-level students and BTEC students sampled in this study is similar in size (40 each) (Figure 4.2). This merits further investigation to discover what factors of each qualification explain this trend. However, this chapter also found minimal difference between the L4 programming performance of those with prior computing or ICT related qualifications and those without. It is also striking that most A-level entrants - identified to achieve a grade higher than BTEC entrants - did not study Computing or IT subjects. Qualitative analysis of interview transcripts found that prior experience does not guarantee a strong understanding of programming concepts due to varied teaching quality and content covered at each institution. Moreover, both of these findings suggest that specialist programming knowledge or experience is not required for success in L4 programming courses. Chapter 5 contributes additional evidence to triangulate this finding.

Section 4.2 discovered that patterns of performance are generally consistent with the published literature; in terms of BME and mature learners as well the wide range of L4 grade that reflects a bimodal distribution. However, whilst notched boxplot analysis identified trends of performance amongst other categories (LEA, nationality, and degree course choice), further investigation found that these samples tended to contain those already identified to perform poorly due to other reasons.

Thus, whilst some measurements of L3 performance and student profile data included on

undergraduate applications may indicate L4 performance, alone, these are too reductionist (Thomas 2012a: 4). Therefore, the next chapter evaluates the indicative and explanatory value of measures of engagement such as L4 attendance, performance across all L4 courses, and peer integration and communication.

5-121