Study outlines

This section introduces the studies reported in this thesis and maps these to research areas and the corresponding chapters.

Chapter 4: Exploration of student application data

The studies reported in Chapter 4 aim to fulfil Objective 3 by identifying potential indicators of learning success amongst three academic years of student application data and previous

qualification grades. Correlation analyses evaluate the indicative value of student application data (such as UCAS points and age). The L4 programming performance of A-level and BTEC entrants, their gender, geographical area of previous education and nationality are also evaluated with

nonparametric boxplots (Tukey 1977). Similar data was collected and analysed from an international partner college for purposes of replication (Fincher & Petre 2004).

3-88 International differences regarding pre-undergraduate qualifications meant that specific

relationships between UK BTEC and A-level qualifications could not be validated by the partner college dataset. However, additional questionnaire data was collected from one cohort of Bucks students, and interviews were conducted with A-level and BTEC students, to better understand their experiences and to propose possible reasons for trends identified from quantitative analysis (Fincher & Petre 2004: 15).

The research objectives of a number of studies reported here often concerned whether differences existed between the grade performance averages of populations. Examples of hypothesis tests were examining differences between male and female students, between BTEC and A-level entrants, between ethnic groups or Local Education Authority (LEA) regions. For such tests, normality of grade distribution was not assumed and the non-parametric and data-visualisation means of boxplots were applied. Boxplots test for differences in medians of populations and usefully demonstrate data distributions as well as the presence of outliers (Chambers et al. 1983; Tukey 1977). The ggplot2 library (Wickham 2009) of the R language (R Core Team 2013) was used to generate boxplots applied to the analyses reported here.

Where analyses hypothesised that two continuous variables were significantly related, for example UCAS points and programming grade performance, Pearson’s correlation was applied. Although parametric assumptions for the normality of grade performance may have been breached for some distributions and Spearman’s Rank correlation may have been applicable, Pearson’s correlation was nevertheless applied for the reasons that grades are true ratio-scale data. Pearson’s is also generally consistent with the approach used by the research sector (Pioro 2006; Stein 2002).

Chapter 5: Exploration of student engagement data

The studies reported in Chapter 5 aim to fulfil Objective 4 by investigating how L4 grades at Bucks and L6 grades at the South Asian Institute for Technology and Mathematics (SAITM) in Sri Lanka correlated with indicators of course engagement. Indicators of course engagement included attendance at both institutions and more detailed indicators from questionnaires concerning engagement behaviours at Bucks.

This study also extends and modifies early work by Mather (2014). Questionnaires were distributed to elicit student perceptions of an L4 introductory programming course to better understand how attitudes and working habits relate to learning progress. The questionnaire adopted a five-item

3-89 ordinal Likert scale for responses and included a five-question test to evaluate understanding of concepts introduced during the programming course. These variables were subject to correlation and multivariate analyses with other metrics, such as for attendance and grades.

Studies that use audio/visual recordings to explore collaboration amongst programmers (Plonka et al. 2011; Nawahdah & Taji 2015; Mather 2004; Zarb & Hughes 2012) demonstrate a wide range of research approaches and analysis techniques. Thus, authors reporting on the analysis of recordings made of commercial programmers working together (e.g. Plonka et al. 2011; Zarb & Hughes 2012), were mainly concerned with the impact of collaboration on business processes, whereas other researchers were interested in the educational impact of collaboration (Nawahdah & Taji 2015; Mather 2004).

Due to the volume of “rich data” involved, Jewitt (2012) after Snell (2011) cautions researchers against analysis that may become “overly descriptive”. Although not using Snell’s technique of pairing “systematic quantitative analysis” with “micro-ethnographic qualitative analysis”, a similar mixed method approach with limited qualitative sampling is adopted here. Quantitative analysis of the video data measured length of time spent on task as well as word and theme frequencies. Qualitative analysis techniques sought to interpret dialogue and actions for underlying working strategies and motivations for learning. Zarb and colleagues’ (2012: 2) two stage analysis process was adopted for efficient interpretation of meaningful video segments. The first stage identifies the most relevant and meaningful video segments for the investigation, and the second stage

concentrates on conducting extensive analysis of those segments.

Chapter 6: Evaluation of teaching interventions and course modifications

Chapter 6 aims to fulfil Objective 5 by evaluating the impact of modifications made to Bucks’ L4 programming course, following certain recommendations from Chapter 5, through the quantitative analysis of attendance and grade data, supplemented by further qualitative analysis of

ethnographic-type observations, interviews and informal conversations. This study discusses the evolution of formative assessment given to successive L4 cohorts at Bucks and includes correlation analysis between formative assessment results and final L4 grades. Summative assessment

modifications are also discussed in the light of results from quantitative analysis of attendance and grade data. Course curriculum changes, teaching methods, and remedial interventions are discussed with reference to the qualitative analysis of ethnographic observations and student feedback.

3-90 The effectiveness of programming pedagogies that encourage active learning are typically evaluated numerically by their impact on grades and retention (Freeman et al. 2014; Porter et al. 2016), in addition to their impact on numerical measures of engagement such as attendance or perceptions of enjoyment and satisfaction (Fotaris et al. 2015). However, increases to numerical measurements of course engagement do not necessarily reflect improved quality of teaching and learning. For example, students may feel more satisfied with their course if they achieve a high grade. They may attend for factors completely unrelated to teaching style. Even though large MIMN studies that analyse large sample sizes provide confidence in results, these findings are rarely considered in the context of existing educational learning theories and lack explanatory value (Almstrum et al. 2005; Doane 2013).

Even though this research does intend to measure impact on grade and attendance, seeking to understand underlying explanatory reasons for engagement has greater value for informing future teaching methods. Therefore, this research aims to address these concerns by discussing

observations, interviews and informal comments in relation to prevalent cognitive and constructivist theories, such as Neo-Piagetian theory (Teague et al. 2013) after Piaget (1952), scaffolding (Robins 2010), cognitive load theory (Sweller et al. 2011), in addition to research concerning attentiveness (Davidson 2011) and cognitive control: distractions and procrastination (Gazzaley & Rosen 2016; Rosen et al. 2011), for evidencing teaching interventions in contemporary contexts (Gordon 2016).

4-91