The assessment of programming courses

Whilst assessment varies from institution to institution, introductory programming courses tend to adopt three main assessment formats. Bennedsen and Caspersen (2007: 35) found, in their survey of 63 CS1 courses, that “on average, 35% of the final grade is due to marked assignments during the course, 35% is from the final exam, and 30% is from some other source (e.g. lab-exercises, midterm exams, or programming projects).” The advantages and disadvantages of these different types of assessment are discussed below.

Laboratory exercises

Laboratory exercises are generally based on smaller and, by their nature, less complex problems. They are often supplied in practical laboratory sessions and provide active learning opportunities for students as well as feedback for tutors. Some institutions assess these exercises and others utilise them as formative activities which prepare students for larger assignments or projects (Kolling & Barnes 2004: 287).

2-70 However, small-scale laboratory programming exercises are more vulnerable to plagiarism because students can easily copy solutions from the internet or from peers (Ahadi & Matheison 2019a). The limited scope of the exercises usually means that there will be fewer alternative solutions and this tempts students to think copied attempts will be harder to identify (Luck & Joy 1998). Williams (2006) found that fewer plagiarism cases were reported in classes when Pair Programming was implemented as the method rewards students working together rather than penalising them. Projects and assignments

Projects and assignments tend to be modelled on larger and more complex scenarios than individual laboratory exercises. They often require students to document the design, testing, and evaluation of their programs, mirroring the processes required in formal software development lifecycles

(Sommerville 2010: 30). In addition to assignment and laboratory exercise submissions, some instructors ask students to ‘walkthrough’ their code (like an informal interview) to ensure code is understood (McCracken et al. 2001: 3).

Even though later laboratory exercises usually build upon concepts introduced in earlier weeks, in any given session exercises tend to focus upon the topics for the current week. However,

assignments often require students to integrate subject knowledge over longer periods (Kolling & Barnes 2004: 287). Studies suggest that students struggle to do this (Luxton-Reily et al. 2017; McCracken et al. 2001), even if they are able to successfully answer questions on isolated concepts or individual topics (Luxton-Reily et al. 2017).

Exams

Computer science courses adopted the examination assessment technique common to other disciplines during the 1960s at a time when undergraduate programmes were being formalised (Lloyd 2013; Jones 1999; Navarro-Lopez 2014). Most introductory programming courses measure learning progress by examinations at the end of the curriculum and some also feature intermediary mid-term exams too. Literature indicates that multiple choice questions (MC) and constructed response questions (CR) are preferred formats for programming examinations. Although online MC exams may offer the convenience of automatic marking, MC questions tend to assess different skills to CR questions (Davidson 2011: 114). MC questions suit the assessment of factual content with defined correct answers (Davidson 2011: 111), whereas, CR questions require students to explain their understanding of concepts and programming scenarios in their own words. The CR format is therefore regarded to more accurately assess ‘higher-order’ thinking skills (similar to essays) and understanding (Davidson 2011: 114 - 115).

2-71 Both MC and CR responses are still vulnerable to memorisation techniques (associated with the surface learning approach) and plagiarism (Luck & Joy 1998). However, Azalov and colleagues (2004) discovered that CR assessment was more strongly correlated with final grades for an undergraduate introductory CS course than a related MC assessment.

Parsons and colleagues (2015) highlighted past studies (Lister et al. 2004; Dehnadi & Bornat 2009) that gave students deliberately complicated and unrealistic code fragments to trace for exam questions. These questions are not logical and not representative of the types of scenarios that commercial programmers work to solve in industry. Incorrect answers to such questions cause educators to wrongly assume that students cannot code (Parsons et al. 2015).

Formative Assessment

The above three types of assessment are typically used as summative instruments towards final grades (Bennedsen & Caspersen 2007) rather than for formative assessment which is primarily to support learning (Lilley & Barker 2007). When there is less emphasis on grades and marks, this removes much of the distraction and pressure for students and also reduces cognitive load (Margulieux et al. 2012; Sweller et al. 2011). This in turn enables students to concentrate on their development (Gibbs 2015). Gibbs (2015) goes further to suggest that limiting summative assessment to simple pass/fail options would remove much of the unnecessary pressure that students face during their first semester.

Formative assessment types widely range from quizzes (University of Exeter n.d.; Radenski 2007; Roberts 2006; Denny et al. 2008) to game activities (Fotaris et al. 2015). Most applications of formative activities report benefits in learning gain and increased student engagement. Feedback from formative activities also informs modifications to teaching content and delivery (OECD 2005: 25). Corbett and Anderson (2001) discovered feedback to be most effective when it is

communicated immediately after the unassessed task and addresses specific steps and features in student work. However, feedback is only effective to the extent that students respond and adjust accordingly (Gibbs 2015; Koulouri et al. 2014: 25).

2-72

Summary of Section 2.3

This section considers literature concerned with the design and delivery of introductory programming courses.

Studies concerning the components that form programming course environments (Section 2.3.1) suggest that students in small class sizes performed better than those in larger classes (Bennedsen & Caspersen 2007). This is thought to be due to increased opportunities for the collaborative and peer learning behaviours that are discussed in Section 2.3.2. Section 2.3.1 also demonstrated a tension between programming languages and the environments designed to be accessible to novices, and the sophistication of tools used in the workplace which, because of their complexity, may hinder introductory-level learning (Jenkins 2001a; McIver & Conway 1996). Industry requires students to possess fundamental and transferable skills in order that they may adapt to changes in the workplace (Section 1.4). Supplementary textbooks were reported not to be particularly helpful for imparting the problem-solving techniques, which, in Sections 2.1.1 and 2.2.2 were found to be a strong predictor of programming success. Nevertheless, in the place of textbooks students increasingly consult alternative online sources for programming help (Utting et al. 2013).

Section 2.3.2 discussed recent developments in pedagogical approaches to teaching programming which are designed to encourage active learning. Such techniques are associated with reducing failure rates (Freeman et al. 2014 Porter et al. 2013a), increasing engagement and motivation (Constantine 1995; Fotaris et al. 2015; McDowell et al. 2006; Nosek 1998; Williams & Kessler 1999), as well as creating efficient and effective peer learning opportunities (Simon et al. 2010; Zingaro et al. 2010), due to students developing similar levels of cognition (Wiggins & McTighe 2006). However, whilst students may be able to develop solutions in less time, this process may not lead to greater and deeper understanding of programming concepts (Eiriksdottir & Catrambone 2011; Mather 2014; Radermacher et al. 2012; Zingaro 2014b). The extent to which learning goals and outcomes are met depends greatly on motivation, which in turn informs student learning approaches and priorities (Section 2.2.2 & Section 2.1.3). Despite significant benefits reported for innovative teaching

practices, educators required more evidence before being willing to adopt new techniques (Perrotta et al. 2013). Educators who were comfortable with familiar teaching methods and materials were often unwilling to make major changes to their courses (Ni 2009; Reis 1998). However, some

educators were more willing to make minor adjustments. Changes to teaching approaches must also be clearly communicated to students if confusion and disengagement are to be avoided (Bevan et al. 2002; Han & Basheti 2010; Porter et al. 2013a; Williams et al. 2008).

2-73 Section 2.3.3 reviewed techniques for assessing programming ability. Laboratory exercises introduce problem-solving in a small and manageable context but are more susceptible to plagiarism. Although assignments and projects provide opportunities to set more complex and challenging problems, most novices have difficulty applying multiple programming concepts and understanding the relationships between concepts (Luxton-Reily et al. 2017). Regarding examinations, constructed response (CR) exam questions proved to be better at predicting final grades than multiple choice (MC) questions (Azalov et al. 2004). However, exam questions can be unrepresentative of industry problems (Barker et al. 2015), and may also misrepresent real coding ability in the workplace. The fact that students also express a fear of exams, known as test anxiety, which can detrimentally affect learning performance, was considered in Section 2.4.