see that numbers can be transformed or “broken" into pieces to make computation easier and to recognise relationships between numbers. This type of grouping will be referred to as place-value strategy. The third grouping, c, will be called random strategy indicating that neither the as-presented nor place-value techniques were applied. Figure 2.11d is the commutative equivalent of Figure 2.11a but has the addition computation done incorrectly. This problem is immediately identified from the clear representation. The approach of reconstructing the solution steps from an audit trail of explicit problem-solving actions provides detail information which can be useful for assessment and feedback processes. The practical application requires a tool that can implement the features described above.
The assessment of processes and strategies is necessary for CAA systems. The first research question in the study investigates the design of CAA for the capture of problem solving steps. The second research question addresses the the need to be able to evaluate the higher-order thinking skills of the Blooms taxonomy using the captured data.
Research Question 2: Will the data captured lead to the detection of strategies and opportu- nities for detailed feedback on primary mathematics word problems?
2.8
Marking, grading and feedback production
After a student has solved a problem and submitted a response, the next step is marking and feedback production by a teacher or assessor. Teachers are required to assess students’ performances based on responses. While essential, marking is time-consuming, potentially error-prone and difficult to complete consistently without bias (Venable et al., 2012). When there are large student numbers, the overall time spent in marking increases dramatically and the time for individual feedback decreases. To address these challenges marking and feedback support tools are employed (Venable et al., 2012).
The purpose of a marking and feedback support tool is to improve the efficiency and effec- tiveness of marking and feedback activities (Berger and Dreher, 2011; Burrows and Shortis, 2011).
1. Improved efficiency of marking and feedback: Reduced marking time – Reduction of the effort in marking and feedback is important to teachers so that they can reduce their workload or devote effort to other teaching activities. Also, it may reduce stress levels in teachers
2.8 Marking, grading and feedback production 51
2. Improved effectiveness of marking and feedback: Improved accuracy and consistency of marks – Correct and fair marks are important to teaching staff, students, and administrators. More ‘objective’ marks that more correctly assess work are important to all concerned.
Marking can be done automatically, semi-automatically and manually.
2.8.1
Manual (online) assessment
As mentioned in the previous section hand-scoring students responses can be time-consuming and takes a substantial amount of teachers time as it relies completely on human effort. However, an advantage of this mode of scoring is that it can produce a high level of detailed feedback and personalization (Denton, 2003). A teacher who is familiar with the strengths and weaknesses of a particular student will be able to tailor his or her feedback to meet the needs of the student by supplying individualised comments and advice. However, manual assessments are subject to human factors such as boredom and fatigue of the teacher which often leads to inconsistencies.
In manually marked CAA systems, the design is similar to the traditional paper-and-pencil tests. The computational tool is used to capture responses from students while human assessors assign marks and provide feedback (see, for example, (Bennett et al., 2008). Studies have shown that manual marking can be frustrating (Bailey and Garner, 2010), stressful (Hogan et al., 2002), time-consuming (Ferns, 2011) and subject to inconsistencies (Brown et al., 2004).
The challenges with manual marking done with paper-and-pencil or computer are summarised below:
• Teachers have difficulty being consistent with assessment—namely, awarding grades based on performance measured against assessment criteria. It can be difficult for a teacher to consistently calculate marks when multiple weighted assessment criteria need to be factored into determining a final grade.
• It is necessary to moderate grades that have been generated by a variety of assessors because differences some occur due to the subjective evaluations by different assessors.
• It takes a large proportion of the teacher’s time to provide feedback and assessment, which often may be simply misunderstood or, at worst is ignored or never collected.
2.8 Marking, grading and feedback production 52
• Providing feedback can be repetitive in nature. For example, the same comment is often applicable to many students who have made the same error.
• Time constraints may mean that feedback is not returned to students quickly enough for it to ‘feed-forward’ into the next learning task.
• Administrative errors can occur when recording grades/marks.
2.8.2
Automatic assessment
A large number of studies have reported on implementing automatic assessments (see Conole and Warburton, 2005; Jordan, 2013). The most obvious motivation for fully automatic systems is the huge savings that could be realised in assessment costs. Automatic assessment also contributes to the objectivity and reliability of the grading system because the system is not affected by boredom and fatigue. However, as mentioned in Chapter 2, several of these studies do not consider the problem-solving process. Grading MCQs and product based assessment tasks are relatively easy as only the final product are compared with the standard or model answer. Grading and providing feedback on the problem-solving process is more challenging. Bescherer et al. (2011) argued that a fully automatic system requires the complete modelling of all students, considering all possible solution paths, errors and misconceptions they might have. Because of this limitation, fully automatic systems have limited opportunities for detailed and personalised feedback (Rebholz and Zimmermann, 2013). Others have argued that a computer does not have the same degree of background knowledge or level of inferential capability as a human marker.
2.8.3
Semi-automatic assessment
Semi-Automatic assessment aims to combine manual and computer efforts in the assessment process. It seeks to support the teacher in marking by providing automatic support where possible. It aims at providing the benefits of fully automatic CAA with the advantages of detailed assessment and feedback by human assessors. Several studies have explored this assessment approach. The principle behind semi-automatic systems is the reduction of repetitive tasks which overwhelms and bores human assessors. The computer is used to group identical components in students’ responses and the human assessor only has to make one judgment (with feedback) on the whole group of identical components. This is repeated over the whole assignment yields a huge speed increase for the markers. In recent
2.8 Marking, grading and feedback production 53
times semi-automatic techniques are increasingly being used in different CAA research areas including computer programming (Ahoniemi and Reinikainen, 2006), mathematical induction (Rebholz and Zimmermann, 2013), Entity-Relationship (ER) diagrams (Batmaz et al., 2009; Tselonis et al., 2005) and free texts (Kakkonen and Sutinen, 2009; Sargent, J et al., 2004). These investigations have applied different frameworks to combine human and computer assessment.
Sargent, J et al. (2004) use the term Human-Computer Collaborative Assessment (HCCA) for free text constructed response type problems. In this approach, the marking was viewed as a process of active collaboration between the human marker and the computer. The computer does the routine work of sorting student responses into different groups while humans make the important judgments on marks. This approach was used by Tselonis et al. (2005) for diagram marking problems. However, the HCCA approach did not provide much intelligence and personalised feedback in the design. Bescherer et al. (2011) developed the model called Intelligent Assessment. This model combines computer-based assessment with human assessment. First, computer software automatically assesses problems in order to detect standard problems and mistakes as well as standard solutions. These set of standard mistakes and solutions are defined when the program is written. Based on these standard solutions and mistakes, the computer is used to provide feedback to the user. Unclassified mistakes, non-standard solutions or solutions which cannot be assessed by the software are automatically forwarded to a human teacher. The teacher then assesses the solution manually and gives the feedback needed by the student. As a consequence, tutors do not need to concentrate on all possible solutions and can spend more time providing feedback on unusual solutions or weak students. The teacher can give more individual and more intensive assessment and formative feedback.
Batmaz et al. in their submission, improved on the HCCA model by developing a framework that used partial marking to mark solution groups manually and the Case-Based Reasoning (CBR) method to mark some of the groups automatically. These studies suggest that semi- automatic assessment approach has the potential to significantly reduce the time and effort required to mark and provide feedback comments to student work for assessment tasks that require constructed responses.
In summary, these studies demonstrated the potential of the approach; however the studies have not clearly focused on supporting broad feedback and improving the efficiency of human assessor. Table 2.3 summarises the features as outlined by Kakkonen and Sutinen (2009), Sargent, J et al. (2004) and Bescherer et al. (2011) of the three major ways marking and feedback can be done.