Exam Marking Tool

Marking of end of semester exams is a complex and intensive process, where numerous academics work on shared data sets against an aggressive time schedule. It is common at St Andrews for marking of exam papers to be shared by several members of staff, in some cases with different academics marking different questions. This poses challenges in collating the marks in an accurate and timely manner once prepared. There is a requirement for confidentiality in handling of marks, which means many easy options for sharing files such as e-mail or Dropbox may be unsuitable.

Clearly, there is a need for a tool which can collect these marks, and automate the process of collating them. The coursework tool had several key limitations that prevented it from being used for marking end of semester exams:

• Configuration and mark entry interfaces are orientated towards managing a single assignment at a time. Each exam question would require a separate assignment entry, meaning the configuration and marking would be impractically time consuming due to need to constantly change between pages during the processes.

• No provision was made for optional assessments within sets. Exams frequently have sets of questions from which the student picks a subset to answer, and the coursework tool could not support this.

• There was no support for transforming marking scales. Exams are typically marked on a linear scale, but may then have marks transformed to a different scale (for ex- ample the 20-point grade scale used at St Andrews).

Note that these constraints reflect the coursework tool in 2005 when the exam tool was designed, support for optional assessments and mark transformation have been since added to the coursework tool.

The exam tool (shown in figure 7.8) provided a single point of truth for student’s exam marks, and was the first tool to audit changes to marks entered. This audit trail was key to encouraging confidence in the correctness of the tool, enabling staff to quickly determine the origin of each mark entered. Staff could work independently and asynchronously, with user input managed as a set of changes to be applied. This meant the staff responsible for

marking different sections of an exam could work concurrently on the same exam, without risk of accidentally overwriting other users’ changes. This is particularly important as it is common for multiple staff to be marking different exam scripts or sections of the same exam script in parallel.

Note that deleting a section will also delete any result data currently entered for questions in that section.

Section Questions Required

Answers Marks per question Deletesection Move

Status Enable status change. Warning: Only system adminstrators can change status once set unmodifiable. Exam and marks can be modified. Students see nothing.

Uploaded question paper: question paper

A 1 Move Up_{Move Down}

B 4 Move Up

Move Down

Total questions 5 Required answers 4 Total marks in paper 140 Maximum mark 120

Update Paper

Delete

entry Percentage (%) Mark (out of 120) Grade Percentage (%) GradeNew mapping rules

1 2 3 4 5 6

Interpolation type: Linear interpolation

Replace map with: Select a model Update Map

Anonymous marking Identification Style: Show only Student IDs

Reporting Style: A single grade is produced, summing over all sections

no file selected Choose File Upload CSV

Uploaded CSV file should have the percentage of maximum mark in the first column, and the corresponding grade in the second column. The downloadable CSV file for this map is an example of this format.

Last Changes

Actor When Action Description Status

Livesey, MichaelJan 14, 2008 3:49:49 PM reset Resetting grade map to model: Guide points May 05 - Pre-honours Succeed

1 60 3 20 0 0 0 19.5 23.4 2 39.5 47.4 4 54.5 65.4 9 69.5 83.4 17 99.5 119.4 20 2007/8-S1 CS1002 (Computer Science) CS1002 Exam

CS1002 - Setup Exam Exam Events ModulesLogout

Marks and Grades Setup Exam Globalmaps

Figure 7.8: Screenshots of Exam Mark Repository

The interface for entering marks was designed around a spreadsheet-like grid for, with individual questions represented as columns and students as rows. Questions were grouped in the interface by the exam section they belonged to. Exam sections in this context contain are sets of one or more questions, of which students are expected to answer a defined number. All questions within a single section are worth the same number of marks. For example a section may contain 3 questions each worth 20 points, of which students answer 2 questions for a maximum of 40 points available from that section.

Once marks have been entered, the marks from each section are then summed to give a student’s total mark for the paper. If a student answered more questions within a section than required, the highest marks from the section were used up to the expected number of individual marks.

Mark Conversion

St Andrews requires grades on a 20 point scale for reporting purposes, and as such exams typically required marks to be translated (or “mapped”) into grades. The mark conversion

tools described in 5.11.5 were initially developed solely for the exam tool. Later adoption of MMS to all academic schools introduced a requirement for accepting coursework marks on varying scales, which would then be converted before reporting. As such, the conversion process was extracted from the exam tool and generalised to allow other tools to utilise it.

Mark Reporting

As with the coursework tool, the exam tool implements an abstract interface for exposing grade data, allowing it to be used as a data source for the final grade tool. This interface is described in 5.13.