7.4 Student Assessment
7.4.1 Coursework Management
Virtually all modules at St Andrews have some form of assessed coursework, with only a tiny fraction limiting assessment to end of term exams. In the academic year 2012/3 alone, there were over 4,500 coursework assignments set for students. Clearly streamlining man- agement of coursework submission and marking is a significant opportunity for reducing overheads.
MMS provides a single multi-role tool for coursework submission, mark tracking and feed- back. It has evolved significantly since its initial development as replacement for the Doc- ument Approval Tool (DAT) from TAGS.
Online Assessment Tracking System
The first version of the coursework tool was the Online Assessment Tracking System (OATS). OATS provided facilities for students to upload coursework either to predefined assignments, or to “ad-hoc assignments”. Uploaded work could then be approved or re- jected by staff. Marks and/or written feedback could be set on the submission for the student to view.
Configuration of predefined coursework assignments required details such as assignment title, due dates (by which the student was expected to have submitted their work) and ac- knowledgement dates, by which staff were expected to have marked the coursework and/or provided feedback. Ad-hoc assignments were intended for use in exceptional cases such as submitting work in progress, adding supporting documentation to coursework after it has
been submitted, where configuration problems mean work cannot be submitted normally, or in any other cases where a predefined assignment was not a suitable answer.
Unfortunately ad-hoc assignments proved to be a significant cause of confusion, in partic- ular in best practises in using them. There were a number of occasions on which students misunderstood the interface, and uploaded work into the ad-hoc assignments despite the availability of a more appropriate predefined assignment. Further, no functionality had been provided to transfer coursework from these ad-hoc assignments into regular assign- ments, which made recovery from incorrect usage time consuming.
The accept/reject mechanism was used to indicate whether the work was as-expected, for example work might be rejected if the wrong file or file type was uploaded, but might also be used if coursework is well below the standard expected. Students could also attach comments on uploaded files for staff to read, examples of where this is useful include providing a note on problems they encountered, any special instructions on how to compile a program, etc.)
In comparison to TAGS, OATS introduced the automatic application of lateness penalties in cases where work was submitted after the due date of an assignment, although it was limited to the lateness policy in use for the School of Computer Science. Assignment due and/or acknowledgement dates could be overridden on a per-student basis, for example where illness required a student was given an extension or even exemption from an assignment. Following the MMS design principle of actively reaching out to users rather than passively waiting for them in case of problems, OATS could send e-mails to remind students and staff of work to be done. In the case of students this was limited to coursework due, and was only triggered after the deadline was missed, as it was felt that the system should not act as a substitute for organisational skills expected of students. For staff there were options to receive e-mail on students uploading work, submission deadlines being missed, and/or acknowledgement deadlines being missed.
Graded Online Assessment Tracking System
The Graded Online Assessment Tracking System (GOATS) was developed in response to changing requirements in terms of coursework marking processes. Screenshots of this re- vised tool are shown in figure 7.7. This revised tool incorporated a number of improvements
and functionality changes:
General Settings
Lateness Penalty* Default Due Time 23 59 Email Options Date to stop emails
Student Tutor Administrator E-mail on upload & Must Acknowledge Alert if not uploaded by Due date Alert if not accepted by Acknowledge by date
1 January
Purpose Default student see grades
practicals true
Anonymous marking? Default Acknowledge delay
false 7 days
* Please Note: The lateness penalty is computed as percentage deduction per hour late. Thus the default of 1 would result in a deduction of 1% of the total awarded grade for every hour late the work was submitted.
Slot Name Short Name Due date Acknowledge by dateSlot weighting Style Delete
... Single Upload ... Single Upload ... Single Upload ... Single Upload ... Single Upload ... Single Upload ... Single Upload ... Single Upload ... Single Upload ... Single Upload ... Single Upload ... Single Upload ... Single Upload
UpdateDelete Selected Slots Add new row 1 1970 Week 01 Practical W01 05/10/2007 23:59 19/10/2007 23:59 0 Week 02 Practical W02 12/10/2007 23:59 26/10/2007 23:59 1 Week 03 Practical W03 19/10/2007 23:59 02/11/2007 23:59 1 Week 04 Practical W04 26/10/2007 23:59 09/11/2007 23:59 1 Week 05 Practical W05 02/11/2007 23:59 16/11/2007 23:59 1 Week 06 Practical W06 09/11/2007 23:59 23/11/2007 23:59 1 Week 08 Practical W08 23/11/2007 23:59 07/12/2007 23:59 1 Week 09 Practical W09 30/11/2007 23:59 14/12/2007 23:59 1 Week 10 Practical W10 07/12/2007 23:59 21/12/2007 23:59 1 Week 11 Practical W11 14/12/2007 23:59 21/12/2007 23:59 1 Week 12 Practical W12 21/12/2007 23:59 11/01/2008 23:59 1 Class Test w6 CT1 22/12/2007 23:59 29/12/2007 23:59 0 Class Test w12 CT2 23/12/2007 23:59 29/12/2007 23:59 0 2007/8-S1 CS1002 (Computer Science) CS1002 Coursework
CS1002 - Uploads Configuration OverviewUploads EventsModulesLogout ConfigurationAlterationsDefine
2007/8-S1 CS1002 (Computer Science) CS1002 Coursework
CS1002 - Uploads Configuration OverviewUploads EventsModulesLogout ConfigurationAlterationsDefine
Figure 7.7: Screenshots of Coursework Management (GOATS)
• Marks were initially limited to the standardised St Andrews grading scale. This change was made in reaction to changes to marking policy across the institution, which meant that all work was expected to be marked against this revised grade scale. Later this would be revised to support other marking scales (see section 5.11.3).
• An API for extracting marks from a tool in an implementation agnostic manner had been added to the MMS core framework, and the GOATS tool implemented this API. This allowed other tools, such as the final grade calculator, to access marks in the coursework tool without requiring an understanding of its internal structure.
• Ad-hoc assignment functionality was removed in order to simplify the user inter- face. Ad-hoc assignments were rarely used correctly, had no strong requirement from users, and frequently caused additional work due to misuse.
• Staff and students could now attach uploaded files to feedback/comments on course- work. For staff this was intended primarily to allow the return of coursework with comments annotated on the work itself, whereas for students this assisted with the replacement of the ad-hoc assignments by allowing them to attach supporting files directly to the submitted work.
• Support was added for anonymous marking of coursework (see section 7.6.4 for fur- ther discussion on anonymous marking policies).
These changing requirements in terms of marking schemes later lead to development of the mark abstraction layer to manage the complexity of tracking and manipulating multiple different marking schemes. Ad-hoc assignment functionality was partially replaced by the file sharing tool, which was extended to provide private file space on a per-student basis.
Coversheet Generation
One of the frequently raised issues faced by academic staff with electronic coursework was the amount of time spent printing out coursework, and the difficulties in tracking course- work where coversheets were missing or incorrect. Coversheets are needed to ensure es- says are associated with the correct student when marks are being recorded and feedback returned.
A combined solution for managing coursework coversheets and printing of coursework was proposed with the intent of reducing the number of independent documents that needed to be printed, and modifying those documents at the same time. The first version of this functionality required students to submit work in PDF format, but could then collate all documents within a single assignment into a single very large document, inserting cov- ersheets between them to identify which piece of work belonged to which student. The resulting document could then be printed in a single batch by secretarial staff, stapled and passed along to markers.
A later version attempted to do the same for Office Open XML1 formats (commonly re-
ferred to as the “Office 2007 formats”, referring to the first version of Microsoft Office that made use of them). Although partly successful, the complexity of the Office Open XML document format made this substantially trickier and less reliable than for PDF documents.
Essay Conversion to eBook
Another approach considered was to eliminate the printing step entirely. The rationale most frequently given for printing coursework instead of marking on a computer was that reading on a monitor is harder than from paper, and this is generally supported by the
1http://www.ecma-international.org/publications/standards/Ecma-376.htm
literature (O’Hara[96] and Shaw[119] are recommended articles on reading on screens vs. paper, especially in context of marking work).
Given that electronic paper devices such as Amazon Kindle2, Sony e-Readers3, etc. are marketed on the basis of being easier to read than a CRT, LCD or other computer display, it was hoped those devices might be an acceptable compromise for academics – not requiring printing, while being easier to mark than onscreen.
However, this required either that students submitted work in a compatible format (ePub4,
Mobipocket5, or similar), or that documents were converted to such a format. Note that Mobipocket is the standard format for the Amazon Kindle e-reader, and ePub is used for most other e-readers. For a small-scale test, it made more sense to look at conversion of documents initially, rather than disrupting student assessments. The ePub format was chosen for this test as it was simpler to produce documents in from a server application – all supported methods of generation Mobipocket files use an external application to package the content[82].
The coversheet and document merging functionality developed provided much of the core of a document converter, allowing essays to be re-written into HTML while maintaining basic formatting (bold, italic and underline) for the first version. Once a suitable HTML document was generated from the source, packaging it with metadata to produce an ePub format e-book was relatively straightforward.
Tests with the conversion produced acceptable results – complex documents could cause problems when being converted to HTML, or lost too much formatting for the result to make sense, however most essays tested were simple enough for the result to be clear (if rather unimaginatively formatted).
A small number of academics were shown these essays or given the ability to use the conversion functionality in MMS themselves. Feedback was disappointing, most found the premise interesting, but display size, lack of ability to mark up work, and time spent managing documents (to transfer to e-readers) were all significant complaints. Potential future developments in this area are discussed in 12.5.2.
2https://kindle.amazon.com/
3http://www.sony.co.uk/product/rd-reader-ebook/tab/overview
4http://idpf.org/epub
5