The problems with computer-aided assessment

There are many professional papers that extol the benefits of computer-aided assessment. These papers tend to report the experiences of a lecturer implementing a CAA system. There are few longitudinal studies that examine the long-term implications of the in- stallation and on-going use of CAA, yet there are some reviews of CAA that attempt to collate the issues. The following discussion is a fresh attempt to achieve a similar result. To classify and explain the relationships between issues, this discussion is divided into three categories, as chosen by McKenna and Bull (2000, p. 25): pedagogical, operational and institutional. Although they do not operate as exclusive or distinct categories — they are “clearly inter-related” — there are internal links within categories that provide much scope for discussion.

Pedagogical issues

Perhaps the most pressing issue for CAA is ensuring that it is accepted by lecturers. There appear to be a number of reasons why lecturers would rather not use CAA with their students. Some feel that CAA is part of a process that makes education impersonal: “It is fair to say that a concern of tutors was that automated feedback was another step on the road to an uncertain impersonal future” (Barker & Lilley,2006, p. 51). Likewise, some lecturers prefer to have an active role in the feedback process, as Barker and Lilley explained:

“Tutors liked the ability to be able to ‘keep a finger on the pulse’ when providing feedback. Some concern was expressed that an automated approach would lead to potential problems going unnoticed. This could not happen when tutors themselves gave feedback.” (Barker & Lilley,2006, p. 49)

The design of questions has already been identified as an issue with respect to assessing understanding and knowledge. Writing questions that adequately meet this requirement was a concern raised by Stephens and Curtis:

“Previous experience of question writing was judged to be mixed amongst lecturers, which can result in questions being provided by lecturers that may not always tie in with their course objectives and may not be assessing at the appropriate level of learning.” (Stephens & Curtis,2002, p. 19)

In addition, there are limitations to the scope of what can be tested using CAA at the current time: assessing proof and reasoning skills using CAA is “difficult to envisage” (Sangwin, 2004, p. 7). Nonetheless, McKenna and Bull believed that assessing deeper understanding can be achieved, but indicated that it requires more thought and creativ- ity:

“The capacity of objective tests to assess a wide range of learning is often underestimated. Objective tests are very good at examining recall of facts, knowledge and application of terms, and posing questions that require short text or numerical responses. Additionally, questions, which are constructed imaginatively, can test analysis, synthesis and [evaluation].” (McKenna & Bull,2000, p. 26)

Though objective tests appear transparently fair, Dermo (2009, p. 211) noted that the selection of questions at random from a question bank was not universally well-received — “clearly there is a perception among students that these are unfair” — and ensuring similar questions are of equal difficulty is another burden that question designers must bear.

Some students also feel that, with some CAA marking schemes, they do not receive the marks they deserve. For many systems, marks are awarded only for final answers since students cannot communicate their method to the computer. Consequently, ”many students felt uneasy with numerical input questions, feeling that too many marks were lost if they made slight numerical errors” (Croft et al., 2001, p. 65).

Furthermore, there is an added expectation that students are comfortable using informa- tion and communication technology to complete CAA tests. Although the study did not involve mathematics assessment, a study performed by Biscomb, Devonport, and Lane suggested that this might be a problem: “one of the major disadvantages noted by staff was the issues of students’ levels of IT competence” (Biscomb et al.,2008, p. p.87).

The lack of reward for marking and the assumption that students are capable of using the technology may be regarded as both operational and pedagogic issues. While they are limitations of the operational capabilities of the system, they are also restrictions to the pedagogical decisions that lecturers must make with regard to assessment.

Operational issues

McKenna and Bull (2000, p. 28) viewed operational issues as being barriers to performing the assessment to “rigorous exam conditions”. However, it is not always the intention for assessment to have the same purposes as exam conditions. If CAA is to be used with the intention to provide feedback to the student, then limitations in what the CAA system can provide in terms of feedback becomes an operational issue.

Operational issues may also have more widespread effects. The first issue that McKenna and Bull raised is that of timetabling: that is, CAA tests ought to be timetabled so as to minimise the chance of conferring. As they noted, there are instances where a cohort is too large to be accommodated in a single computer laboratory (Beevers et al., 1999; Conole & Warburton, 2005). In such instances, McKenna and Bull advise testing one group immediately after testing the other in order to ensure no conferring is possible. Regardless whether invigilation is possible, many CAA systems feature a randomisation technique that makes it highly probable that no two students in a cohort will receive identical tests (Gikandi, Davis, & Morrow, 2011; Green et al., 2004; Pollock,2002). By using this technique, students are faced with an obstacle to cheating, although it does not eliminate this cheating completely (Green et al., 2004). The effect of this cheating is unknown — “we really have little idea of the true extent of this problem outside invigilated labs” (Sangwin, 2005, p. 6).

The issue of security was another concern of McKenna and Bull: primarily the storage of the question and answer files, and students’ responses; but also ensuring that students’ identities are checked as they take CAA tests. As for storage, McKenna and Bull (2000, p. 28) reported that, at the University of Luton, question and answer files are removed from the server when they are not in use.

A further point is that technology is not infallible (McKenna & Bull,2000). Ke, Yingwei, Yajun, and Runhua (2010) added:

“Computer equipment has its own limitations, it may not always be available, or be in reliable working order. Additionally, test takers’ mental status, such as scare [fear] and anxiety, has a negative impact on tests. Screen size, and

graphics capabilities could also be limitations in practice of E-assessment.” (Ke et al., 2010, p. 1659)

Some equipment and software issues affect some students more so than others when they occur. Biscomb et al. (2008, p. 87) noted that one of the lecturers “considered that the use of technology might be more problematic for students with special educational needs, with particular reference to dyslexic students”.

This is an aspect of mathematics-based CAA that has been considered by Greenhow (2000), with the use of font, size and colour to aid visually impaired students, and by Sangwin (2005), with the implementation of mathematics rendering that can be used with students that are visually impaired. Though such adjustments have been introduced to aid those with visual impairments, it is not clear from the literature the extent to which other disabilities have been considered.

Time appears to be a limiting factor in the maintenance and updating of a CAA system, particularly when considering the development of new CAA tasks. Designing questions has been identified as a time-consuming process that warrants careful consideration. It appears in reality that lecturers seldom have the necessary time for such an undertaking (Stephens & Curtis, 2002).

Students must also manage their time using CAA around other commitments: having access to the tests at all times can provide some relief to their timetables (Pollock,2002). However, students desire support during assessment and the 24-hour availability of CAA can lead some students to believe that such support is also available all day(Bertolo & Lambert, 2007).

Institutional issues

McKenna and Bull (2000) believed that having adequate staff development to ensure that novice and experienced practitioners could maintain and update the system was important for the ongoing success of the system.

It could be argued that a lack of such training is a missed opportunity for the discussion of good practice. With many options available, lecturers approach CAA “in an individ- ualistic manner. (Stephens & Curtis, 2002, p. 19). This might cause some confusion for students that experience different approaches from different lecturers. McKenna and Bull (2000, p. 27) reinforced this point: “Stephens, Bull, and Wade (1998) have argued for a systematic, fully-embedded implementation of CAA across an institution and sug- gest nine recommendations for its successful introduction”. There is little by means of

the governance of CAA use within higher education institutions; however, this seems no different from the governance of assessment more generally:

“Of the 25-quality assurance staff who returned the CAA Centre question- naire, only one reported having separate exam regulations in place for the governance of CAA and this was specific to the use of OMR in formative coursework, not for summative CAA. This finding is consistent with Yorke’s observation that managing assessment at an institutional level is an under- developed practice in UK higher education.” (Yorke,1998).

(McKenna & Bull, 2000, p. 27)

Stephens et al. (1998) placed great importance on the cohesion of staff members to ensure that CAA becomes a success:

“It is essential to have the support of all staff involved in designing, imple- menting and maintaniing such systems, including academic, technical and support staff. Suspicion surrounding new initiatives, innovations and tech- nology makes it crucial that every effort is made to ensure the smooth running and effective operation of such systems. This can only be achieved through a strategic mandate with general institutional approval.” (Stephens et al.,

1998, p. 285)

Achieving universal support for CAA, however, seems challenging. Some lecturers — and students — might prefer to conduct tests on paper; others feel that CAA does not adequately test students. Consequently, there may be much support for CAA while some remain sceptical:

“The introduction of CAA in the form of objective tests in multiple choice question format has generated much but not universal enthusiasm among students and staff.” (Woodbury, Ratcliffe, & Thomas, 2001, p. 16)