Under what conditions is the CAA system an effective assessment?

There was also a desire for assessment to be objectively fair and consistent, which was also a requirement from the department and the university. Students also desired this fairness and consistency; they wished to be credited fairly for what they knew and understood.

5.3.2 Under what conditions is the CAA system an effective assess- ment?

The CAA system was effective for all the students in the interview study to some extent. All the students in this study were aiming to achieve a particular mark in the assessment and, for the most part, the practice tests enabled them to achieve these aims. This made the practice tests an intrinsic component of the assessment; from the interview data, it would seem that the CAA system might not have been as effective had it consisted solely of the summative tests.

However, the scope for the system to be effective was limited by the students’ inability to choose more challenging learning goals after the summative test. The summative test was regarded as the end of learning, after which they could gain no further credit for learning.

As for the students, the CAA system was effective for the lecturers in the interview study to some extent. It offered regular assessments to students, who could take responsibility for managing their own time to complete the assessments, and gain immediate feedback from which they could improve.

It also achieved the lecturers’ aims to encourage students to use the lecture content on problems between the lectures. Indeed, the students reported in the questionnaires and in the interviews that they would commit substantial time and effort to ensuring that they could carry out the procedures that were asked of them for the summative test. The tests were also regarded by students and lecturers as being objectively fair. While randomisation of the questions minimised the possibility that two students would face exactly the same test, the questions were similar enough between tests to ensure there were no differences in difficulty. Students also trusted the CAA system to be accurate in its marking, though the lecturers had reported instances when the solutions stored in the CAA system were incorrect.

Both lecturers and students were broadly satisfied with the feedback, insomuch that the students could identify where they had gone wrong and improved their approach to

the question. However, it was a concern for the lecturers that the students were too dependent on the feedback, and some had learned to mimic the methods given in the solutions. It was also reported in an earlier focus group that some students believed that it was possible to pass the test without attending the lectures (Broughton et al., 2012).

5.3.3 What would make the CAA system more effective as an assess- ment tool?

Identifying where the assessment ceases to be effective offers the point at which the assessment could be made more effective. For the students, the limitations come largely after the summative assessment. This occurs because students are unwilling or unable to acquire new learning goals.

The lecturers also feel that the CAA summative test results are not necessarily good measures of a student’s understanding or learning. That is, the goals that students set for themselves in terms of marks do not reflect the depth of understanding that lecturers desire to see in their students.

These indicate that there is a need for students to acquire learning goals after the CAA summative test, or for the CAA system to be able to assess students to a depth of understanding that is not currently available.

In Eta’s case, he had goals beyond what the CAA tests could offer him. He desired more conceptual understanding and wished the CAA tests could guide him towards his goals. It is not just the CAA system that prohibited Eta to achieve his learning goals: he also found that his other learning commitments and the lack of time made pursuing his goals impossible. Consequently, the effectiveness of the CAA system ceased once he had completed the summative test.

Both lecturers and students found that time was a barrier towards extending the scope of effectiveness for the CAA system. The students were under pressure to complete assignments and were managing their time between different modules. Lecturers found it hard to find time to make improvements to the CAA system: Nu had attempted to write more questions, but found this process was not straightforward and yielded little benefit.

The lecturers tended to use the CAA system as an exercise for students to complete to supplement their learning. The marks they were allocated were sufficient enough to encourage students to use the tests, but not substantial enough to make a significant contribution to their overall module grades. For the mathematics lecturers, this meant that cheating was not unduly concerning; for the engineering lecturers, there remained

a desire for the marks to indicate where students were struggling. The engineering lecturers were keen to prevent students from collaborating the summative test, and so used invigilation of the summative tests to prevent cheating.

In order to address these contradictions between their aims and what the CAA system offered, the lecturers adopted diverse practices. Some used alternative forms of assess- ment to assess students on their conceptual understanding; one lecturer used summative tests conducted on paper instead of online to introduce real-life contexts and additional challenge to students.

While these diverse practices extended the effectiveness of the lecturers’ assessment regime, this diversity had an impact on the effectiveness of the CAA system for stu- dents. Theta found that the change in practice affected her ability to pass the tests. It would seem that Theta had become dependent on the context of the CAA system ques- tions, which resulted in her becoming confused when real-life contexts were introduced in the second semester.

Theta’s scenario showed that there was a problem with the contextual dependency on the CAA system, and that some students could not apply their knowledge to a context that was not familiar to them. For the lecturers that conducted the summative test in this way, it helped to identify these shortcomings in the students’ awareness and to differentiate between students’ levels of understanding. However, Theta merely wished to pass the test and neglected to return to the practice tests after failing the summative test; the CAA system ceased being effective for her at this point.

This case, too, exemplifies the belief that a summative test represents the conclusion of learning. The students rarely revisited the practice tests after the summative test, though some did so to revise for their exams. The lack of a tangible reward for continuing learning beyond the summative test is pervasive and was a significant limiting factor to the scope of effectiveness for the CAA system.

It highlights a need for a shift in the community towards a cultural reward for continued study. Although the marks awarded cannot change after a summative test, there may be other rewards to be had — for example, it may be encouraged as a means to prepare for a future course, or for a side project, or for revision for exams. The interview study with students suggested that there needs to be an ulterior motive for students to invest time in further study: even when students are already engaged and interested in pursuing deeper understanding.

This could also be achieved by ensuring that the CAA system offers the depth of challenge that lecturers desire. However, while Nu had attempted to write new questions for the system, he did not find it to be worthwhile and required significant time investment. The

lecturers, as a whole, were not convinced that the CAA system could assess students to level of conceptual understanding that they desire. This led to some lecturers using an alternative assessment tool, used alongside the CAA tests, and some to replace aspects of the CAA system with a paper analogue in which the questions could probe a student’s conceptual understanding.

While these responses from the lecturers mitigated against the limiting factors of the CAA system’s scope of effectiveness, they do not address the limiting factors themselves. The students’ reticence towards setting new goals — particularly towards those aimed at their conceptual understanding — could be addressed by setting questions that aim to test their understanding to this level. Nu had tried to achieve this, and found that the process was time-consuming: investment in developing questions that tackle conceptual understanding could address this problem.

Lambda had replaced the summative CAA test with a paper test, since he felt that students were becoming dependent upon the context of the CAA questions. This was observed in another module, where Theta failed tests in her second semester in a similar paper-based test. This somewhat legitimises Lambda’s concerns about students’ context- dependency. This could be addressed by introducing questions that test students’ ability to apply their knowledge to unfamiliar contexts — Lambda was keen to introduce feasible, real-world problems for students to solve using their prior knowledge.

Conclusions

The purpose of this final chapter is to review the findings of the student and lecturer studies in light of what this project intended to achieve. It begins with a discussion of the initial problem in evaluating an existing assessment tool and how the model for effective assessment was derived from the literature and the theoretical framework. Next, the key findings from the studies address the research questions and provide a response to the initial problem. Finally, this chapter discusses how the findings of this research may be used elsewhere.

6.1

How can an assessment tool be evaluated?

The purpose of this research was to evaluate a computer-aided assessment (CAA) system. However, the literature exposed the lack of a rigorous evaluation of mathematics-based computer-aided assessment: either for specific systems or for its use generally. There are many practitioner-based accounts of introducing such systems in institutions or individ- ual courses; many of these reported the positive effects and lacked a critical evaluation, leaving many long-term effects unreported.

The intention of this study was to examine the scope of effectiveness of one CAA system for both student and lecturer users and to provide such an evaluation. An earlier focus group study and informal conversations with lecturers had offered some insights into their practices and some of the issues they had experienced.

Choosing a theoretical framework was important for developing an evaluation framework informed by theory and literature. Cultural-historical activity theory (CHAT) offered a framework that modelled individuals engaged in social activity, motivated by need and mediated by tools. Assessment is a typical human activity in that regard, and assessment

is a culturally-created tool that enables students to prove their learning. Thus the ability to demonstrate that learning has taken place is crucial to the success of an assessment; calling it “effective assessment” defined this characteristic of the learning tool.

Cultural-historical activity informed several important elements of the model for effective assessment. CHAT holds that all human activity is purposeful and that individuals are influenced by their culture and history when forming goals. Therefore, students and lecturers will always have aims when embarking on assessment, and they will form their aims individually with influences from colleagues and further afield. This model offers a measure of an assessment’s effectiveness; a critical narrative of students’ and lecturers’ use of the assessment tool identifies where the assessment has been effective and is no longer effective for those individuals.

It was important to establish the students’ and lecturers’ aims in order to understand why their activity diversified. The model for effective assessment proposed that effectiveness should be measured with respect to the learning aims of students and lecturers; in other words, an assessment would be deemed to be effective while it fulfils the intentions of those using it.

The notion of internalising and externalising information in the learning process arose from Engeström’s (2001) work on expansive learning: students acquire knowledge from external sources and internalise this information; they must then externalise this in- formation to demonstrate their learning. Therefore, the model for effective assessment holds that assessment is an essential part of the learning process. Assessment defines the opportunity to demonstrate that learning has taken place. Without assessment, there is no evidence that learning has happened.

The assessment literature suggested that the process of learning ought to encourage students to become less dependent and more autonomous. CHAT also argues that goal-forming is cyclic: Both Engeström (2001) and Tikhomirov (1988) described goal formation as cyclic: once a goal is reached, the individual should become more aware of what else is achievable. Therefore, it is a condition of the model for effective assessment that, in order for an assessment to remain effective, students must develop autonomy and pursue more challenging goals.

While CHAT emphasises the social nature of activity, the model for effective assessment emphasises that individuals are unique insomuch that their histories and identities are unique, and therefore it follows that their aims and actions are unique. So while an assessment tool may be effective for one lecturer, it might not be effective for another lecturer or for the students. However, deductions can be made from looking at several

case studies and identifying where an assessment has been effective and common causes for it ceasing to be effective.