Lecturer questionnaires

A questionnaire (Appendix C, p. 167) was circulated to the nine lecturers that were using the CAA system with their first year students with the purpose to establish how the lecturers use the system, why they use the system and to explore some of the issues that lecturers may have experienced when using the system.

The responses from the questionnaires demonstrated that there was a diverse range of practices adopted by the lecturers (Table 5.7), with four of the nine adopting somewhat unique practices. All the lecturers granted their students access to the practice tests, but it was in the summative test practice that lecturers differed most. One lecturer did not offer a summative test, but allowed access to the CAA practice tests so that students could test their knowledge without being judged on it directly.

One lecturer replaced the CAA summative test with an invigilated paper test. The other lecturers differed in terms of whether the CAA summative test was invigilated or not: three invigilated the test and four did not.

The lecturers were asked about their stances on collaboration for CAA testing. For practice test phases, all lecturers were happy for students to collaborate, but only three

CAA summative test @c@Invigilated

paper test No test Invigilated Non-invigilated Access to practice test granted more than one week before a test Access to practice test granted after the test 0 2 1 1 Access to practice test granted less than one week before a test Access to practice test granted after the test 0 1 0 0 Access to practice test not granted after the test 3 1 0 0 Total 3 4 1 1

Table 5.7: Lecturer practices for the use of CAA

lecturers explicitly encouraged students to collaborate. Four lecturers prohibited collab- oration in the summative test by invigilating the test session; the four lecturers that did not invigilate the summative test were against the idea of students collaborating; nonetheless, they did not communicate their wishes for students to work alone to their cohorts.

Table 5.8 indicated that they tended to agree that using CAA freed time, was convenient, provided opportunities and motivation for students to practise and offered immediate feedback; however the quality of the feedback offered to students appeared to be less of a motivator to provide these CAA tests to students. Furthermore, the lecturers disagreed on the ease of setting up these tests.

Some row totals are not equal to nine in Table 5.8: some lecturers opted not to respond to all the questions.

When asked about the bank of questions that are contained within the CAA system, there was no consensus as to whether the questions posed sufficient challenge to the

I use CAA with students because. . .

Strongly disagree

Disagree Neither agree nor disagree

Agree Strongly agree it is easy to set CAA tests

for my students

1 1 0 4 2

it was used by a previous lecturer

0 0 0 4 4

I am encouraged to by the department

0 0 6 1 1

students receive immediate feedback

0 0 2 4 4

students receive good quality feedback

0 2 3 3 1

CAA frees up time 0 0 0 5 4

CAA is convenient 0 0 0 5 3

CAA provides students with opportunities to practise

0 0 1 4 4

CAA provides students with motivation to practise

0 1 2 5 0

Table 5.8: Lecturers’ reasons for using CAA

students (two believed they do, five did not, two neither agreed nor disagreed). How- ever, the lecturers felt that developing questions took too much time (four agreed, five neither agreed nor disagreed) and was too difficult (three agreed, six neither agreed nor disagreed). Only three had attempted to develop questions for the CAA system and only two thought it would be worthwhile doing so. The remaining lecturers had not attempted to write new questions and did not believe it would be worthwhile.

This offered an early glimpse into the contradictions that lecturers faced while using this system. The questions that the CAA system displayed to students were not sufficiently challenging, yet there was no consensus between the lecturers on whether it would be worthwhile to develop more questions.

In a later group of questions in the questionnaire, the lecturers offered some insight as to why the questions might not have offered the challenge that they desired. They were asked the extent to which they agreed that the CAA system was able to test students on their recall, procedural and conceptual ability.

They were most convinced by the statement that the CAA system was able to test their students’ abilities to carry out mathematical procedures and methods (eight agreed, one neither agreed nor disagreed). Five lecturers believed that the CAA questions tested

students’ recall; three disagreed. One lecturer felt that the system tested students’ con- ceptual knowledge, but three disagreed. The lack of agreement may be due to the dif- ferences between the mathematics and engineering questions, or due to diverse practices that the lecturers adopted.

An invigilated test may well be a test of recall if no additional materials are permitted — indeed, the lecturers that strongly agreed that the CAA summative tests were a test of recall also invigilated their tests — whereas a non-invigilated test may have invited students to use their own materials and thus negated the need for recall.

Both banks of questions — for the mathematicians and for the engineers — were abun- dant with questions aimed at testing the students’ ability to carry out procedures, yet only one lecturer believed that these questions posed sufficient challenge to the students. There was some desire to develop more questions for the system to address the lack of challenge. These data suggest that perhaps this is because these lecturers wished to add the ability to test conceptual ability more routinely in the CAA tests, and at that time the CAA questions could not provide this challenge.

If this is true, the contradiction that lecturers faced was between the desire to test students’ conceptual knowledge, with the same efficiency and immediacy provided by the CAA tests, and the difficulty in setting new questions on the system. In order to resolve this contradiction, the lecturers would have to find a means of testing students conceptual knowledge.

The questionnaire data provided two means by which some of the lecturers have at- tempted to achieve this. One lecturer used invigilated paper tests to introduce a con- ceptual dimension to the assessment. Three lecturers had attempted to develop new questions for the system, despite the difficulty and time cost of doing so.

The conclusions from the questionnaire allude to the existence of contradictions within an established practice of using CAA with students. The lecturers reported that the system is convenient and efficient, and provided students with the means and motivation to practice. However, the assessment itself was insufficient to test the breadth of ability that the lecturers wished to test of their students and there are indications within the data of how the lecturers have sought to overcome this problem.