Eective strategies for online testing in large classes

Eective strategies for online testing in large classes

School of Mathematics and Statistics University of New South Wales, Australia

UNSW Australia and me

University of New South Wales Sydney Australia

54000 students

Large Science and Engineering Faculties 25% international students

UNSW in top 8 of 35 Australian Universities

School of Mathematics and Statistics teaches approximately 3000 rst year students

Jonathan Kress

Senior lecturer in the School of Mathematics and Statistics At UNSW since 2001

Courses using Maple TA at UNSW

School of Mathematics and Statistics

MATH1131/1141 Science and Engineering (1700)

assumed knowledge / class test preparation / online Maple / invigilated Maple lab

MATH1151 Actuarial Stuides (200) class test preparation / online Matlab MATH1011 Lower level rst year course (100)

weekly tutorial preparation, online Maple MATH1031 Maths for Life Sciences (400)

class test preparation, Maple MATH1041 Stats for Life Sciences (400)

online quizzes, invigilated R or Excel lab

Higher Year Stats/Numerical Methods/Engineering Maths (2000) intro to matlab, various online tests

Faculty of Engineering

Higher year undergraduate and Masters Engineering (1000) mostly longer assignments

Why use online testing and Maple TA?

Regular testing encourages students to keep up to date Provides immediate feedback, anytime, anywhere Reduces labour on routine marking

(What should be done with that saving?) Allows repeated attempts according to student need Personalisation of learning

encourages collaboration that isn't just copying

Analysis of student responses can be used to improve teaching Very exible algorithmic question generation and grading It lacks features we don't want during invigilated tests

eg, messaging or access to course materials Plan to use adaptive questions in the future

Dierent scenarios for use of Maple TA

Formative assessment for learning Assumed knowledge quiz (placement test)

for students in MATH1131 who should be in MATH1011 Class test preparation

in the week preceding a written class test Weekly tutorial preparation

a short online test preceding each week's tutorial Progressive (mastery) tests

must pass each test to access the next test Practice followed by actual test

unlimited practice but limited number of attempts count Practice tests

no marks awarded, feedback only Summative assessment of learning

Invigilated lab tests

How students view Maple TA

Students expect to be rewarded with marks.

Students expect things to work without having to read instructions. Student confusion about the naming of Maple TA and Maple lead to an increase in student use of Maple!

Student comments (2013 and 2014)

Feedback from quizzes was very helpful. Constant tests are good.

Online tests are mostly just annoying and take up valuable time.

(Course could be improved by) more online questions. . .

The online tests and Maple tests instead of assignments were the best features for this course.

The online tests helps in getting a better grip over the subject. (Best features of the course) on-line tests

I found the online tests provided a good revision tool as they provided immediate feedback which I could use to improve.

Should have more smaller online tests on maple ta so that it forces students to learn and be up to date with the course as it moves quickly. Regular online and class tests prompt us to be continually up to date. . . The online quizzes kind of force you to learn how to do each question, and since you have 5 tries you can work on a question until you get it right. More weighting on online tests. The time used to complete the online tasks was not appropriate in comparison to the percentage it makes up. Online quizzes were well set out. It was useful to have several attempts.

Engagement

Do students engage with tests?

In MATH1131 online tests are worth approximately 1% each. Approximately 90% of students attempt each test. In MATH2019 online tests do not count towards nal grade.

Between 1% and 10% of students attempt the tests. Incentives

Marks count towards nal grade

0.5% per 30 minute online test seems sucient! Immediate feedback

Flexibility

allow some tests to be missed or failed eg only count best 8 of 12

allow repeated attempts only count best mark allow plenty of time

Set and forget

Our main current use of Maple TA is to provide practice questions and feedback to students with minimal intervention.

Provide written and screen capture video instructions. Link to Maple TA provided next to link to instructions.

Maple TA class homepage tells students to read the instructions. Compulsory Using Maple TA test must be completed rst.

Set and forget

Allow multiple attempts.

Revision only versions of tests available after deadline.

Course outline states no extenstions or additional attempts allowed. Only best attempt counts.

Allow some tests to be missed without penalty. Questions contain embedded instructions.

Tutorial preparation tests in MATH1011

Encourage regular preparation for tutorials.

MATH1011 has separate algebra and calculus components.

Weekly tutorial preparation tests for the calculus introduced in 2004.

Amnesty for missed tests in week 5 of semester. Course survey comments requested similar tests for algebra.

Evidence of increase in average score for nal exam questions related to online tutorial prep questions.

Class test preparation

Provide feedback to help students prepare for class tests.

Initially allowed 3 attempts but spent too much time dealing with special pleadings for extra attempts so increased to 5 attempts.

Invigilated tests

Maple component of MATH1131 is assessed with invigilated lab test students book time for test on Moodle

student ID cards are checked

computers (112) have exam mode with no internet access Before 2010

paper based test

student Maple worksheets collected, printed then marked by hand and returned in tutorial class

very labour intensive

students had no feedback prior to test Since 2010

Maple TA test

test is available for unlimited practice on Maple TA before lab test test restricted to lab IP range

easy immediate grading once all tests have nished Since 2014

MATH1041 Stats for Life Sciences has an invigilated lab test partly graded by Maple TA.

Allowing for special cases

Although we say no extenstions or additional attempts will be granted, in reality there will be special cases.

We use revision only versions for special cases.

Revision only tests also depend on Declaration and Using Maple TA tests.

Revision only test become available as soon as `for marks' deadline has passed.

Summary

Expect bugs in your questions.

Expect students to do uncontrolled tests collaboratively. Students won't attempt optional tests.

Assume students won't read the instructions.

Assume students will start the test when they are not prepared. But if you just want students to learn, the solution to all of this is. . .

don't worry!

Oer enough marks that the students care but you don't. Provide lots of ways for lost marks to be recovered. Have a strict non-intervention policy.

Summary

Expect bugs in your questions.

Expect students to do uncontrolled tests collaboratively. Students won't attempt optional tests.

Assume students won't read the instructions.

Assume students will start the test when they are not prepared. But if you just want students to learn, the solution to all of this is. . . don't worry!

Oer enough marks that the students care but you don't. Provide lots of ways for lost marks to be recovered. Have a strict non-intervention policy.