Use and Effect of Practice Exams

In the same vein as retakes, we were interested to see how the practice exams were affecting students’ exam scores. The practice exams were delivered to students differently in 2018 and 2019, the change inspired by our results in 2018.

In 2018, when the practice exams were given as optional immediate feedback problems (and one version sup- plied with narrated animated solutions as help, alongside the problems), we saw that students’ engagement with the online practice exams had no correlation to their performance on the actual exam. There were three versions of practice exams online, corresponding to about 80 questions combined. Figure 10.7 shows the dis- tribution of students’ scores on the exam based on the number of practice problems they completed. Even students finishing nearly every online practice exam still had exam scores ranging down below 40%. On the practice exams in 2018, students’ performance was not a good predictor of their actual exam perfor-

Figure 10.7: A scatterplot of students’ hour exam 1 scores compared to the number of practice exam questions they completed.

Figure 10.8: Students’ hour exam 1 scores compared to their fraction of correct practice problems in 2018, for students who used at least 20 practice problems.

mance. Figure 10.8 shows students’ exam scores compared to their practice exam scores for students who attempted at least 20 practice problems (nearly one practice exam). About a third of students performed worse on the actual exam than the practice. From looking at students’ logs from the practice exams, many students submitted multiple answers within seconds of each other, suggesting that they were not using the practice exams conscientiously.

The delivery method of the practice exams was changed for Spring 2019 in Physics 211; rather than supplying solutions beside the problems, they were given to students after submission, and they were only allowed a single submission for each problem. By changing the delivery format of the practice exams, they became much better predictors of students’ final exam scores, shown in Figure 10.9. Additionally, students improved over the course of their study with the practice exams. Recalling that a single practice exam is around 25 questions, Figure 10.10 shows students’ scores on their first 24 questions (the first practice exam) compared to their score on the rest of their practice exam problems. Students who attempted at least two practice exams saw an improvement of 8% from their first practice exam to the rest of their practice.

Figure 10.9: Students’ hour exam 1 scores compared to their fraction of correct practice problems in 2019, for students who used at least 50 practice problems. Orange points correspond to students scoring 2 standard deviations lower on the exam than their practice. Red points correspond to scoring 3 standard deviations lower.

Figure 10.10: Scatter plot of student scores on their first practice exam (24 questions) compared to later practice exams, for students in 2019 who submitted answers for at least 50 questions.

10.4

Discussion

Both retake exams and practice exams can be components of frequent testing, but we saw that practice exams had a larger impact on student performance. On the retake exams for students in Physics 211, we saw that increasing the stakes of the retakes from Physics 100 (no risk on quiz retakes) to Physics 211 (exam retakes can reduce score) affected the fraction of students using retakes, especially reducing the number of high performing students using the retakes. The retakes in both courses were intended for students who scored poorly on their first test, so encouraging students to take retakes intentionally rather than by default was a good result. One also hopes that making an intentional decision to use the retake and the additional stakes would encourage students’ study between the original and retake exams; students scored an average of 10% higher on their retake exams compared to 7% in Physics 100 quiz retakes. As in Physics 100, retakes helped lower scoring students approach their classmates’ scores, but it is unclear whether these improvements are primarily corrections from statistical fluctuations via a regression toward the mean; it is possible that the result is largely from students who performed below their ability on the first exam and were given an opportunity to show their expertise on the retake.

From their first exam to their second exam in Physics 211, students who used retakes scored 11% higher despite the average exam scores for Exam 1 and Exam 2 being comparable. However, students using retake exams tended to be lower-performing students and we saw that the lowest performing students in 2017 (when there were no retakes) also saw about a 10% increase from Exam 1 to Exam 2. This could be attributed to the frequent testing model; students who had already done poorly on one exam could be more likely to change their behavior for the next exam, and self-selection could be driving the result. Alternately, this improvement could also be a statistical regression toward the mean. The lowest performing students on the first exam saw no significant difference on their final course grades in the years with and without the retake exams, so we believe that the retakes were successful at smoothing out fluctuations but not necessarily successful at improving student learning.

The practice exams, on the other hand, showed student learning improvement and better tracked with students’ exam grades, when they were delivered in a format to simulate actual exams. In 2018, when the practice exams were given alongside help videos and allowed students to submit multiple answers, students’ participation and performance on the practice exams were not correlated with their actual exam scores. Giving multiple attempts at problems may be sending an incorrect message; students who are able to solve problems with help videos and multiple attempts may falsely think they are able to recreate the problems on the exam in a single attempt with no notes or help. This could be explained by “the illusion of understanding,” where students reading or watching worked examples interpret familiarity as competence, but cannot recreate problem solutions that they believe they understand [65, 165–167]. Schroeder saw that students who self-reported full confidence in understanding a worked example still were unable to successfully complete a the exact same problem within the same class session [65]. Chi et al cite that the illusion of understanding can be tempered if students who are asked to actively use an explanation to do another task [168], but the multiple choice and multiple attempt nature of the practice exams did not require students to demonstrate their competence after using help videos.

When practice exams were delivered in 2019, forcing students to only submit one answer and withholding solutions until they had made an attempt may have given students a better sense of their competency with the material and required them to engage more actively with the problems. Their performance on these

practice exams was correlated better with their actual exam scores, and students saw improvement from their first practice exam to their next. The practice exams were still available alongside notes and had no time limit, making them more lenient than actual exams, but a better formative assessment than in the previous year.

When considering frequent testing, the results from our retakes and practice exams in Physics 211 suggest that practice exams, when implemented as close to testing conditions as possible, were better for student learning. In particular, grading the practice exams more like an exam gave valuable formative assessment to students. Rather than inflating their sense of competency, such as in 2018 when students performed much better on the practice exams than the actual exams, in 2019 students were able to see improvement in their practice exams and have that movement reflected in their exam scores. In Physics 100, students requested pretests for their bi-weekly quizzes, and this study provides evidence that their addition is valuable.