Student Engagement

Some of the most striking results from the semester spoke to student attitudes and behavior; it is not surprising that first-term students over an entire semester would engage differently than volunteers for a single clinical trial, but the gravity of that difference was unexpected. Most students began the term using the homework as it was designed, but their useful behaviors deteriorated over the semester.

One dramatic behavior that emerged was students’ tendency to “skip” through levels, spending very little time answering the questions. We observed that as the term went on and the difficulty of the levels increased, many students were actually spending less time answering all the questions in a level in a given try. We believe that this behavior arose from the use of one of two nonproductive strategies or some combination. One strategy would be to try to take advantage of the multiple-choice format by simply choosing answers at random to questions that they had no idea how to answer. Statistically, this strategy is very unlikely to be successful, as most levels had around 5 questions and most had at least 5 answer choices. Another strategy was to take advantage of the cyclic nature of the finite number of versions. Since a given level had four unique versions which cycle, students could study the solutions to the first version they were given, then choose answers quickly for next three versions to return and apply what they had learned from the first version’s solutions to the repeated version (albeit with different numbers) on their fifth try. Anecdotally, many students readily admitted to this strategy.

A signature of the second strategy is seen in students’ pass rates as a function of their attempt. Figure 5.5 shows the fraction of students who mastered, according to their attempt number for three levels at different times in the semester. The first and fifth attempts are the same version, and the repeated version is indicated in orange. In the first level of Week 2, students worked diligently through all attempts, as seen by a somewhat steady rate of mastering for each try. As the semester progressed, students’ mastery rates on intervening tries dropped. The last level of Week 3 shows students working earnestly on their first attempt, lower rates of mastery on tries 2-4, and then a bump again at the fifth try. By Week 7, mastery rates were low on all attempts but their fifth.

If material is particularly difficult, the spike in performance at the repeated versions could be natural; students working within the system as it is intended would still have an advantage on a version they had seen previously. By looking at the distribution of time spent by students on their last attempt before the repeated version on these same levels, plotted in Figure 5.6, we likewise see that the number of students spending less than one minute on a version increases as the semester progresses. By Week 7, over half of the students who do not master on their fourth attempt were spending under a minute on a version.

Looking at the average amount of time students spent per try on their second through fourth tries across all levels in the semester, we identified students engaging in these “skipping” strategies as those who spent less than one minute on an attempt (or less than 1/4 of the median pass time, if that is under a minute). The trend over the semester is shown in Figure 5.7. As the semester progressed, the number of students spending very little time with the levels themselves increased to nearly half by the end of the semester. These fractions are not of the students who had not yet mastered, as in previous plots, but include all students who attempted each level.

Despite low mastery rates in many levels, students’ interactions with the solution videos were also less frequent than for students in the clinical trials. Broken down by try, Figure 5.8 shows the average fraction

Figure 5.5: Fraction of students mastering by Try (T1 is Try 1, etc.) for three levels in different weeks in the semester. The orange versions are repeated, and students’ performance is much higher on repeated levels, which is more prominent as the semester progresses. It should be noted that the fraction for each try is the fraction of remaining students, meaning that each subsequent group is smaller and a selectively contains students who struggled on previous attempts.

Figure 5.6: Histogram of time spent on students’ fourth attempt, if they did not master on that attempt. Earlier weeks show more spread in the distribution, while later weeks show large numbers of students spending less than one minute on their attempt of multiple questions. Note that the scale of the vertical axis is different in each plot, with larger numbers in later weeks.

Figure 5.7: Fraction of students spending under one minute, or under 1/4 of the median pass time (whichever is lower), on an attempt for each try, Try 2 through Try 4 (T2-T4) at each level throughout the semester.

Figure 5.8: Fraction of solutions viewed by students in Physics 100 in Fall 2014 compared to students in the second clinical trial. Viewing is defined to be watching at least 20% of the video’s length, and the fraction is out of the total number of incorrect questions. Using this definition, it is possible to get a fraction larger than 1, if students watched videos to correctly answered questions.

of solution videos watched by students in Physics 100 compared to students in the second clinical trial. The threshold used for counting a video as “watched” was set to 20% of the length of the video, and the fraction was out of the number of problems the students answered incorrectly. Both instances had a diminishing fraction of solutions watched as the number of tries increased, but the Physics 100 students began at a much lower fraction, less than half of the rate of the students in the clinical trial. Checking their solution watching over the semester also reveals that productive behaviors declined with time. Using a 10% threshold for watching, Figure 5.9 shows the students’ fraction of solution videos watched week by week.

While we did not expect students in the semester-long course to behave as diligently as students in our clinical trials, we did not anticipate such dramatic differences, nor the decline over the semester. These behaviors can in part be explained by declining student affect, described in the next section.

Figure 5.9: Average solution viewing fraction over the semester. Threshold for viewing was 10% of the solution video’s length, and the fraction is out of the total number of incorrect questions. Using this definition, it is possible to get a fraction larger than 1, if students watched videos to correctly answered questions.