Potential Causes of Frustration

As seen above, one cause of student frustration was a lack of agency in how often to repeat practice. Although repetition is inherent in mastery learning, students also cited that the versions were distinct from each other, which amplified frustration. The versions were designed to call on the same principles and we believed they were isomorphic, but students performed differently on versions within the level, revealing differences to students we had not anticipated. The mismatch in difficulty between versions was also cited as a deterrent for studying solution videos; one student compared the experience to being a mouse in a maze, where they were taught to solve it and then put back at the beginning of an entirely different maze. Many students did not believe that the solution video to one version’s questions would help them navigate the next version’s.

Because students were randomly assigned to their first version within a level, data allowed us to compare their scores on the first attempt as a proxy for the difficulty of versions and questions within each level. A severe example of mismatched levels is shown in Figure 5.11. This level required students to synthesize skills from the previous levels to solve a Newton’s Second Law problem, but Versions 1 and 3 involved rotated coordinate axes, a problem on a ramp, while Versions 2 and 4 used traditional coordinate axes. The stark difference in performance on these versions gave us information about student difficulties that we had not anticipated.

Considering the mismatch of versions over the entire semester, a threshold was set for 20% performance difference to constitute a level as being mismatched in versions. This could either be overall mastery rate (one version had at least a 20% difference in mastery rate than another version) or identified by having at

Figure 5.11: Students’ scores by question and mastery rate for their version on their first attempt, which can be used as a proxy for version difficulty.

Figure 5.12: An example of the setup for a two-body Newton’s Second Law level. Students were asked to correctly identify equations for each block to eventually solve for their acceleration.

least two questions which had a 20% difference in performance between versions. Some levels had smaller differences between questions which cumulated to a systematic difference over the whole set. Others had specific questions that were dramatically different, but a floor effect in the mastery rate made identifying them by mastery rate alone ineffective. In the level shown in Figure 5.11, both conditions were met; there were specific questions showing large differences which translated to a difference in the overall mastery rate. Identifying levels which satisfied either one or both of the conditions, 20 of the 32 levels were mismatched by our definition. Their distribution throughout the semester was consistent, with fluctuations but no general trend; each week had at least one mismatched level.

The data collected from the semester allowed us to identify skills which students struggled with that were taken for granted in the creation of the content. In some cases, these were related to math skills, such as the example discussed with the mismatched versions. We expected students to be proficient in rotating vectors to new coordinates, but the students needed support. Other examples of math difficulties were also revealed through the data. Another notable example was students’ performance on a two-body Newton’s Second Law level, one which did not have mismatched versions. An example of one of the version’s situation is shown in Figure 5.12; students were asked in questions 1-4 to (1) identify the correct equation of motion for M1, (2)

identify the correct equation of motion for M2, (3) specify that the accelerations were the same magnitude

but in different directions (a1= −a2), as specified by the coordinates, and (4) specify that the tensions on

each block were the same magnitude. Of the students who could correctly answer questions 1-4, only 47% could correctly solve for acceleration. Despite having the correct equations and relationships necessary to solve the problem, students struggled to perform the algebra to solve the system of equations. Most of the content in the course was not in support of math competencies, but relied on students’ ability to complete math tasks.

5.4

Discussion

The primary takeaway from the first year of implementing mastery-style homework was that if we want the students to use the materials productively, affect needed to be addressed directly. Because the results from our clinical trial had no such issues, we neglected to consider students’ emotional response to the system while dealing with it for an entire semester. Many students were not using the homework as intended, but rather circumventing the learning portions (sincere attempts at the problems and the solutions) in order to get their points as quickly as possible. Though this behavior is not completely unexpected for first-semester college students, the scale and impact of student frustration and desperation was greater than anticipated. As the term progressed, these issues compounded; students became frustrated, used the system in ways that did not help them learn, and then felt more frustrated when they could not demonstrate learning. To address this, we needed to look for ways to improve student attitudes about mastery-style home-work to encourage them to use the system in a useful way.

A potential cause of frustration was mismatch among versions of the same level, as far as content and difficulty. Wording or coincidence among multiple-choice answers sometimes made specific questions harder or easier than in other versions, which led to either frustration by passing students before they were prepared for the following level or frustration because the practice and solution videos did not prepare them for an unusually difficult problem on their next version. Further, scenarios that we considered equivalent had vast performance differences. One reason that students cited for their lack of interest in the solution videos was that they felt that each new version was unrelated to the last, and thus had little motivation to review missed questions. Literature has shown that novices and experts tend to classify problems differently; experts are more likely to classify problems as similar based on underlying principles whereas novices tend to focus on surface features [108]. This may explain some of the students’ belief that versions were not equivalent, but their different performance rates on versions suggests that as “experts,” we underestimated how details affected the complexity and skills necessary to solve problems in different scenarios.

Students’ response to the mastery homework may have been informed by their goal orientation, as described by Dweck [34]; mastery is useful for students with learning goals orientation, who are motivated primarily by wanting to gain competence in skills. The language and behaviors by students suggest that they were primarily concerned with desperation for points and frustration with repetition, more often associated with performance goal orientations. Especially because the scope of the audience of Physics 100 extends to all of the engineering college, it is likely that there are students in the course who do not have an explicit interest in learning physics, but are taking the course to fulfill that requirement. The benefits of mastery would then be diluted for students approaching the exercises with performance rather than learning as their motivation.

Moving forward, it was clear that affect needed to also be addressed directly. General student attitudes towards the homework were resentful, which may have also been due to students’perceived lack of agency. The language used by students portrayed the situation as something “done to them” as opposed to something “for them,” which did not promote students to take ownership of their learning. Rather than a tool, the homework was seen as an obstacle to overcome by any means. Our main objective for the following year was to reduce frustration and shift student attitudes to improve student morale and increase productive behavior.