Motivation for Full Study

One might argue that only some students who are performing well express confusion because they are critically engaging with the material; other students may have such a strong grasp of the material that they perform well without any confusion. Similarly, some poorly performing students may not know how little they know, while others are so confused that they perform poorly and can only ask questions. The existence of these different groups within the population could explain the results presented here.

The data analyzed here were previously studied and presented to motivate the notion that expressions of confusion may actually be associated with positive course outcomes (C. Crouch, 2000). However, the results presented here differ from those previously presented; we find that only one of the two questions exhibits a positive relationship between expressions of confusion and performance, whereas the previous findings suggest that both questions exhibit this

relationship. The source of this discrepancy, detailed below, is in the criteria by which students’ responses are sorted. We also describe several limitations to the implications of this analysis, and ultimately motivate subsequent full study.

3.5.1 Discrepancies between Analyses

In the analysis by Crouch and Mazur, students’ expressions of confusion were sorted as “related” or “unrelated” to each of the content-related questions. Expressions of confusion about

60

capillarity and pressure are labeled as “related” to the first question, but expressions of confusion about closely-related topics like contact angles and cohesive forces were not. Moreover, general statements (e.g., everything was confusing) and specific statements (e.g., the question itself was confusing) were also labeled as “unrelated.” Coding according to such criteria necessitates making several assumptions about students’ intent when they make specific statements. In the analysis presented here, we carefully avoid these assumptions using the coding scheme presented in Section 3.3.2. Using these descriptive labels of statements made by students – in contrast to

evaluative (or assumptive) labels – allows us to code each response objectively. Only at the end,

after coding is complete, do we determine which descriptive codes – not individual responses – should be considered “related” or “unrelated” to each question. Also, as described in Section 3.3.4, multiple researchers worked together to establish the objective validity of the coding criteria presented here. Therefore, the two analyses of the same data differ.

One might argue that this post-coding determination of whether particular expressions of confusion are related to each of the content-related questions could affect the observed

relationships. To test this, we also consider different comparisons in our analysis: (1) students who express no confusion and students who express confusion, and (2) students who do not express confusion related to the question under consideration and students who do express confusion related to the question under consideration. The results of these alternate sorting criteria agree with the results presented above; therefore, the post-coding determination does not significantly alter the observed relationships.

61 3.5.2 Limitations of Pilot Analysis

Although the hypothesis introduced above is a viable explanation for these results, it is certainly not the only one. Limitations include: (1) the inability to control for other, potentially significant background variables, (2) the specific way in which questions were posed, and (3) the fact that this reading exercise is only one measurement.

A more general metric like the final exam or final grade may better represent student learning or performance, as the reading exercises were not actually evaluated for correctness. The

willingness to express confusion may relate more directly to confidence in one’s abilities than to performance in the course. These additional variables, among others, could significantly inform – or even alter – the observed relationships.

Even the information we have limits our ability to draw conclusions from the results. As noted above, students were only asked what they found most confusing. The group of students who express confusion that is unrelated to the content-related question under consideration may be actually composed of students who are less confused and students who are not confused about the question. If this is the case, the proportion of the students who respond with the correct answer from each of these groups may significantly alter the results.

As this assignment is essentially one data point – there are 151 students, but only one reading exercise – there are numerous alternative hypotheses that account for the observed results. The first question, although challenging, does not necessarily require students to understand

capillarity to respond correctly; it is sufficient to recognize that the pressure must be equal at equal heights in a static system. Therefore, expressions of confusion may not correlate with confusion about capillarity for this reason, which has nothing to do with students’ metacognition. The second question, for different reasons, does not require students to understand Laplace’s

62

law; the correct response is simply stated in the text. Students may have completed the reading assignment and, appearing to “perform” better than their peers who did not complete the reading assignment, express confusion about the subject. In other words, one question is only

superficially related to the material the other question can be answered directly from the reading without any understanding. This may explain why we observe a different pattern of results in the two questions. Moreover, there is no evidence to suggest that students who express no confusion simply did not complete the reading assignment.

We attempt to address all of these shortcomings in the full study, which we describe in the subsequent section. However, some of these challenges stem from the fact that we simply cannot know what students do not say. We can study and interpret expressions of confusion, but we must always acknowledge that students’ expressions of confusion may not precisely correspond to confusions that they have in mind. Only the former, however, is a candidate for assessment of student learning.