Changing the System by Measuring

We acknowledge that the act of asking students to participate in so many “metacognitive activities” throughout the year could certainly change their behavior. We do not know, for example, the extent to which confidence and performance are positively related simply because students were asked about their confidence.

JiTT and PI are designed to make physics instruction more metacognitive. Frequent

opportunities for reflection about sources of confusion, coupled with collaborative interactions in the classroom and a strong focus on the value of self-directed learning, are built on the

assumption that learning outcomes benefit from such activities. We note that, despite the much more prominent role of lecture during the fall semester, the relationships observed across both semesters are qualitatively similar.

In spite of all the metacognitive exercises, the average self-efficacy score of students in the course virtually did not change over the duration of the spring semester (Table 3.5). This is somewhat surprising, as we might have expected to observe some positive changes.

84 3.8.3 Implications for Instruction

There are two distinct questions about confusion that are particularly relevant to instruction: (1) should we assess students’ expressions of confusion, and (2) how do we interpret students’ expressions of confusion? The answers to these questions depend on the manner in which confusion is assessed.

When relevant factors are not controlled for, expressions of confusion are either negatively- related or unrelated to reading exercise performance, confidence, final grade and self-efficacy. Thus, if instructors assess expressions of confusion in such a way that they are unable to control for the factors considered in this study, then “the less confusion, the better.” However, given that the relationship between course outcomes and expressions of confusion is so weak, the best advice may be simply to avoid assessing expressions of confusion in such an uncontrolled way.

On the other hand, when relevant factors are controlled for, expressions of confusion are negatively-related to reading exercise performance, confidence and self-efficacy but positively- related to final grades. Thus, if instructors assess confusion in such a way that they can control for these negatively-related factors, such as the means of assessing confusion that is presented here, then they can distill the productive role of confusion from the unproductive roles. Monitoring the productive role of confusion affords instructors the opportunity to both assess students’ metacognition as they engage in activities and perhaps even promote such constructive expressions of confusion.

Ultimately, expressions of confusion, when carefully assessed, can play a positive, important role in instruction.

85 3.8.4 Limitations

We note several specific limitations of this study.

Asking students to express their confusion before responding to content-related questions could introduce a stereotype threat that affects performance. This effect would induce a negative relationship between confusion and correctness, which is precisely what we observed; we cannot determine the extent to which this stereotype threat may be amplifying the negative relationship. Ironically, the confusion question was posed before the other questions because we did not want to introduce bias in the other direction; that is, we did not want to influence students’ responses about their confusion by asking them challenging questions about the reading assignment

immediately beforehand. We were also concerned that students would say, “I am confused about the previous question,” rather than something related to the reading (in the fourth “follow-up” question, many students responded in precisely that way).

Another limitation of our analysis is that we depend upon students to rate their confidence, confusion and physics self-efficacy on scales that are not calibrated to any external standards. One student may interpret his/her abilities and understandings very differently than another student who, objectively, may be very similar. In addition to the potential disconnect between what students think and what they express, this lack of calibration introduces another potential disconnect between what students express and what they mean to express. Ideally, we would develop calibration activities that students could use to gauge their own responses, but we are also very conscious of the low tolerance students have for inconveniences. When more work is required to complete the assignment, the likelihood that students will skip the introspection process increases. Perhaps descriptive rubrics or sample responses would help, though such materials may bias responses of students.

86 3.8.5 Future Directions

The relationship between expressions of confusion and student reasoning, not merely

correctness, could be much more revealing. As we have this data already, one potential direction of future research involves analyzing students’ explanations to investigate how these relate to expressions of confusion. Furthermore, the free-response data collected here can be used to inform and redesign the multiple-choice component of the questions posed over the course of this study. From this data set, we can also identify particular students – high-performing students who express slight confusion, low-performing students who express much confusion, etc. – and investigate their free-response data to better understand confusion as a form of assessment.

From a more theoretical point of view, now that we observe students’ distinct responses to separate measures of confidence, confusion, and self-efficacy, we are able to explore the ways in which each of these measures relates to the theoretical constructs of self-concept, self-efficacy, metacognition, motivation, identity, and numerous other components of the theories described in Section 3.2. Perhaps we must consider the possibility that self-efficacy, like knowledge (diSessa, 2006), exists in pieces and may or may not be activated in different contexts.

Finally, from a more practical point of view, now that we observe a productive role of confusion in instruction, we may be able to further explore the causal relationships between this kind of confusion and course outcomes. Additionally, we may be able to explore the relationship between changes in expressions of confusion, as well as changes in the productiveness of

87

3.9 Conclusion

Students who express more confusion tend to also express lower confidence, lower self-efficacy, and, to a slight degree, weaker performance on reading exercises. However, when we control for all of these factors, students who express confusion tend to also perform better overall, as

measured by final grade. Thus, we suggest that, when relevant factors are not controlled for, expressions of confusion may be either bad or merely uninformative; but when relevant factors

are controlled for, assuming that we are interested in relating assessment of confusion to final

grades, confusion is good. In other words, we are able to identify and isolate the productive role of confusion using surveys and reading assignment activities that are entirely accessible to instructors. Thus, if instructors assess confusion as presented here, then they can monitor the productive role of confusion, assess students’ metacognition as they engage in activities, and perhaps even find ways to promote such constructive expressions of confusion.

88