Methods – Pilot Analysis

In order to investigate the results from previous work and better assess our ability to answer the primary research question, we first reexamine the data collected and analyzed by Crouch and Mazur. In this pilot analysis, we addressed the following research question: What is the relationship between students’ performance on the content-related questions and their expressions of confusion?

In the subsequent full study, we apply the insights from the pilot analysis toward a much larger question involving many more reading exercises and several additional measures of student learning and engagement.

3.3.1 Course Information

We analyzed data collected from an introductory physics course at Harvard University during the fall of 1998. This was the first-semester course in a one-year sequence for non-physics

concentrators. The instructor employed PI and the pre-class components of JiTT. Students received credit for participation in, but not correctness of, their responses to both pre-class and in-class questions.

Importantly, in this course, pre-class reading assignments included only free-response questions. The two content-related questions were intended to be thought-provoking and counterintuitive, so students would have to critically engage with the material before class. Questions required students to explain their reasoning. In each assignment, the confusion question was posed after the two content-related questions.

Students submitted responses online using the Interactive Learning Toolkit (ILT), a course management platform developed for use with PI and JiTT.

54 3.3.2 Measures

In the pilot study, we analyzed two facets of a single pre-class reading exercise: student performance on each of the content-related questions and student expressions of confusion.

For this assignment, students were required to read Chapter 15 of General Physics

(Sternheim & Kane, 1991). In the chapter, the authors detail the principles and mathematics that describe surface tension, capillarity, and Laplace’s law. The subsequent reading exercise

consisted of the following questions, posed in this sequence:

(1) Consider the capillary rise of liquid in a glass tube (shown in Figure 3.1). How does the pressure at point P at the surface of the liquid compare to the pressure at point Q of equal height?

Figure 3.1: The capillary rise of water in a glass tube.

(2) Two identical balloons are connected to a tube (shown in Figure 3.2). Balloon B is inflated more than balloon A. Which way does the air flow when valve P is opened?

55

Figure 3.2: System of two balloons connected by a valve.

(3) Please tell us briefly what points of the reading you found most difficult or confusing. If you did not find any part of it difficult or confusing, please tell us what parts you found most interesting.

Students responded to these questions in free-response, short-essay form. Although this format provides students flexibility in their responses, it allows them to submit correct answers with incorrect justifications, “incorrect” answers with well-reasoned justifications, and a plethora of other ambiguous responses. We could have labeled responses “completely correct” or “not completely correct,” but then correct responses without justification, of which there are many, could alter the apparent relationship between performance and expression of confusion. Therefore, instead of subjectively making judgments and labeling responses as “correct” or “incorrect,” we labeled responses descriptively, according to the outcome stated by the student (e.g., in question (2), labels included “flows to the left,” “flows to the right,” and “does not flow”).

We employed a similar descriptive approach in labeling student expressions of confusion. As detailed in Table 3.1, we coded responses as relating to the textbook (explanation provided,

56

figures, equations, examples, symbols, problems and terminology), concepts (laminar flow, cohesive forces, surface tension, contact angle, capillarity, Laplace’s law, pressure), the reading exercise itself (question one and question two), general goals and “real world” applications, which means the response relates to connecting the formal content to actual practices. Different concepts are associated with different sections of the text: Sections 15.1 and 15.2 relate to the concepts of contact angle, surface tension and capillarity; Section 15.3 relates to the concept of Laplace’s law; and later sections relate to other, more applied concepts. A single response may require multiple labels.

Table 3.1: Codes used to label student responses to confusion question.

textual section conceptual section other section

explanation any laminar flow 14 question 1 15.1 and 15.2

figures any cohesive forces 15.1 question 2 15.3

equations any surface tension 15.1 objectives any

examples any contact angle 15.2 applications 15.2 or 15.4

symbols any capillarity 15.2 not confused none

problems any Laplace’s law 15.3

terminology any pressure 15

By coding expressions of confusion this way, we are able to sort the responses according to several different criteria: textual vs. conceptual, confusion vs. no confusion, and question-related confusion vs. non-question-related confusion. There are no statistically significant differences in the proportion of students who respond with the correct answer between groups that express text- related confusion and concept-related confusion. Thus, we collapse these codes together and

57

focus on the content areas to which the expressions of confusion relate. Because students were asked to identify only the points they were most confused about, we cannot assume that they were exhaustive in their responses.

To summarize, students were sorted based only upon these two facets of their responses: the answer provided to each of the content questions and the explicit sources of confusion identified.

3.3.3 Sample Description

The sample consists of 151 students. As we have access to only the responses of these students to this pre-class reading exercise, we cannot comment on the relationships among additional factors (e.g., pre-course knowledge, final grades). Overall, 43 students (28%) responded with the correct answer to the first content-related question, 37 students (25%) responded with the correct answer to the second content-related question and 16 students (11%) responded with the correct answers to both content-related questions. 62 students (41%) expressed confusion related to the first content-related question, 47 students (31%) expressed confusion related to the second content- related question, and 39 students (26%) expressed no confusion.

3.3.4 Analytic Methods

To investigate the differences in performance between groups of students who express confusion about different parts of the reading, we performed chi-squared tests to determine whether there is an association between the confusion and correctness classifications. This test permits

comparisons between multiple groups. If only two groups are under comparison (i.e., a 2 × 2 table), the z test for comparing proportions between two groups can also be used, though both tests yield precisely the same result (Moore, McCabe, Duckworth, & Alwan, 2008).

58

Two researchers independently coded students’ free responses to reading exercise questions and subsequently discussed coding. All discrepancies were resolved during discussion, and the coding criteria were refined to eliminate future discrepancies. Once the criteria for each code were sufficiently well established through discussion, remaining reports were coded

independently.