Pre/Posttests

The pre/posttests given to the participants are discussed here. Both tests were composed of questions designed to determine the argument skill level of the study participants. Each was composed of the three questions and are shown in Appendix A. Commentary that explains why each question type was chosen along with example responses are provided. The purpose of the pretest was to help answer inquiry question one and to also provide a baseline from which to make a comparison to poststudy skill level.

In addition to these important investigatory purposes, knowing the incoming argument skill level of students was simply pedagogically valuable. Lawson (2004) claims that science teachers need to know the intellectual development level of students at the start of any course that requires scientific reasoning. He suggests that without sufficient intellectual development, students cannot construct scientific arguments or express scientific explanations. Efforts to construct arguments will “fall apart” unless students have developed their hypothetico-deductive reasoning abilities, which are present (but undeveloped) at birth, to sufficient levels (Lawson, 2004, p. 322). According to Lawson (2004), this occurs through an awareness that grows with time through reflection and application.

Using the stages of intellectual development described by Piaget, Lawson (2004) expresses that humans develop intellectually from the preoperational to concrete operational level of intellectual development sometime between the age of seven and preadolescence. At the concrete level, students can test hypotheses descriptively, but not causally. Causal hypothesis testing is associated with the formal operational stage of development which manifests in early to late adolescence. In order for a student to argue scientifically at the level required in AP Physics 1, she must have developed to the formal operational stage. Although Lawson couches his ideas in

Piaget’s model of development, he carefully states that, “… use of the Piagetian stage labels does not imply acceptance of his theory concerning their underlying operations…” (Lawson, 2004, p. 323). Lawson’s (2004) work represented a good foundation on which to base a pretest for my research. The pretest was meant to identify if students can construct scientific arguments. Therefore, items on that instrument consisted of various argument construction tasks that indicate human intellectual development up to and including the formal operational level.

Each of the items asked on the pre/posttests required a student to make a claim on given information and then fully explain why they made that claim. The student’s response was used to determine whether she skillfully supported the chosen claim with evidence and justification (rationale) according to the rubric shown in Table 7. This rubric was modeled after that used by Gotwals, Songer, and Bullard (2012) but with the claim descriptor, accurate, taken out. The removal was made so that the score assigned to the response did not depend upon which claim the student chose. Instead, the score was based upon the level of support offered for any chosen claim. Cavagnetto and Hand (2012) make clear that the rationale for an argument does not have be made via completely separate declarations, but may be meshed within claim and evidence statements. However, in the rubrics developed by Gotwals, Songer, and Bullard (2012) that were customized versions of the general rubric designed to evaluate a specific task, responses that included implicit rationale were not evaluated at the highest possible level. For their customized rubrics, explicit reasoning was required to receive a level four evaluation. For purposes of this investigation, that requirement seemed appropriate based on expectations for sufficient argument structure as demanded by the College Board on AP Physics 1 exams (see Figure 4, p.61). Thus, I added the italicized word explicit shown in the level 3 and level 4 rubric descriptions for clarity and to

emphasize the importance of constructing complete arguments that could be objectively analyzed to as high a degree as possible.

Table 7. Structure rubric

Argument Structure Evaluation

Description

Level 4 Student constructs a complete evidence-based explanation (with a claim, appropriate and sufficient evidence, and explicit reasoning that ties the two together).

Level 3 Student makes a claim and backs it up with sufficient and appropriate evidence but does not use explicit reasoning that ties the two together. Level 2 Student makes a claim and provides evidence to support the claim.

However, the evidence is either insufficient or inappropriate.

Level 1 Student makes a claim but does not back it up with evidence (even though a rationale may be attempted).

I expected that the use of the structure rubric with the pre/post test data would allow me to produce a snapshot in time at two important points in the study. They turned out to be just that; momentary views of the path an individual student was traveling amidst a jungle of competing concerns. With that in mind, I did not approach these assessments as ways to prove how lack of reasonable evidence or justification in the response was necessarily suggestive that the student could not support claims in general. Instead, I viewed them as indicative of how a student chose to respond to a particular question at a specific moment (Sandoval & Millwood, 2005). That said, if a pattern of not using appropriate evidence and explicit rationale throughout the pretest or posttest was noted, this may, among other possibilities, suggest that (1) the student was unaware of the requirements for creating a valid scientific argument, (2) was unable to analyze the given data for appropriate evidence, (3) assumed that the “audience” for which the response was intended shares a common understanding of warrant implicit in the response (Manz, 2015), or (4) was

simply not interested in performing at the level she was capable of. In the first three cases, my working hypothesis (see section 3.3) was tested via a pre/posttest comparison. In case three, for example, improving argument skill may have been as simple as impressing upon students the importance of explicitly stating their reasoning in a way that logically connects the evidence to the claim and doesn’t require assumptions to be made. In the fourth case, responses to pretest and posttest questions would lose meaning.