118 EXPERIMENT-TO-CAUSATION INFERENCE: UNDERSTANDING CAUSALITY IN A PROBABILISTIC SETTING

Table 4.6

Summary of Elements within Uncertainty that May Need to be Addressed in Instruction for Experiment-to-Causation inference

Element Description of reasoning and ideas

Causal evidence Understanding that in a properly executed randomized comparative ex- periment causality can be established if the values of the explanatory variable (treatment) are randomly assigned to the units.

Randomization Test Understanding the purpose of the test and reasoning and ideas un- derpinning the quantification of uncertainty towards experiment-to- causation inference (see Figure 5).

Tail Proportion Understanding that the aim is to detect a signal, the treatment effect, under the obscuring effects of noise or chance variation. A small tail proportion indicates a signal has been detected, while a large tail pro- portion indicates a signal has not been detected suggesting that noise could be obscuring the signal or there could be no signal, just noise, implying that a claim cannot be made. (See discussion section on this metaphor.)

Treatment is effec- tive

Understanding that the treatment is effective element is composed of a chance component and a treatment effect component.

Rare occurrence Realizing the possibility, although small, that a difference in centers at least as large as that observed could happen by chance alone. That is, the observed difference may be a rare occurrence and the wrong inference may have been made (Type 1 error—not covered in our two- lecture introductory instruction).

Generalization Understanding that care must be taken with any generalization to a wider group than those in the study who were volunteers with partic- ular characteristics (e.g., male, high blood pressure). The population is all those who participated in the experiment. Inappropriate to think about a wider population.

Tendency Understanding that the inference is about the tendency of the treatment group as a whole to improve, not every individual.

Confounding vari- ables

Understanding that unknown or potential confounding variables can be treated as chance explanations, which are accounted for in the method of random assignment and in the re-randomization distribution. Design issues (e.g.,

group size) Realizing a design issue such as group size is not a problem. Under-standing that smaller group sizes require a large observed difference in centers in order to detect whether the treatment is effective under the obscuring effects of chance variation compared to larger group sizes. Sample-to-

population in- ference

Realizing that a designed experiment uses volunteers, does not take a sample from the population, and does not aim to make an inference about a population; rather it aims to make an inference about an inter- vention.

DISCUSSION AND IMPLICATIONS 119

Table 4.6 – continued from previous page

Element Description of reasoning and ideas

Contextual knowl-

edge Understanding that claims are based on the data in hand and that con-textual knowledge, for example, about possible biological mechanisms for the observed difference in centers is used for the next stage of an in- vestigation. Realizing that one’s own contextual knowledge and beliefs can bias perceptions or leads one’s thinking astray.

All knowledge is

uncertain Acknowledging that there are other sources of uncertainty such asquantification of uncertainty for statistical inference as well as the un- certainty about current knowledge being overturned in the future.

4.6 Discussion and Implications

Research has largely focused on sample-to-population inference and has consistently documented a tendency for students to think deterministically or causally and to not take sample size into account (e.g., Kahneman, 2011; Meletiou-Mavrotheris, Lee, & Fouladi, 2007). Students, however, when first introduced to experiment-to-causation inference do not seem to be willing to use causal thinking from the designed experi- ment. Rather, they tend to focus on many considerations of uncertainty and causality, such as sources of variation within the study design, the idea that all knowledge is uncertain, the group size is too small (not a concern in this situation), not every in- dividual case benefits from the treatment, the observed difference might be a rare occurrence, the group of people on whom the experiment was conducted were vol- unteers, wondering about the biological mechanism behind the treatment, chance explanations, and chance is acting alone. The idea that chance and treatment act alongside one another also needs to be addressed in instruction. Making judgments from data, therefore, involves students in untangling considerations regarding uncer- tainty in the realms of statistical inference, data quality, and data validity.

Cognitively coordinating, attending to, and building conceptions of uncertainty for experiment-to-causation inference requires a teaching sequence that gradually develops more sophisticated notions of uncertainty and causality which addresses the elements of uncertainty identified in Table4.6. The integrated textbook, Core-Plus Mathematics (Hirsh, Fey, Hart, Schoen, & Watkins, 2008), has learning trajectories that address experiments and causation using randomization tests with CPMP-Tools (Keller, 2006). Hart, Hirsch, and Keller (2007) believe these tools provide cognitive amplification, resulting in a conceptual understanding of inference. Also of note, is Madden’s (2008) research, which, although addressing different research questions, was able to demonstrate that high school mathematics teachers who participated in a four-day professional development course could successfully compare distributions using the randomization test with CPMP-Tools and Fathom™ (Finzer, 2005). Similar to the findings of many researchers about developing students’ probabilistic reason- ing (e.g., Garfield et al., 2012; Konold & Kazak, 2008), however, we conjecture that developing students’ understanding of causality in a probabilistic setting will require multiple experiences over several years.