98 EXPERIMENT-TO-CAUSATION INFERENCE: UNDERSTANDING CAUSALITY IN A PROBABILISTIC SETTING

4.3.1 Uncertainty, Modeling, and Technology

Uncertainty in statistical inference lies at the nexus of statistical and probabilistic rea- soning. Formally, in statistical inference, uncertainty is embodied in concepts such as confidence intervals and significance testing, the understanding of which is based on big ideas such as random behavior, independence, variation, distribution, the Law of Large Numbers and sampling distributions (Gal, 2005; Konold & Kazak, 2008; Pratt, 2005). In this formality, statistical inference is the process through which un- certainty is quantified. The broader process of inference, however, combines this quantification with two other unquantifiable types of uncertainty, data quality and data validity.

Data quality is the uncertainty related to the quality of the design of the study, non-sampling errors, and the measures, data, and information gathered that are used in making inferences; the unquantifiable sources of variation, which researchers at- tempt to minimize in the conduct of a study, that give researchers reason to hesitate in making claims. Data validity is the uncertainty about whether the right data were collected, whether the right questions were asked of the data, whether confound- ing variables explain the findings, whether the process that generated the data has changed over time and hence applications of any findings are no longer valid, and whether the researchers’ mental model of the world matches reality. This type of un- certainty also includes doubt from the knowledge that findings are based on current knowledge and that findings can be overturned in the future in the face of new evi- dence (e.g., Scarf, Imuta, Colombo, & Hayne, 2012) leading to the realization that all knowledge is uncertain, which can lead to skepticism about any evidence. In ef- fect, when students make an inference or claim, they need to consider or weigh the evidence on these three types of uncertainty. A major question is how to untangle these three types of uncertainty when developing students’ reasoning about making judgments from data with respect to experiment-to-causation inference.

Researchers have examined how people make judgments under uncertainty (Kahneman, 2011), at what age students understand the construct of uncertainty (Langrall & Mooney, 2005), and how young students articulate uncertainty (Ben- Zvi, Aridor, Makar, & Bakker, 2012). They have been surprised at the deep-rooted cultural bias towards deterministic thinking, which seems to interfere with devel- oping students’ ability to reason probabilistically (Fischbein, 1975). To conceptu- alize the world non-deterministically requires long-term experiences and reflection upon probabilistic situations including an emphasis on modeling random behavior (Garfield, delMas, & Zieffler, 2012; Greer & Mukhopadhyay, 2005). The purpose of such modeling is to mimic random behavior in a real world system in an effort to understand the behavior of the real-world system, to answer questions about that sys- tem, and to predict future outcomes in the real-world system (Pfannkuch & Ziedins, 2013).

Modeling random behavior underpins the quantification of uncertainty using for- mal methods for statistical inference (e.g., confidence intervals and significance test- ing). We believe that introductory statistics students should be introduced to the quantification of uncertainty via bootstrapping and randomization methods rather

LITERATURE AND BACKGROUND 99

than through the conventional parametric approaches for inference that rely on a mathematical formalization (e.g., t-test, ANOVA). In line with Cobb (2007), we think the logic of inference, and the “big ideas” and concepts underpinning inference are more transparent to students using these methods, and are transferable to a wider range of situations. Also, these methods are becoming more prevalent in statistical practice (Hesterberg, Moore, Monaghan, Clipson, & Epstein, 2009). Moreover, the research of Madden (2008a, 2008b, 2011), Garfield et al. (2012) and Tintle, Topliff, Vanderstoep, Holmes, and Swanson (2012) point to positive outcomes in students’ statistical inferential reasoning when using randomization methods to teach infer- ence for probabilistic situations.

The bootstrap and randomization methods can also be mediated through visual representations, which allow some concepts to become more accessible to students. Technology helps students link multiple representations—visual, symbolic, and numeric—and enhances their understanding through promotion of a visualization approach to learning (Sacristan et al., 2010). Dynamic software can allow students to analyze directly the behavior of a phenomenon, to visualize statistical processes in ways that were not previously possible, such as viewing a process as it develops rather than analyzing it from the end result. Such representational infrastructure al- lows access to statistical concepts previously considered too advanced for students. As Wood (2005, p. 9) states, simulation approaches “offer the promise of liberat- ing statistics from the shackles of the symbolic arguments that many people find so difficult.”

4.3.2 Our Approach to Experiments and Inference

The experiments we refer to henceforth are comparative experiments that have both an intervention and random allocation to groups. The random allocation is performed in an attempt to make the group comparisons “fair”; a design that can facilitate causal inferences about the effects of an intervention. To assist introductory statistics stu- dents in making a direct conceptual connection, we adopted as a basic principle that the “inferential method should mirror the process of data production” (Wild, Pfannkuch, Regan, & Parsonage, 2013, p. 9). That is, the data is produced by ran- dom allocation to treatment groups and therefore the inference method should be based on random re-allocation to treatment groups. As (Teague, 2006, p. 169) stated:

The experimenter must always pay careful attention to the design of an experiment, since the method of analysis is determined by the manner in which the experimental units are randomized to treatments. The way you randomize is the way you analyze.

To enable students to make an experiment-to-causation inference we expect them undertake three actions: (1) thinking about the data obtained from an experiment; (2) conducting the randomization test by modeling random behavior; and (3) making a claim about the data. All these actions involve drawing on the underpinning ideas about uncertainty in making inferences.