104 EXPERIMENT-TO-CAUSATION INFERENCE: UNDERSTANDING CAUSALITY IN A PROBABILISTIC SETTING

In our teaching approach we purposefully have the students use language that we hope will partially convey particular concepts. For p-value, we use “tail proportion”, for the null hypothesis we use “chance is acting alone”. However, the dynamic vi- sualizations that we use only show the tail proportion, so we expect inference argu- mentation to remain difficult for students.

In addition to reasoning from the tail proportion, students also have to consider the aforementioned ideas related to causation. Causal, or deterministic, thinking is the predominant mode of thinking within society, with most people not willing to accept the role of chance. Biehler (2011) used an inference example cited in Makar, Bakker, and Ben-Zvi (2011) to point out that the use of probabilistic thinking may lead to other issues. The example, which compared the physical fitness for fifth and sixth graders used the following language: “From these two samples, we infer that the physical fitness in sixth grade is probably better than in seventh grade?” (p. 152). Biehler (p. 6) observed: “‘Probably’ better expresses uncertainty. However what have we exactly gained? All our knowledge is uncertain. We can add this to every sentence we say.” Hence, if people have the point of view that all knowledge is uncertain then they may be unwilling to use causal language, even to express inferences from experiments.

4.3.4 Research Questions

Since there seems to be little research in the area of experiment-to-causation infer- ence with regard to conceptions of uncertainty, and since the VIT software is new and untested with respect to students’ reasoning processes, we believe an exploration into students’ concepts of uncertainty when using inference may contribute to the exist- ing knowledge base. To examine students’ reasoning processes regarding causality and uncertainty in the context of making partially informal experiment-to-causation inferences, we will focus on the following specific research questions:

1. What reasoning processes do students use when thinking about the observed data from an experiment (Action 1)?

2. What ideas and reasoning processes do students use when recalling the random- ization test (Action 2)?

3. What argumentations do students use when making a claim about data from an experiment (Action 3)?

4.4 Subjects and Methods

The findings presented in this chapter come from a collaborative research project involving 33 team members and over 2700 students. The research team was com- prised of a statistical software conceptual developer, an international advisor, two education researchers, two resource developers, five professional development fa- cilitators, eight university lecturers, and 14 secondary school teachers. Using prin-

SUBJECTS AND METHODS 105

ciples of design research (Hjalmarson & Lesh, 2008), the development process in- volved two research cycles, each consisting of four phases: (1) from an identified problematic situation, understanding and defining the conceptual foundations of in- ference; (2) development of new resource materials and dynamic visualization soft- ware (VIT); (3) implementation with Year 13, introductory university, and workplace statistics students; and (4) retrospective analysis followed by modification and sup- plementation of resource materials. The focus of design research is to support and engineer new types of reasoning and thinking in response to problematic situations. As well as being pragmatic through producing an educational product that can be used by teachers, design research can also lead to new educational theories and areas of research (Bakker, 2004).

4.4.1 Participants and Procedure

The research reported in this chapter focuses on the pre- and post-instruction written responses and interviews of six introductory university and workplace students. Sev- enteen university students (randomly sampled from 200 volunteers in a population of n = 2553) and nine workplace volunteers (sampled from n = 14) participated in the interview process. Eleven of these 26 students were randomly allocated the ran- domization posttest in class (others completed a bootstrapping posttest). We chose to concentrate on the responses of six of these students (S1 to S6) because they were interviewed by the same research assistant, and were able to articulate their ideas. These six students’ prior experience of statistics would be fairly representative of about 60% of the university and workplace cohorts. Occasionally we refer to the written responses of the wider cohorts to give an indication of the prevalence of the reasoning under discussion.

None of the participants had any experience with experiment-to-causation in- ference or the randomization test. All students experienced the same learning tra- jectory of two 50-minute lectures for the randomization test, which incorporated hands-on activities, attention to language and verbalizations, and VIT dynamic visu- alizations. Learning occurred within the classroom setting for the university students (class sizes ⇡ 450) and a professional development workshop setting for the work- place students (n ⇡ 20). An assignment component allowed students to use the VIT software to perform the randomization test that was demonstrated as part of the teaching sessions. For a detailed description of the teaching sequence, see Budgett, Pfannkuch, Regan, and Wild (2013).

4.4.2 Assessment Items and Data Analysis

Test and interview items that are discussed in this chapter are provided in the Ap- pendix. Data from the tests were entered into spreadsheets. The first two authors of this chapter initially developed either hierarchical or non-hierarchical descriptors and coding frameworks for each assessment item based on the student data. The decision as to whether a hierarchical or non-hierarchical descriptor was necessary depended on the type of assessment item. At least 200 student responses were independently