Chapter 3. Methodology
3.2 Data Variables
Determining students’ scores and approaches across the software boxes and
tasks lent itself easily to an experimental design as students can be easily assigned to
experimental groups. This design enabled the measurement of students’ performance
and observation of their approaches. A summary of the data variables that were
collected for the Main Study, based on the literature is presented in Table 5. An
additional variable, ‘Problem’ is included in this table. Problem is used as a way of
organising the tasks and is further discussed in Sections 3.3.4 (p.62) and 3.3.5 (p.63).
The variables were grouped into independent variables, non-varying covariates,
varying covariates and dependent variables. Independent variables were variables that
the researcher manipulated such as the assignment of tasks and software boxes to
students. Covariates were variables that this research had no control over but were
present and may influence the study. The non-varying covariate was a variable that was
not influenced by this research design, that is, the covariate remained the same (or near
the same) throughout the study. Thus, a student’s mathematics confidence was expected
to stay the same for the duration of the study. The varying covariates, such as the
propensity towards self-explanations, on the other hand were assumed to be dependent
Table 5: Variables used for collecting data
Variables Description Data type
Independent Variables
Boxes Types of software Categorical
Black-Box Software does not show steps
Glass-Box Software shows steps
Open-Box Software allows interaction at steps
Problems Types of problems Categorical
Problem 1 Toy manufacturing application
problem
Problem 2 Lumber manufacturing application
problem
Problem 3 Mathematically abstract problem
Tasks Types of Tasks Categorical
Mechanical Procedural knowledge using during
solving
Interpretive Conceptual knowledge used during
solving
Constructive Both procedural and conceptual
knowledge used during solving
Non-Varying Covariates
Mathematics Confidence Confidence of the student to do mathematics
Quantitative
Processing Levels (A) The deep or surface processing levels that students take when solving all tasks
Quantitative
Varying Covariates
Variables Description Data type that students use when solving each
task
Self-Explanations The propensity of students to generate out-loud explanations when solving each task
Qualitative
Dependent Variable
Performance Scores that students have made on the interpretive and constructive tasks
Quantitative
Explorations Frequency of using the software for testing numbers or conjectures for the mechanical, interpretive and
constructive tasks
Categorical
Explanations Types of Explanations Categorical
Mathematical Frequency of written mathematical
explanations for interpretive and constructive tasks
Real-Life Frequency of written real-life
explanations for solving the interpretive and constructive tasks
The final variables were the dependent variables. The data collected for these
variables were determined by an outcome of the intervention at different levels of the
independent variables, which in this case were the solving of tasks when provided with
a software box. The outcomes of the intervention were the scores on the tasks,
categorising whether students explored with the software, and categorising self-
explanations into real-life or mathematical explanations.
A mixed-methods methodology was employed (Creswell, 2003) for collecting
data are collected for answering the research questions. Quantitative and qualitative data
may either be collected concurrently or sequentially. Both the quantitative and
qualitative data can then later be used for triangulation. In the concurrent method, both
types of data are collected at the same time, whilst in the sequential method, either of
the two data types are collected first then followed by the collection of the other data
type.
In the experimental design employed for the Main Study, quantitative and
qualitative data were collected concurrently. Through the use of the concurrent
triangulation method, agreement between the quantitative and qualitative data was
sought. In the experimental design, empirical data were collected for the tasks
(performance scores, explorations and explanations frequencies, and processing levels)
which aided in determining if there were any statistical differences or similarities
amongst the software boxes. However, this data do not on its own shed any light on
why differences or similarities might be occurring. More data were thus necessary to
understand why the statistics found were significant and what influenced its
significance. Therefore observations (audio/video recordings and note-taking) of what
students were doing when solving the tasks were conducted. This qualitative
audio/video and note-taking data were then triangulated with the statistical findings.
This meant that even though statistical differences may show quantitatively that there
were no differences between the software boxes, subtle variations in how students
interacted with the software boxes were then able to be obtained via the observational
data.
The use of quantitative data ensured that there was an extent of rigour and
internal validity within the experimental design since statistical probability was used for
was being measured would be able to answer the research questions, that is, the
instrument being used would accurately measure or represent the concepts. They further
explained that internal validity referred to the rigour under which the study was
conducted whilst external validity referred to the extent that the results could be
transferable or generalisable to a population (see also Campbell and Stanley, 1963).
The study was based on university students and this meant that all variables
could not be accounted for in an experimental design, since students’ characteristics can
not be held constant as their behaviour is constructed and reconstructed during the
course of experiment (Hammersley and Atkinson, 1995). To account for this effect, the
qualitative data were collected to gain insight and provide richer data on the students’
behaviour (Hammersley and Atkinson, 1995; Savenye and Robinson, 1996),
particularly when determining how certain approaches were used by a student in solving
the tasks. With qualitative studies there is potentially more subjectivity than in the
quantitative studies and thus precautions were made to minimise bias or at least be
reflexive on how the interviewer/ observer influenced the data (Hammersley and
Atkinson, 1995). Therefore in the observation notes, the researcher noted any activities
that may have influenced the students (for example, notes were made when the
researcher told the students of wrong data inputs into the software box). Further, in the
transcripts, the researcher’s comments and actions were also included.