Data Analysis Procedures

In analyzing the data the researcher strove for methodological rigour (Clement, 2000). But before discussing data analysis procedures it was necessary to briefly describe the relationship between reliability and validity of the research instruments used in this study. While conducting research we make lots of different inferences or conclusions on matters related to the process of doing research. Like the bricks that go into a building wall, these intermediate process and methodological propositions provide the foundation for the substantive conclusions that we wish to address which involve measurement or observation (Trochim, 2000). And according to the author (Trochim, 2000), whenever we measure or observe we are concerned with whether or not we are measuring what we intended to measure and/or do we have a dependable measure or observation in a research context. We reach conclusions about the quality and consistency of our measures – conclusions that will play an important role in addressing the broader substantive issues of our study. For this purpose the ideas of validity and reliability will be discussed in the subsequent section prior to the discussion on qualitative and quantitative data.

106 3.4.1 Validity and Reliability

Reliability and validity are two main psychometric characteristics used to assess the quality of the measuring instruments used in research. The central concept in measurement is reliability which essentially implies consistency or repeatability. The two main components to consistency are stability which refers to consistency over time and internal consistency (Punch, 1998). From the constructivist position, reliability in interpretive research is replaced by the idea of dependability (Guba & Lincoln, 1989).

Validity is the second central concept in measurement. The meaning of concept validity can be found in answer to the question: how do we know that the instrument is measuring what we think or designed it to measure? A second view focuses on whether the interpretations we make from the measurements are defensible, concentrating on the interpretation of data rather than the measurement instrument (Punch, 1998). Validity can be classified as internal or external. Internal validity is defined from a positivist paradigm as the degree to which variations in an outcome or dependable variable can be attributed to controlled variation in an independent variable. External validity defined from a positivist perspective is the approximate validity with which we infer that the presumed cause-effect relationship can be generalized to different individuals, situations, or time (Guba & Lincoln, 1989).

In qualitative research, validity relates to whether findings of the study are “true and certain” (Guion, 2002, p. 1). “True” in the sense of the results accurately reflecting the real conditions and “certain” in the sense that the findings are supported by evidence. “Certain” also implies that the “weight of the evidence” supports the conclusions and there is no reason to doubt the results (Guion, 2002, p. 1).

Among the threats to internal validity the researcher identified in this study were of maturation and testing. The maturation threat can operate when biological or psychological changes (over the six weeks) occur within the students and these changes may account for in past or in total for effects discerned in the study. The testing threat may occur when changes in the test scores occur not because of the

107

teaching intervention but rather because of repeated testing. This is of particular concern when identical pretest and posttest were administered in this study.

3.4.2 Qualitative Data

Qualitative data for this study were collected from student interviews. A total of 9 students were selected purposefully to be interviewed. Student interviews were recorded and transcribed verbatim in order to allow the researcher the ability to thoroughly assess student conceptual understanding (difficulties, misconceptions and conceptual change) of mathematics. For the purpose of the interview the researcher chose to use what Fontana and Frey (1994) characterised as a semi-structured interview. The semi-structured form of interview allows the researcher to gain the most amount of insight into the difficulties and misconceptions students encountered, examine whether or not conceptual change took place or determine the status of a student’s conceptions; while at the same time allowing the interviewer to use structured questions as a means of keeping the interview focus within certain succinct areas of importance. The maximum length of each interview was 15 – 20 minutes.

It was decided that an efficient way of checking the validity of the Algebra Diagnostic Test was to interview students from the study group. The misconceptions previously identified would no doubt be revealed in the errors made by the students interviewed. By examining the thought processes of students in this way, it was also hoped that any misconceptions which had previously not been revealed, would be identified. It was also necessary for the interviewer to use probes to facilitate student problem solving (e.g., if students do ‘this,’, then prompt for “that”). Students were asked to explain their thinking and selection of solution strategy (e.g., “Tell me more about your thinking on this problem”). Questions proposed by Hewson and Thorley (1989, p. 550) like, “How would you explain that to a friend?” were included to encourage students to reflect on their own conceptions and thus to elicit status information (Treagust et al., 1996). It was hoped that through students’reasoning and explanation, the researcher can draw accurate conclusion on whether students have

108

understood the concept in question and as an indication of successful conceptual change.

The researcher was particularly interested in student participation during class, student perceptions of the diagnostic teaching unit taught, and student behavior throughout the teaching of the unit and the testing procedure. All of the qualitative data were then analysed and combined with quantitative data as a means of answering the research questions.

3.4.3 Quantitative Data

The Algebra Diagnostic Test and TOMRA data were analysed to verify the internal consistency of each scale and discriminant validity. The internal consistency of each scale on the Algebra Diagnostic Test and TOMRA were determined using the Cronbach (1951) alpha coefficient using the student as the unit of analysis.

The internal consistency, mean, standard deviation, and t-value were statistical analyses used to evaluate and interpret data obtained from the diagnostic test and the TOMRA scales. The average item means (scale mean divided by the number of items) were used enabling important camparisons of the scales. Since these achievement measures involved a pretest-posttest design, it was necessary to use statistical t-tests for paired samples with the resulting p-value to form a decision. If the p-value is .05 or greater, then the relationship between variables is considered to be inconclusive. Then, there is no need to compute Cohen’s d (effect size index) to provide information about how strongly the variables are related. However, if the p- value is less than the traditional value of .05, the results are statistically significant, and support is inferred for the relationship under study (Gigerenzer, Swijtink, Porter, Daston, Beatty, & Kruger, 1995). In such a situation, effect size index (the difference in means expressed in standard deviation, σ, units) were calculated to provide information about the magnitudes of the effect. The estimates of magnitude of effect or effect size tell us how strongly the variables are related, or how large the difference between variables is. Its calculation enables immediate comparison to increasingly larger numbers of published studies. Cohen (1992) proposes as a convention, the arbitrary effect size (d) values of .20 as small, .50 as medium, and

109

.80 as large enables us to compare an experiment’s effect-size results to known benchmark showing how far and in what direction a given treatment pushes performance along the normal distribution curve (Cohen, 1988). Abelson (1995) predicts that “as social scientists move gradually from the reliance on single studies and obsession with null hypothesis testing, effect size measures will become more and more popular” (p. 47).

3.5 Summary

This study involved four research questions. The nature of the research questions mandated that various instruments be used in order to gain answers relevant to the questions being asked. As this study used Form 2 students (Grade 9 equivalent) as the sample population, and as instruments needed to collect data from this sample population were not readily available, the researcher found it necessary to either design new instruments for data collection, or to amend presently existing surveys. For this reason, the researcher amended various instruments. These instruments included an amended form of the Algebraic Thinking Test for students renamed the Algebra Diagnostic Test and amended form of the Test of Science-Related Atiitudes (TOSRA) renamed the Test of Mathematics-Related Attitudes (TOMRA). As a result, the preceding section has given a description of the various instruments used in this study.

This chapter has given a comprehensive description of diagnostic teaching methodology and how it was to be implemented. A detailed description of data collection and analysis procedures including quantitative and qualitative data was also provided.

110

Chapter 4