Data Management and data analysis

Data collected from the students’ questionnaires was placed in self-sealing plastic envelopes on which were labels giving the name of the school, the number of students who responded, the date collected and the researcher’s signature. The teachers’ evaluative questionnaire and interview schedule was placed in a separate envelope for each school. Checklists of the students’ school results were placed in self-sealing plastic envelopes on which were labels giving the name of the school, the number of students whose school results were recorded and the researcher’s signature.

All data was checked for completeness of the questionnaires as a way of controlling quality. The researcher managed to get 476 complete records of school results from the 516 students who completed the students’ questionnaires. It appeared that usually students who had missing records were those who had transferred from one school to another. The questionnaires were numbered separately in order to differentiate them, as S1, S2, S3 etc, through to the end, and the teachers’ combined evaluative and interview schedules were numbered as TI, T2, and T3 through to the end. For open-ended questions, the most frequently occurring responses were coded. The data entry clerks, under the supervision of the statistician, entered data from the questionnaires using SPSS software.

The SPSS statistical software, with its capacity to handle quantitative and qualitative data, was used for analysis. Descriptive statistics allow the researcher to organize the data in ways that give meaning and facilitate insight. The researcher used percentages, means and standard deviation to give some indication of how scores were dispersed (Bryman & Cramer 2001:69). The paired t-

test compared the means of the same participants to determine whether the average coursework marks and average examination marks differed significantly.

ANOVA was used to compare the means of three or more variables. ANOVA is essentially an F test in which an estimate of the between-groups variance (or mean square as the estimate of the variance is referred to in analysis of variance) is compared with an estimate of the within-groups variance by dividing the former by the latter. The total amount of variance in the dependent variable (e.g. learning outcomes) can be thought of as comprising two elements, that which is due to the independent variable (e.g. area of location), described as (explained variance) and that which is due to other factors (error or residual variance). If the between-groups’ (that is explained) estimated variance is considerably larger than that within groups (that is error or residual), then the value of the F ratio will be higher, which implies that the differences between the means are unlikely to be due to chance (Bryman & Cramer 2001:144). ANOVA reveals whether there is a significant difference between groups, but does not inform us where this difference lies.

Post-hoc analysis produces homogeneous subsets of the groups under analysis and shows how the groups differ. It is done only after the data has been initially analysed (Bryman & Cramer 2001:148). The differences amongst means and relationships of factors impacting on learning and learning outcomes were computed statistically. Data were presented in tables, figures and histograms.

3.12 Validity and Reliability

In research, the concepts reliability and validity refer to the measurement of data as it answers research questions. The aim of the study was to collect information that was as reliable and valid as possible. For the research results to be reliable and valid, the information gathered had as far as possible to reflect the realities of the participants. It was therefore the researcher’s obligation, throughout the research process, to consider circumstances that could influence the findings. The stratified random sampling used by the researcher in selecting schools (research sites) and the simple random sampling used by the researcher to select participants (students and teachers) helped in achieving external validity. The extent to which a study’s results can be generalized or applied to other students and settings reflects its external validity (Huitt et al 2001:2).

The instrument that measures the variables is central in determining reliability and validity of data. The goal of using a reliable instrument is to attain accuracy, which has consistency, stability and repeatability as its attributes. It is also important to know the validity of the

measure, that is, whether the instrument used to collect data actually measures what it is

supposed to measure (Brink & Wood 1994:170-171).

In this study, the construction of the student questionnaire involved the use of portions of existing instruments. Permission for such use was sought and granted, which enhanced the validity of the questionnaire. All questionnaires were subjected to pilot testing and adjustments

were made as indicated to improve accuracy. Careful design and pre-testing of instruments

reduce bias caused by instrumentation that includes vaguely phrased questions or questions placed in an illogical order, fixed or closed questions on topics about which too little is known and open-ended questions without guidelines on how to ask or answer them (Varkevisser 1991:148). The questionnaires were submitted to the supervisors of this thesis so that they might judge content validity. Content validity is concerned with the sampling adequacy of items for the construct that is being measured. Content validity of an instrument is necessarily based on judgement (Polit and Hungler 1995:418).

The evaluation research design allowed for the use of a variety of data-collecting methods and the comparison of results in answering research questions. The single-subject experimental approach adopted by the study helped in achieving high internal validity since subjects served as their own controls. The participants in comparison with the control and experimental groups were functionally equivalent at the beginning of the study so that the observed differences between the groups as measured by the performance dependent variables at the end of the study were not biased (Huitt, Hummel, Kaeck 2001:1,2).

Variability or bias in observations occurs when participants change behaviour because of research and interviews done without guidelines, and when researchers differ in what they observe and measure (observer variability) (Varkevisser et al 1991:149). This study ensured the quality of data by careful selection of the research assistant, training him and providing him with guidelines and by on-going supervision in the data collection process, so that observer bias was minimised. The effect of the interview on the teachers was reduced because the researchers gave

adequate explanations of the purpose of the study, allowed sufficient time for the interview and assured the teachers that data collected was confidential.

The primary methods used to achieve internal and external validity are randomisation and the use of a research design and statistical analysis that are appropriate to the type of data collected and the questions the investigator is trying to answer (Huitt, Hummel, Kaeck 2001:3).