Data analysis & interpretation

Since this study utilises the mixed-method methodology, both qualitative and quantitative data were gathered, thereby necessitating different methods of analysis.

4.6.1 Statistical analyses of quantitative data

As described in 4.5.1, four quantitative instruments were used in this study to gather data, namely the Burt Word Reading Test, a Cloze test, an Exploratory Test (ET) and Strategy Transfer Test (STT). The first three tests were used to obtain baseline data about the research participants before the start of the intervention, and the Strategy Transfer Test was used to measure transfer of strategy knowledge – as taught during the intervention - in the experimental group upon completion of the intervention, and for comparison with similar measurements in the ET data for before-and-after

measurements within the experimental group. The following statistical analyses were performed on the data from the respective instruments:

A Mixed Model Repeated Measures ANOVA was used to compare learners‘ reading age (as measured by the Burt Word Reading Test) with their real age, and to compare their reading age with their comprehension ability (as measured by the Cloze test). The comparison between reading age and comprehension ability was performed within and between the respective grade groups to establish whether gaps – where they occur – not only exist between learners, but also between grade groups.

A One-way ANOVA was used to compare the measurements from the ET and STT for the learners in the experimental group. A comparison was done for the three measurements obtained from both tests, namely Questioning, Summarisation and Monitoring. The results of this comparison provided an indication of whether transfer of strategy knowledge as taught during the intervention took place.

In addition to the ANOVAs a Pearson correlation was used for determining whether a correlation exists between learners‘ measured reading ages and their comprehension ability.

However, for all their value, the abovementioned three tests mainly report on

whether an intervention made a difference; in order to report on the extent of the

difference made by an intervention, a different type of test is necessary. For this reason an Effect Size test (refer next section) was also be performed on the quantitative data. A spreadsheet calculator based on Thalheimer and Cook (2003) was used to determine the effect size, using the mean scores obtained from the One- way ANOVA.

4.6.2 Effect size or statistical significance

When communicating the findings of studies there is a tendency to focus on whether or not some intervention had the intended effect, and less attention to how much of an effect the intervention had (Valentine & Cooper, 2003). For example, it may be possible to show that a reading intervention increased reading scores more than the usual reading instructional techniques, but it is often more difficult for researchers to determine how much of a difference the intervention made. Researchers need to know if the intervention‘s effects are large or small, meaningful or trivial. For a while

now, data analysts have been advising researchers in the behavioural sciences that, in addition to a test for statistical significance, an effect size measure should also be reported with their findings (Olejnik & Algina, 2000). The reason for this request, according to Olejnik & Algina (2000:241) is that ―statistical significance does not imply meaningfulness‖ and that ―small differences can be statistically ‗significant‘ because of a large sample size‖. Since the data from which statistical analyses for this study can be drawn was taken from a relatively small sample (n=163), I deemed it sensible to include an effect size measure.

Using Cohen‘s d an effect-size analysis (see 4.6.1) was performed on the ET and STT measurements to compare the differences (if any) in learners‘ scores before and after the research intervention. Cohen‘s d measures the meaningfulness of an intervention and reports its results as the size of the effect of the intervention, as well as the percentage of change recorded from the comparison to the treatment, for example small (≥ -0.15 and <.15), medium (≥.40 and <.75), very large (≥1.10 and <1.45), etc. Thalheimer and Cook (2003), whose spreadsheet calculator was used in this study, utilise the following effect size scale for the relative size of Cohen‘s d:

Negligible effect (≥ -0.15 and <.15) Small effect (≥ .15 and <.40) Medium effect (≥ .40 and <.75) Large effect (≥ .75 and < 1.10) Very large effect (≥ 1.10 and < 1.45) Huge effect (> 1.45)

They further use the following scale for the relative size of the percentage change from comparison to treatment:

Huge decrease <-75

Very large decrease (≤ -50 and ≥-75) Large decrease (≤ -30 and ≥-50) Medium decrease (≤ -15 and ≥-30) Small decrease (≤ -5 and >-15) Negligible change (≤ -5 and <5) Small increase (≥5 and <15) Medium increase (≥ 15 and <30) Large increase (≥ 30 and <50) Very large increase (≥50 and <75) Huge increase >75

It must be pointed out that effect size analyses have their critics who question whether effect size measures actually contribute to a better understanding of research results. These critics claim that what practitioners really want to know, are answers to questions such as ―What can the participants of the treatment do because of the intervention that the control group cannot do?‖ (Olejnik & Algina, 2000:277).

Such questions raise validity issues that depend on the meaning of the measures used, the heterogeneity of the populations being compared, the specific levels of the variables studied, the strength of the treatments, and the range of treatments. However, since no composite scores – total scores which measure underlying variables - were calculated for this study, a reliability test was not deemed applicable.

4.6.3 Analysing the qualitative data

Fraenkel & Wallen (2008:476) state that the process of analysing qualitative data requires a researcher to take a holistic view of all gathered data before segmenting and reassembling them into categories. Various approaches for ‗segmenting and reassembling‘ qualitative data have been developed over time. For example, Miles & Huberman (1984) use a three-tiered approach of analytic progression that reduces a complete data set into broad categories, and then into themes, while Tesch (1990:95-96) describes qualitative data analysis as segmenting the whole data set into ―relevant and meaningful units‖ followed by the categorisation of the data segments. A third similar approach is offered by Boeije (2010:78) who suggests the analysis of data into ―fragments‖ that relate to similar themes where after distinctions are made between relevant fragments and sorted into groups or categories.

The analysis and interpretation of this study‘s qualitative data were guided by principles from the three abovementioned researchers, namely Miles & Huberman (1984), Tesch (1990) and Boeije (2010), all of whom propose what is effectively a three-step process in analysing data. According to Patton (2002:437) the researcher has two primary sources to draw from in organising the analysis once data collection has ended: (1) the questions generated during the conceptual and design phase of the study (research questions) and (2) analytic insights and interpretations that emerged during data collection. My analysis of the qualitative data gathered during this study was directed by the study‘s conceptual framework (see Figure 1) and the research questions (see 4.1.3).

It should be pointed out that because the focus of this study is the teaching of reading comprehension (reading strategy instruction), the aim was not to perform a detailed word-by-word discourse analysis of learner and teacher utterances; where any discourse analysis occurs it will focus on identifying changes in a particular teacher‘s instruction or to contrast two teachers‘ instruction with each other.