An effect size is a statistical estimate of the strength of a phenomenon in standardised units. In the context of this research, the effect size provides an indication of the difference between the intervention and control groups for the three outcome measures. Whilst the model coefficients also provide an indication of this, the effect size standardises these coefficients so that they can be compared directly with each other and across other research studies. Without standardisation, the size of coefficient is dependent on the scale and units of the outcome measure and so it is not possible to compare these directly.
As specified by the EEF, the effect size calculated in this report is Hedges g. This is a similar effect size measure as Cohens d but uses a standard deviation that is pooled between the intervention and control groups. The tables that follow provide some summary detail on the three outcome measures and how Hedge's g was calculated.
Whilst we have used the post-test standard deviations in the calculations of effect size in this report,16 we provide detail on the pre-test measure to enable readers to use this detail to re-calculate them.
Pre-test measure (KS2 Reading fine score)
Three tables are shown. The first relates to the main analyses that involve all participants. The second and third are for the FSM and non-FSM subsamples respectively and relate to the subsample analyses for the primary outcome (GL New Group Reading Test).
Table AIll.1 Main analyses: KS2 Reading Score—all participants
Control Intervention All Respondents
Mean 24.3 24.7 24.5
Standard deviation 4.27 4.29 4.28
n= 192 199 391
Pooled standard deviation 4.28
Table AIll.2 Subsample analyses: KS2 Reading Score—FSM participants only
Control Intervention All Respondents
Mean 24.0 24.1 24.1
Standard deviation 3.90 4.11 3.98
16
This is so that the approach is consistent to the one used in the EEF Booktrust Summer Active Reading RCT evaluation report, which was also a literacy transition trial related to reading for pleasure.
Control Intervention All Respondents
n= 59 57 116
Pooled standard deviation 4.00
Table AllI.3 Subsample analyses: KS2 Reading Score—non-FSM participants only
Control Intervention All Respondents
Mean 24.5 24.9 24.7
Standard deviation 4.43 4.35 4.39
n= 133 142 275
Pooled standard deviation 4.39
Primary Outcome (GL New Group Reading Test)
The standard deviations shown in these three tables are used to calculate the hedges g statistics in this report. The first relates to the main analyses that involve all participants. The second and third are for the FSM and non-FSM subsamples respectively and relate to the subsample analyses for the primary outcome.
Table AllI.4 Main analyses: primary outcome measure—all participants
Control Intervention All Respondents
Mean 88.9 88.8 88.8
Standard deviation 10.88 11.55 11.21
n= 192 199 391
Pooled standard deviation* 11.23
Table AlIl.4 Subsample analyses: primary outcome measure—FSM participants only
Control Intervention All Respondents
Mean 85.9 88.0 86.9
Control Intervention All Respondents
n= 59 57 116
Pooled standard deviation 11.30
Table AllI.5 Subsample analyses: primary outcome measure—non-FSM participants only
Control Intervention All Respondents
Mean 90.2 89.1 89.6
Standard deviation 10.54 11.61 11.10
n= 133 142 275
Pooled standard deviation 11.11
Table AlIl.6 Secondary outcome 1 (PIRLS 2011 'Students Like Reading' scale)
(Just one table is shown for both secondary outcomes because no subsample analyses were conducted.)
Control Intervention All Respondents
Mean 14.4 14.6 14.5
Standard deviation 5.52 5.95 5.74
n= 158 173 331
Pooled standard deviation 5.75
As shown in Appendix I, this is a pre-validated scale that is a composite of eight questionnaire items. A list-wise missing values approach was adopted which led to a compounded proportion of 15% (n=60) of missing responses on this secondary outcome.
Table AlIl.7 Secondary outcome 2 (PIRLS 2011 'Students Motivated to Read' scale)
Control Intervention All Respondents
Mean 13.3 13.6 13.5
Standard deviation 3.80 3.69 3.75
n= 169 183 352
As shown in Appendix I, this is a pre-validated scale that is a composite of six questionnaire items. A list-wise missing values approach was adopted which led to a compounded proportion of 10% (n=39) of missing responses on this secondary outcome.
Effect Size Worked Example
Referring to Table 3.10 (Section 3.5, Outcomes and Analysis), the main effects model shows a coefficient of -0.72 for the intervention group with a standard error of 0.874.
The standard error can be used to calculate 95% confidence intervals for the coefficient: 95% Confidence Interval = Coefficient +/- (1.96 x standard error) = -0.72 +/- (1.96 x 0.874) Upper limit of confidence interval = -0.72 + 1.713 = + 0.988
Lower limit of confidence interval = -0.72 - 1.713 = -2.436
This tells us that, once random variation is taken into account, the coefficient cannot be confidently distinguished from zero.
The coefficients and confidence intervals can be converted into an effect size by dividing them by the standard deviation. For Cohen's d, this would be the standard deviation of the primary outcome for all participants (i.e. s=11.21) whilst for hedges g, the standard deviation would be the one that is pooled from the control and intervention group (s=11.23).
The pooled standard deviation is calculated using the following formula:
*
For the primary outcome, this is calculated to be s= 11.23
Converting the coefficient and 95% confidence interval limits into (Hedge's g) effect sizes:
Hedges g = coefficient / pooled standard deviation = -0.72 / 11.23 = - 0.06 Hedges g confidence interval upper limit = +0.988 / 11.23 = + 0.09
Hedges g confidence interval lower limit = - 2.433 / 11.23 = - 0.22 As reported in the 'Outcomes and Analysis' section:
Appendix lV Measuring MyChoice! Engagement
Details on the number of pupils per secondary school who placed an order for books on the MyChoice website were provided by Unitas. The proportion of pupils who placed an order ranged from 0% (in 4 schools) up to 100% (15 schools) with an average of 72% of pupils placing an order per school.
Secondary School Pupils placing book orders n= 29 Mean (sd) 72% (37 pp) Min 0% Max 100%
60