Factors that could impact outcomes
The overall results of the impact evaluation suggest that Maths Counts had a positive effect on the General Maths outcome for KS2 children (see Table 11). Indicative results suggest it was more effective for the weakest Year 3 pupils (see Table 19), but not for the older cohorts and those who were ever eligible for free school meals in the last six years. It should be noted that these are based on smaller sample sizes than the total cohort and will be less secure. Also, baseline data on the youngest cohort (Year 3) was different to the baseline data collected for the older cohorts (Years 4, 5, and 6) which also makes comparisons between younger and older cohorts difficult.
One possible reason for the difference between years could be that the programme was designed to address very basic maths problems and targets the very low performing children. The selection criteria for eligibility were specifically those who were not likely to achieve the Year 2 programme of study. Some schools may not have many of these children and selected the slightly higher performers. As the programme is meant to address very low-level or basic maths problems, it may be seen by the higher attainers as patronising. This could have had an adverse effect on the motivation of the better able pupils. This was also the professional view of the developer and the teaching staff (see Appendix F, feedback from schools). A slightly higher proportion of EverFSM6 children (54%) were from the older age cohort, which may explain the slightly suppressed results for the older cohort.
The larger effects for the Year 3 could be due to the fact that the KS1 scores were defined in broad descriptive categorical variables whereas those for the older children (Years 4, 5, and 6) were recorded in numerical point scores. However, because the post-scores are independent of this, the gain scores are unlikely to be an artefact of the change in KS1 recording.
The process evaluation suggests a number of other factors that may have impacted on the outcomes, and some lessons that can be learnt from this trial. One factor was pupil absences; this impacted on the optimal delivery of the programme and led to about 18% of the treatment pupils (n = 56) having fewer than the recommended minimum number of sessions. The CACE analysis shows that compliance
to the minimum number of sessions is associated with greater impact. Strong leadership support is needed to ensure this.
Some children also displayed other learning and social-emotional and behavioural difficulties which could not be addressed by the programme. The recent EEF/Royal Society review suggests literacy as an important mediating factor in learning science (Nunes et al., 2017). Poor literacy skills and dyslexia can affect children’s learning. Research evidence suggests that there is a close link between low income, mental health, and attainment at school (Bradley and Green, 2013; Gutman and Vorhaus, 2012; PHE, 2014). The problem with a number of these children may not be simply lack of understanding of mathematical concepts. So while the programme is successful in supporting those who felt lost in a whole-class environment, it may be less successful in helping those who have more complex needs. Feedback from teaching staff suggests that the programme is less suited to those with wider and more complex learning difficulties (see Appendix E). Perhaps future trials may also want to consider how these children can be supported in other ways to help them access the curriculum. Enhancing the general well-being of children may be a precursor to effective learning (The Public Health England Report, 2016; Weale, 2017).
A small number of children were observed not being able to read mathematical symbols This suggests that a further area for development may need to be around understanding of basic mathematic symbols. Staff absences and staff turnover have also affected optimal delivery. This can be disruptive to some children who take time to develop a rapport with adults. Staff absences and turnover also meant that children missed lessons and did not have the continuity that may have been beneficial.
Perceived impact on pupils’ wider outcomes
Overall, staff and pupils were very positive about the programme. Pupils particularly liked the one-to- one individual attention and many of the activities. They found the pace of the lessons met their needs and this gave them the confidence to learn maths. The learning environment was supportive and unthreatening. Children felt that if they did not understand a concept they could always ask the LP whereas in a whole-class environment they could not and often felt lost. Teaching staff liked the structured protocol starting with the diagnostics, which helped to identify the individual needs of the child. The lesson activities and the resources are all readily available, which saved time having to think of interesting things to do with the children. All the staff we spoke to commented on how much more confident children had become. The initial anxiety about maths had been slowly eroded. There was evidence of this, not only during the sessions, but also in the classroom. We also observed and heard about changes in pupils’ attitude towards learning maths. They were more willing to have a go, to make mistakes. Even if test score gains are limited, this perceived improvement in confidence is a positive first step towards learning maths and overcoming maths anxiety for the children involved. However, it should be noted that there was no empirical evidence that the programme had an impact on attitudes to maths as measured by a standardised test.
Empowerment of teaching assistants
The programme also had an impact on teaching assistants. Maths Counts differs from its predecessors (for example, Numbers Count) in that it was conducted by teaching assistants rather than classroom teachers. The trial has shown that teaching assistants can be effectively deployed to deliver the programme. Some of the teaching assistants reported that they had learnt a lot about how to utilise the various common, but rarely used, resources, such as the Numicon and Dienes. Two teaching assistants have since conducted a workshop for other teaching assistants in their school on the numerous ways
Education Endowment Foundation 54
shown that, if properly trained, teaching assistants can be effectively deployed to support children’s learning. This is consistent with the EEF guidance report on the use of teaching assistants (Sharples et al., 2015).