A cluster randomised controlled trial and evaluation and cost-effectiveness analysis of the Roots of Empathy schools-based programme for improving social and emotional well-being outcomes among 8- to 9-year-olds in Northern Ireland

(1)

This is a repository copy of

A cluster randomised controlled trial and evaluation and

cost-effectiveness analysis of the Roots of Empathy schools-based programme for

improving social and emotional well-being outcomes among 8- to 9-year-olds in Northern

Ireland

.

White Rose Research Online URL for this paper:

http://eprints.whiterose.ac.uk/129099/

Version: Published Version

Article:

Connolly, Paul, Miller, Sarah, Kee, Frank et al. (4 more authors) (2018) A cluster

randomised controlled trial and evaluation and cost-effectiveness analysis of the Roots of

Empathy schools-based programme for improving social and emotional well-being

outcomes among 8- to 9-year-olds in Northern Ireland. Public Health Research. ISSN

2050-4381

https://doi.org/10.3310/phr06040

[email protected] https://eprints.whiterose.ac.uk/ Reuse

Items deposited in White Rose Research Online are protected by copyright, with all rights reserved unless indicated otherwise. They may be downloaded and/or printed for private study, or other acts as permitted by national copyright laws. The publisher or other rights holders may allow further reproduction and re-use of the full text version. This is indicated by the licence information on the White Rose Research Online record for the item.

Takedown

If you consider content in White Rose Research Online to be in breach of UK law, please notify us by

(2)

PUBLIC HEALTH RESEARCH

VOLUME 6 ISSUE 4 MARCH 2018 ISSN 2050-4381

DOI 10.3310/phr06040

A cluster randomised controlled trial and evaluation

and cost-effectiveness analysis of the Roots of

Empathy schools-based programme for improving

social and emotional well-being outcomes among

8- to 9-year-olds in Northern Ireland

(3)

(4)

A cluster randomised controlled trial and

evaluation and cost-effectiveness analysis

of the Roots of Empathy schools-based

programme for improving social and

emotional well-being outcomes among

8- to 9-year-olds in Northern Ireland

Paul Connolly,

1 _{* Sarah Miller,}

1 _{Frank Kee,}

2 Seaneen Sloan,

1 _{Aideen Gildea,}

1 _{Emma McIntosh,}

3 Nicole Boyer

3 _{and Martin Bland}

4

1

_{Centre for Evidence and Social Innovation, Queen}

_’

_{s University Belfast, Belfast, UK}

2

_{Centre of Excellence for Public Health Research (Northern Ireland), Queen}

_’

_s

University Belfast, Belfast, UK

3

_{Institute of Health and Wellbeing, Public Health and Health Policy, University of}

Glasgow, Glasgow, UK

4

_{Department of Health Sciences, University of York, York, UK}

*Corresponding author

Declared competing interests of authors:Frank Kee reports that he chairs the National Institute for Health Research (NIHR) Public Health Research (PHR) programme Research Funding Board. Emma McIntosh reports that she is a member of the NIHR PHR programme Research Funding Board.

Published March 2018

DOI: 10.3310/phr06040

This report should be referenced as follows:

(5)

(6)

Public Health Research

ISSN 2050-4381 (Print)

ISSN 2050-439X (Online)

This journal is a member of and subscribes to the principles of the Committee on Publication Ethics (COPE) (www.publicationethics.org/).

Editorial contact: [email protected]

The full PHR archive is freely available to view online at www.journalslibrary.nihr.ac.uk/phr. Print-on-demand copies can be purchased from the report pages of the NIHR Journals Library website: www.journalslibrary.nihr.ac.uk

Criteria for inclusion in thePublic Health Researchjournal

Reports are published inPublic Health Research(PHR) if (1) they have resulted from work for the PHR programme, and (2) they are of a sufficiently high scientific quality as assessed by the reviewers and editors.

Reviews inPublic Health Researchare termed‘systematic’when the account of the search, appraisal and synthesis methods (to minimise biases and random errors) would, in theory, permit the replication of the review by others.

PHR programme

The Public Health Research (PHR) programme, part of the National Institute for Health Research (NIHR), evaluates public health interventions, providing new knowledge on the benefits, costs, acceptability and wider impacts of non-NHS interventions intended to improve the health of the public and reduce inequalities in health. The scope of the programme is multi-disciplinary and broad, covering a range of interventions that improve public health. The Public Health Research programme also complements the NIHR Health Technology Assessment programme which has a growing portfolio evaluating NHS public health interventions.

For more information about the PHR programme please visit the website: http://www.nets.nihr.ac.uk/programmes/phr

This report

The research reported in this issue of the journal was funded by the PHR programme as project number 10/3006/02. The contractual start date was in January 2012. The final report began editorial review in September 2016 and was accepted for publication in June 2017. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The PHR editors and production house have tried to ensure the accuracy of the authors’report and would like to thank the reviewers for their constructive comments on the final report document. However, they do not accept liability for damages or losses arising from material published in this report.

This report presents independent research funded by the National Institute for Health Research (NIHR). The views and opinions expressed by authors in this publication are those of the authors and do not necessarily reflect those of the NHS, the NIHR, NETSCC, the PHR programme or the Department of Health and Social Care. If there are verbatim quotations included in this publication the views and opinions expressed by the interviewees are those of the interviewees and do not necessarily reflect those of the authors, those of the NHS, the NIHR, NETSCC, the PHR programme or the Department of Health and Social Care.

© Queen’s Printer and Controller of HMSO 2018. This work was produced by Connollyet al.under the terms of a commissioning contract issued by the Secretary of State for Health and Social Care. This issue may be freely reproduced for the purposes of private research and study and extracts (or indeed, the full report) may be included in professional journals provided that suitable acknowledgement is made and the reproduction is not associated with any form of advertising. Applications for commercial reproduction should be addressed to: NIHR Journals Library, National Institute for Health Research, Evaluation, Trials and Studies Coordinating Centre, Alpha House, University of Southampton Science Park, Southampton SO16 7NS, UK.

(7)

Public Health Research

_{Editor-in-Chief}

Professor Martin White Director of Research and Programme Leader, UKCRC Centre for Diet and Activity

NIHR Journals Library Editor-in-Chief

Professor Tom Walley Director, NIHR Evaluation, Trials and Studies and Director of the EME Programme, UK

NIHR Journals Library Editors

Research (CEDAR), MRC Epidemiology Unit, Institute of Metabolic Science, School of Clinical Medicine, University of Cambridge; Visiting Professor, Newcastle University; and Director, NIHR Public Health Research Programme

Professor Ken Stein Chair of HTA and EME Editorial Board and Professor of Public Health, University of Exeter Medical School, UK

Professor Andrée Le May Chair of NIHR Journals Library Editorial Group (HS&DR, PGfAR, PHR journals)

Dr Martin Ashton-Key Consultant in Public Health Medicine/Consultant Advisor, NETSCC, UK

Professor Matthias Beck Professor of Management, Cork University Business School, Department of Management and Marketing, University College Cork, Ireland

Dr Tessa Crilly Director, Crystal Blue Consulting Ltd, UK

Dr Eugenia Cronin Senior Scientific Advisor, Wessex Institute, UK

Dr Peter Davidson Director of the NIHR Dissemination Centre, University of Southampton, UK

Ms Tara Lamont Scientific Advisor, NETSCC, UK

Dr Catriona McDaid Senior Research Fellow, York Trials Unit, Department of Health Sciences, University of York, UK

Professor William McGuire Professor of Child Health, Hull York Medical School, University of York, UK

Professor Geoffrey Meads Professor of Wellbeing Research, University of Winchester, UK

Professor John Norrie Chair in Medical Statistics, University of Edinburgh, UK

Professor John Powell Consultant Clinical Adviser, National Institute for Health and Care Excellence (NICE), UK

Professor James Raftery Professor of Health Technology Assessment, Wessex Institute, Faculty of Medicine, University of Southampton, UK

Dr Rob Riemsma Reviews Manager, Kleijnen Systematic Reviews Ltd, UK

Professor Helen Roberts Professor of Child Health Research, UCL Institute of Child Health, UK

Professor Jonathan Ross Professor of Sexual Health and HIV, University Hospital Birmingham, UK

Professor Helen Snooks Professor of Health Services Research, Institute of Life Science, College of Medicine, Swansea University, UK

Professor Jim Thornton Professor of Obstetrics and Gynaecology, Faculty of Medicine and Health Sciences, University of Nottingham, UK

Professor Martin Underwood Director, Warwick Clinical Trials Unit, Warwick Medical School, University of Warwick, UK

Please visit the website for a list of members of the NIHR Journals Library Board: www.journalslibrary.nihr.ac.uk/about/editors

(8)

Abstract

A cluster randomised controlled trial and evaluation and

cost-effectiveness analysis of the Roots of Empathy

schools-based programme for improving social and

emotional well-being outcomes among 8- to 9-year-olds

in Northern Ireland

Paul Connolly,

1

_{* Sarah Miller,}

1

_{Frank Kee,}

2

_{Seaneen Sloan,}

1

Aideen Gildea,

1

_{Emma McIntosh,}

3

_{Nicole Boyer}

3

_{and Martin Bland}

4

1_{Centre for Evidence and Social Innovation, Queen}_’_{s University Belfast, Belfast, UK}

2_{Centre of Excellence for Public Health Research (Northern Ireland), Queen}_’_{s University Belfast,}

Belfast, UK

3_{Institute of Health and Wellbeing, Public Health and Health Policy, University of Glasgow,}

Glasgow, UK

4_{Department of Health Sciences, University of York, York, UK}

*Corresponding author [email protected]

Background:There is growing consensus regarding the importance of attending to children’s social and emotional well-being. There is now a substantial evidence base demonstrating the links between a child’s early social and emotional development and a range of key longer-term education, social and health outcomes. Universal school-based interventions provide a significant opportunity for early intervention in this area and yet the existing evidence base, particularly in relation to their long-term effects, is limited.

Objectives and main outcomes:To determine the effectiveness and cost-effectiveness of Roots of Empathy (ROE), a universal school-based programme that, through attempting to enhance children’s empathy, seeks to achieve the following two main outcomes: improvement in prosocial behaviour and reduction in difficult behaviour.

Design:A cluster randomised controlled trial and an economic evaluation. A total of 74 primary schools were randomly assigned to deliver ROE or to join a waiting list control group. Seven schools withdrew post randomisation and a further two withdrew before the immediate post-test time point. Children (n=1278)

were measured pre test and immediately post test, and then for 3 years following the end of the programme. Data were also collected from teachers and parents.

Setting and participants:The intervention schools delivered ROE to their Year 5 children (aged 8–9 years) as a whole class.

Intervention:ROE is delivered on a whole-class basis for one academic year (October–June). It consists of 27 lessons based around the monthly visit from a baby and parent who are usually recruited from the local community. Children learn about the baby’s growth and development and are encouraged to generalise from this to develop empathy towards others.

Results:Although it was developed in Canada, the programme was very well received by schools,

parents and children, and it was delivered effectively with high fidelity. ROE was also found to be effective in achieving small improvements in children’s prosocial behaviour (Hedges’g=0.20;p=0.045) and

DOI: 10.3310/phr06040 PUBLIC HEALTH RESEARCH 2018 VOL. 6 NO. 4

(9)

reductions in their difficult behaviour (Hedges’g=_–0.16;p=0.060) immediately post test. Although the

gains in prosocial behaviour were not sustained after the immediately post-test time point, there was some tentative evidence that the effects associated with reductions in difficult behaviour may have remained up to 36 months from the end of the programme. These positive effects of ROE on children’s behaviour were not found to be associated with improvements in empathy or other social and emotional skills (such as emotional recognition and emotional regulation), on which the trial found no evidence of ROE having an effect. The study also found that ROE was likely to be cost-effective in line with national guidelines.

Conclusions:These findings are consistent with those of other evaluations of ROE and suggest that it is an effective and cost-effective programme that can be delivered appropriately and effectively in regions such as Northern Ireland. A number of issues for further consideration are raised regarding opportunities to enhance the role of parents; how a time-limited programme such as ROE can form part of a wider and progressive curriculum in schools to build on and sustain children’s social and emotional development; and the need to develop a better theory of change for how ROE works.

Trial registration:Current Controlled Trials ISRCTN07540423.

Funding:This project was funded by the National Institute for Health Research (NIHR) Public Health Research programme and will be published in full inPublic Health Research; Vol. 6, No. 4. See the NIHR Journals Library website for further project information.

(10)

List of tables

TABLE 1 Summary of meta-analyses of previous evaluations (n=7) of the

ROE programme 3

TABLE 2 Description of the outcomes, measures and reliability pre and post test 10

TABLE 3 Number of intervention and control schools in each of the participating health and social care trusts at pre test and immediately post test 14

TABLE 4 Data from ROE trial collected for the economic evaluation 18

TABLE 5 List of sensitivity analyses 22

TABLE 6 The ICER for cost-effectiveness analyses on SDQ 23

TABLE 7 Schools in the sample, by type, compared with the Northern Ireland

population in 2011/12 25

TABLE 8 Sample characteristics 26

TABLE 9 Comparison between control and intervention groups on

demographic characteristics 27

TABLE 10 Comparison between control and intervention groups on primary and

secondary outcome measures pre test 27

TABLE 11 Comparison of demographic characteristics between those who completed pre-test questionnaire only and those who completed pre- and

post-test questionnaires 28

TABLE 12 Comparison of outcome measures pre test between those who completed pre-test questionnaire only and those who completed pre- and

post-test questionnaires 28

TABLE 13 Summary of main effects immediately post test (T1) 29

TABLE 14 Comparison of effect sizes for the primary outcomes used in the main analysis (teacher-rated SDQ) with those measured by parent- and child-rated

SDQ and teacher-rated CBS 30

TABLE 15 Statistical significance of coefficients for interaction effects added to statistical models for child gender, family socioeconomic status and number of

siblings, respectively, immediately post test (T1) 31

TABLE 16 Comparison of ROE sample at baseline and post-test SDQ mean scores

with UK, Ireland and US national averages 33

TABLE 17 Duration (minutes) of PDMU delivered in class per week by intervention

and control teachers 33

(15)

TABLE 18 Summary of main effects at 12-month follow up (T2) 34

TABLE 19 Summary of main effects at 24-month follow-up (T3) 35

TABLE 20 Summary of main effects at 36-month follow-up (T4) 36

siblings, respectively, at 12-month follow-up (T2) 37

siblings, respectively, at 24-month follow-up (T3) 37

TABLE 23 Statistical significance of coefficients for interaction effects added to statistical models for gender, socioeconomic status and number of siblings,

respectively, at 36-month follow-up (T4) 38

TABLE 24 Comparison of main effects estimated with the observed data only

with the effects estimated using data sets with multiple imputation 39

TABLE 25 Variable descriptions and missing data percentages 40

TABLE 26 Outcome variables for cost 40

TABLE 27 Cost of the ROE intervention 42

TABLE 28 Unit costs of public sector service use 43

TABLE 29 Cost-effectiveness results 44

TABLE 30 Characteristics of previous quantitative evaluations included in the

meta-analysis 86

TABLE 31 Risk-of-bias summary: authors’judgements about each risk-of-bias

item for each included study 89

TABLE 32 Multilevel models fitted for the teacher-rated prosocial behaviour

(SDQ) variable at each time point 95

TABLE 33 Multilevel models fitted for the teacher-rated difficult behaviour

(SDQ) variable at each time point 96

TABLE 34 Multilevel models fitted for the child-rated‘reasons why a baby cries’

variable at each time point 97

TABLE 35 Multilevel models fitted for the child-rated‘ways to help a crying

baby’variable at each time point 98

TABLE 36 Multilevel models fitted for the child-rated emotional recognition

(16)

TABLE 37 Multilevel models fitted for the child-rated empathy variable at each

time point 100

TABLE 38 Multilevel models fitted for the child-rated emotional regulation

TABLE 39 Multilevel models fitted for the child-rated bullying (victim) variable at

each time point 102

TABLE 40 Multilevel models fitted for the child-rated bullying (bully) variable at

each time point 103

TABLE 41 Multilevel models fitted for the child-rated quality-of-life variable at

each time point 104

TABLE 42 Comparison of effect sizes immediately post test between the models used in the main analysis, these models extended with additional covariates and

then analysed with multiple imputation of missing data 105

(17)

(18)

List of figures

FIGURE 1 Logic model for the ROE programme 9

FIGURE 2 Flow diagram of recruitment and testing of children 13

FIGURE 3 Pattern of missing data in (a) costs and (b) QALYs 41

FIGURE 4 Cost-effectiveness plane representing 1000 bootstrapped cost and

QALY pairs 45

FIGURE 5 Cost-effectiveness acceptability curve showing probability of ROE

being cost-effectiveness compared with the control 45

FIGURE 6 Forest plot of SMDs between intervention and control for

prosocial behaviour 92

FIGURE 7 Forest plot of SMDs between intervention and control for

aggressive behaviour 92

FIGURE 8 Forest plot of SMDs between intervention and control for child-reported

empathy 94

FIGURE 9 Forest plot of SMDs between intervention and control for child-reported

emotional regulation 94

(19)

(20)

List of supplementary material

Report Supplementary Material 1 Research instruments used in study

Supplementary material can be found on the NIHR Journals Library report project page (www.journalslibrary.nihr.ac.uk/programmes/phr/10300602/#/documentation).

Supplementary material has been provided by the authors to support the report and any files provided at submission will have been seen by peer reviewers, but not extensively reviewed. Any supplementary material provided at a later stage in the process may not have been peer reviewed.

(21)

(22)

List of abbreviations

CAMS Child Anger Management Scale

CBS Child Behaviour Scale

CHU9D Child Health Utility–9D

CI confidence interval

EQ-5D-Y EuroQol-5 Dimensions youth

version

ES effect size

HSCT Health and Social Care Trust

ICC intracluster correlation coefficient

ICER incremental cost-effectiveness ratio

ISRCTN International Standard Randomised

Controlled Trial Number

NICE National Institute for Health and

Care Excellence

NIMDM Northern Ireland Multiple

Deprivation Measure

PDMU Personal Development and Mutual

Understanding

QALY quality-adjusted life-year

ROE Roots of Empathy

SD standard deviation

SDQ Strengths and Difficulties

Questionnaire

SEL social and emotional learning

SMD standardised mean difference

T time

(23)

(24)

Plain English summary

C

hildren’s early social and emotional development remains a significant predictor of their future social, education and health outcomes. Roots of Empathy (ROE) is a school-based programme that is delivered on a whole-class basis for one academic year. It seeks to increase children’s empathy, leading to positive behaviour change and preparing them better for later life. It consists of 27 lessons based around the monthly visit from an infant and parent, who are usually recruited from the local community. This study provides a robust evaluation of the effectiveness and cost-effectiveness (value for money) of ROE. ROE involved 74 primary schools in Northern Ireland randomly divided into two equal groups that either delivered ROE during 2011–12 or acted as a control group. The effectiveness of ROE was measured immediately at the end of the year and for the following 3 years (up to 2015). The study found that ROE was effective in increasing the children’s prosocial behaviour and reducing their difficult behaviour immediately at the end of the programme. Although the effects on prosocial behaviour fell away after the first year, there was some possible evidence that the effects on reducing difficult behaviour may have been sustained for the following 3 years. Although originally developed in Canada, ROE was very well received by teachers, parents and children, and it was effectively delivered in schools in the context of Northern Ireland. The study also found that ROE was likely to be a cost-effective use of society’s resources as a means of improving children’s quality of life.

(25)

(26)

Scientific summary

Background

Children’s early social and emotional development remains a significant predictor of future social, education and health outcomes, and there is substantial evidence linking early social and emotional development to later academic performance and a number of key health outcomes. The recent Marmot Review in England (Marmot M.Fair Society, Healthy Lives: The Marmot Review. Strategic Review of Health Inequalities in England Post-2010. Executive Summary. London: Department of Health; 2010) identified the policy objective of giving every child the best start in life as its‘highest policy recommendation’(p. 14), placing particular emphasis on reducing inequalities in the early development of physical, cognitive and non-cognitive skills. Among some of the key recommendations is the need to prioritise developing the capacity of schools to address and improve children’s‘social and emotional development, physical and mental health and well-being’(p. 18).

A substantial body of evidence now exists that suggests that well-designed school-based prevention programmes can be effective in improving a variety of social, health and academic outcomes. Roots of Empathy (ROE) is a universal school-based social and emotional learning (SEL) programme that has been developed and implemented in Canada, and has only recently been introduced into the UK. It is delivered on a whole-class basis for one academic year and consists of 27 lessons, which are based around a monthly classroom visit from an infant and volunteer parent (typically the mother) who are usually recruited from the local community. Children learn about the baby’s growth and development through interactions and observations with the baby during these monthly visits. ROE is a mentalisation-based programme that aims to develop empathy in children. The labelling of feelings and the exploration of the relationship between feelings and behaviour is achieved through the mother–infant interaction as observed by the children in the classroom.

Several evaluations of ROE have been conducted to date and this report synthesises the findings from these. Of seven eligible studies, only one was a (cluster) randomised controlled trial. The pooled data from these studies suggest that ROE is effective in leading to small improvements in prosocial behaviour [standardised mean difference (SMD) 0.13] and reductions in aggressive behaviour (SMD–0.18). There is no evidence to suggest that it is effective in improving other SEL outcomes among children, in this case empathy and emotional regulation. Only one evaluation studied the longer-term impact of the programme, suggesting that after 3 years the intervention group had poorer prosocial behaviour than the control group [SMD–0.12, 95% confidence interval (CI)–0.17 to–0.07]. With respect to aggressive behaviour 3 years post intervention, the intervention group was displaying only slightly less aggressive behaviour than the control group (SMD–0.06, 95% CI–0.09 to–0.03) and, although statistically significant, this effect was greatly reduced from that observed immediately post test (SMD–0.25).

Objectives

Given the limited existing evidence base for ROE, the aims of the current evaluation are to:

l evaluate the immediate and longer-term impacts of ROE on social and emotional well-being outcomes among 8- to 9-year-old pupils

l evaluate the cost-effectiveness of the programme.

(27)

The purpose of the research is to answer the following research questions.

1. What is the impact of the programme post test, and up to 3 years following the end of the programme, on a number of specific social and emotional well-being outcomes for participating children?

2. Does the programme have a differential impact on children depending on their gender, the number of siblings they have and their socioeconomic status and/or the socioeconomic profile of the school? 3. Does the impact of the programme differ significantly according to variations in implementation

fidelity found?

4. What is the cost-effectiveness of the programme in reducing cases of aggressive behaviour and increasing prosocial behaviour among school-aged children?

Methods

This study consisted of a cluster randomised controlled trial, a qualitative process evaluation and a cost-effectiveness evaluation.

Sample

Seventy-four primary schools from four of the five trust areas in Northern Ireland were recruited to the trial between March and June 2011. All primary schools and their Year 5 cohort were eligible to take part in the study. Schools were randomly assigned to each of the intervention (n=37) and control (n=37)

groups. The intervention schools received the ROE programme in their selected Year 5 class for one academic year (2011/12). The remaining schools in the waiting list control group continued with the regular curriculum and usual classroom activity.

Outcomes and measures

The primary child outcomes are increases in prosocial behaviour and decreases in difficult behaviour as measured by the teacher-rated version of the Strengths and Difficulties Questionnaire (SDQ). Additional data from alternative sources (parent- and child-rated SDQ) and alternative measures (teacher-rated Child Behaviour Scale) were collected in order to triangulate the data. Secondary outcomes included understanding of infant feelings (Infant Facial Expression of Emotions Scale), recognition of emotions (Emotion Recognition Questionnaire), empathy (Interpersonal Reactivity Index), emotional regulation (Child Anger Management Scale), bullying (Revised Olweus Bully/Victim Scale) and quality of life [Child Health Utility–9D (CHU9D)]. The additional information was the parents’home postcode, the number/age of any siblings the child had, the parents’highest level of qualification and the parents’occupation.

Data collection

Initial pre-test data from the children, parents and teachers were collected in October 2011. The first post-test data were collected in June 2012 and data were collected again at 12, 24 and 36 months. Teachers were asked to complete a questionnaire for each participating child at each time point. Parents were contacted by post and asked to complete a questionnaire and return it to the research team in a Freepost envelope. Field workers administered questionnaires to the children on a whole-class basis.

Seven schools withdrew from the study before the start of the trial; however, retention rates were good overall, with 1182 pupils tested pre test and 902 (76.3%) remaining in the study at the final 3-year follow-up.

Cost-effectiveness analysis

The economic evaluation aimed to conduct:

1. a cost–utility analysis comparing the costs and utilities of the two groups over a 3.75-year period 2. a cost-effectiveness analysis comparing costs and effects between groups such as decreases in difficult

behaviour and increases in prosocial behaviour as measured by the SDQ.

(28)

The base-case analysis compared the ROE intervention group with the usual classroom activities control group in terms of (1) costs incurred over the 3.75-year period and (2) quality-adjusted life-years (QALYs) gained over the 3.75-year period. Data were collected at five time points: pre test, post test, 12-month follow-up, 24-month follow-up and 36-month follow-up. A cost–utility analysis was undertaken, in which costs considered from a public sector perspective (2014 GBP) and health outcomes were measured by QALYs. Health utilities were measured using the CHU9D. All of the analyses were performed on individual patient-level data, taking clustering into account, and collected from the ROE trial.

Resource use was measured over the length of the trial and made up of the following data collection: (1) resource use resulting from the delivery of the intervention, (2) NHS resource use and (3) societal costs.

Process evaluation

A qualitative process evaluation was conducted alongside the trial to provide in-depth qualitative data on both the implementation and outcomes of the ROE programme. The delivery process of the programme was monitored and tracked across all schools, and a more detailed inquiry of underlying broad patterns outlined from across the schools was the focus of an in-depth case study approach conducted in six of the intervention schools. Interviews and focus groups were carried out with school personnel, local programme co-ordinators, volunteer mothers, children and parents. Observational classroom data were also collected.

Results

Immediately post test

After controlling for pre-test scores and clustering, children who participated in the ROE programme were rated by their teachers as more prosocial (effect size,g= +0.20;p=0.045) and as exhibiting less difficult

behaviour (g=_–0.16;p=0.06) than those in the control group.

With regard to the secondary outcomes, children who participated in the ROE programme were able to report a greater number of reasons why infants cry (effect sizeg= +0.24;p=0.01). It is important

to note, however, that part of the intervention involves explicitly teaching children about how infants communicate and why they cry, and it is conceivable, therefore, that this measure is biased in favour of the intervention group. Furthermore, the effect is small. No evidence of any differences between the groups was found in relation to the other secondary outcomes.

Prespecified subgroup analyses were undertaken to explore whether or not the programme worked better according to gender, socioeconomic background and number of siblings. No clear or consistent pattern emerged to suggest that there are underlying differential effects. The programme was found to have been uniformly delivered with high fidelity across all intervention schools. It was, therefore, not possible to assess the potential moderating effects of varying levels of fidelity on the outcomes achieved.

Effects at 12-, 24- and 36-month follow-up

The initially positive effects on prosocial behaviour were found to disappear at all subsequent time points. Moreover, there were no statistically significant differences in scores between those in the intervention group and those in the control group at any of the subsequent follow-up time points for any of the other outcome variables (at 12, 24 or 36 months post intervention). There also remained no clear or convincing pattern of any subgroup effects at any of these subsequent time points.

However, and in relation to total difficulties (as measured by the teacher-rated SDQ), the effect size

immediately post test appears to have been maintained at the 12-month (g=_–0.14), 24-month (g=_–0.13)

and 36-month (g=–0.14) follow-up time points. However, and because of the reduction in sample size

owing to attrition, this effect is not statistically significant and so it needs to be treated with a degree of caution.

(29)

Sensitivity analysis

Multiple imputation was used to test whether or not attrition introduced any bias into the findings;

however, the findings using the imputed data sets are broadly similar to those using just the observed data.

Cost-effectiveness analysis

Overall, it is estimated that the average cost of delivering ROE is £4057 per school and £175 per pupil. The incremental cost of delivering ROE was £153 (95% CI £14 to £292). The incremental QALY gain from ROE was 0.0160 (95% CI–0.0143 to 0.0462). Against generally accepted national guidelines, the findings of this present study suggest that ROE is a cost-effective intervention. In particular, the National Institute for Health and Care Excellence suggests that interventions costing the NHS<£20,000 per QALY gained

are cost-effective. It also suggests that those costing between £20,000 and £30,000 per QALY gained may be cost-effective. For the present evaluation, the incremental cost-effectiveness ratio was £9571 per

QALY gained (95% CI–£87,776 to £106,676). It was found that ROE had an 83.1% chance of being

cost-effective at the £20,000 per QALY threshold and a 90.1% chance at the higher threshold of £30,000 per QALY.

Process evaluation

The ROE programme was delivered with high fidelity, with all lessons being delivered in all of the intervention schools. This was seen as being the result of the clearly defined structure of the programme and the strong training and ongoing support provided to ROE instructors in schools. The programme was also very well received overall and was felt to include good resources and be linked in closely with the Northern Ireland curriculum, particularly the element on personal development and mutual understanding. Five key issues emerged from the qualitative process evaluation:

1. A belief among some that it would be beneficial for ROE instructors to be teachers within the school to facilitate stronger communication and planning between the instructor and the class teacher.

2. A perception that the delivery of the programme in the first year may have been a little more

challenging, especially for those schools where the ROE instructor was not a teacher within that school. 3. A concern regarding the resources required to deliver the ROE programme, especially if the ROE

instructor is to be one of the teachers within the school, and whether or not it is sustainable in the longer term.

4. A concern that the ROE programme lasts for only 1 year and is not followed up in subsequent years. Additionally, and relatedly, there was a view among some that it would be worthwhile building the key knowledge and skills among children at an earlier age and before the ROE programme, with some mentioning the Seeds of Empathy programme.

5. The relative lack of involvement of or engagement with parents around the programme and how this may have been partly restricted because of the emphasis on maintaining fidelity to the existing programme.

Conclusions

First, this trial has provided strong and robust evidence that ROE did have a positive impact on children’s behaviours in the directions expected immediately post test. More specifically, there is evidence that the programme enhanced children’s prosocial behaviour and some evidence that it reduced difficult behaviour, above and beyond the typical effects associated with attending school.

Second, the trial has also provided clear evidence that, although ROE was originally developed in Canada, it is possible to deliver it extremely effectively and with fidelity in a different country and a different cultural context, in this case Northern Ireland.

Third, the trial found no evidence to support the hypothesised theory of change. It is not possible to conclude with certainty how ROE achieved positive behavioural effects without associated increases in social and emotional outcomes. However, it is clear from the qualitative process evaluation that the

(30)

children enjoyed the ROE lessons and that the lessons did, progressively, help to encourage the development of a collective sense of concern and caring for the baby, which may have resulted in a positive shift in the group norms (i.e. class norms) of prosocial behaviour. Peer groups play an important and influential role in the development of children’s behaviour and attitudes, and they are an important social context within which individual development takes place. Further research is required to explore this possible explanation for behavioural change.

Fourth, the current ROE programme provides only limited opportunities to engage with parents. However, and as found through the process evaluation, there is significant interest among teachers and some parents for a greater degree of parental involvement in the programme. It is, therefore, recommended that consideration be given to incorporating greater parental involvement in the future.

Finally, the findings are not so positive in relation to the sustainability of initial gains in prosocial behaviour. In this respect, further work would be beneficial in terms of developing a more holistic and progressive curriculum that seeks to use evidence-based programmes such as ROE but in a way that is able to sustain and build on the short-term gains found in a developmentally appropriate way.

Trial registration

This trial is registered as ISRCTN07540423.

Funding

Funding for this study was provided by the Public Health Research programme of the National Institute for Health Research.

(31)

(32)

Chapter 1 Introduction

T

his report presents the findings from a cluster randomised controlled trial evaluation of the Roots of Empathy (ROE) programme. This chapter provides the background for the study and a description of the programme. The methodology for the evaluation is outlined inChapter 2. The quantitative findings from the trial regarding the impact of the programme on pupil outcomes and the cost-effective analysis are reported inChapter 3and the findings from the accompanying qualitative process study are set out in

Chapter 4. Key issues emerging from the findings are set out inChapter 5.

Rationale for current study

There is a growing consensus in academic and policy circles regarding the importance of attending to young children’s social and emotional well-being. There is substantial evidence that links early social and emotional development to later academic performance1_{and a number of key health outcomes, such as stress} and mental health.2_{Deficits in basic skills, such as the ability to identify emotions, tend to have wide-ranging} implications, including being rejected by others and excluded from peer activities and being victimised.3_Such deficits are also related to lower peer-rated popularity and teacher-rated social competence.4,5_Chronic physical aggression during primary school also increases the risk of violence and delinquency throughout adolescence in boys.6,7_{In turn, this can lead to destructive forms of emotion management, such as} alcohol abuse.

In recognition of this, a comprehensive set of public health guidelines was published by the National Institute for Health and Care Excellence (NICE) in 2008, the aim of which was to encourage the promotion of social and emotional well-being in primary school children.8_{According to the guidelines, child well-being} not only is important in its own right but can also be a determinant of success in school and physical health. The guidelines recommend that schools create an ethos that supports positive behaviours for learning and successful relationships, and also provide an emotionally secure and safe environment that protects against bullying and violence and offers teachers and practitioners the support they need in developing children’s social and emotional well-being.

However, perhaps the most significant recent development has been the publication of the Marmot Review in England.9_{At the heart of the review}_’_{s key recommendations is the policy objective of giving every child} the best start in life. Of the six policy objectives identified by the review, this one was held up as its‘highest policy recommendation’and reflected the review’s life course perspective. Alongside a call to increase the proportion of overall expenditure allocated to the early years, the review also placed an emphasis on reducing inequalities in the early development of physical and emotional health and cognitive, linguistic and social skills, and thus building resilience and well-being among young children. This should be done, according to the Marmot Review, through investment in‘high quality maternity services, parenting programmes, childcare and early years education to meet need across the social gradient’(p. 16).9 A second, linked, policy objective identified by the review is to enable all children, young people and adults to maximise their capabilities and have control over their lives. This, in turn, should be achieved by ensuring that schools, families and communities work in partnership to improve health, well-being and resilience. Among some of the key recommendations made in this regard is the need to prioritise developing the capacity of schools to address and improve children’s‘social and emotional development, physical and mental health and well-being’(p. 18).9

Most recently, a report commissioned by the Early Intervention Foundation in the UK (March 2015) explored the relationship between social and emotional skills in childhood and long-term effects into adulthood.10_{The authors found that self-control and self-regulation are the most important childhood} social and emotional skills in relation to positive adult outcomes. Similarly, they found that self-perception,

(33)

self-awareness and social skills were important influences on many adult outcomes. There was no clear evidence linking motivation or resilience to adult outcomes; however, emotional well-being in childhood was found to be important for mental well-being as an adult. An equally important finding from this report was that social and emotional development is just as important as cognitive development, if not more so, in some respects, for future life.

Scientific background

A substantial body of evidence now exists to suggest that well-designed school-based prevention programmes can be effective in improving a variety of social, health and academic outcomes for children and young people.11,12_{Several reviews have been conducted in the area of social and emotional learning} (SEL) programmes and, although the types of intervention, participants and outcomes have varied between reviews, the consensus is that well-designed universal school-based programmes have a positive impact on child outcomes.13–16

The most relevant and recent of these reviews is Durlaket al.’s17_{meta-analysis, which focused exclusively} on school-based universal SEL programmes and their impact on a number of pupil outcomes, including SEL skills, attitudes, positive social behaviour, conduct problems, emotional distress and academic performance. The analysis comprised 213 programmes and 270,034 pupils. The mean effect sizes for each outcome ranged from–0.22 (conduct problems) to 0.57 (SEL skills), which, the authors note, is consistent with effect sizes reported by other studies and reviews of similar programmes and outcomes. The most effective SEL programmes in this review (defined as those that had a significant and positive impacted on all six outcomes) were those that did not experience implementation problems and, consistent with Paytonet al.’s14 conclusions,14_{also incorporated the following four recommended practices commonly referred to as}

‘SAFE’:

l Sequenced–applying a planned set of activities to develop skills in a step-by-step fashion

l Active–using active forms of learning (i.e. role plays, behavioural rehearsal with feedback)

l Focused–devoting sufficient time to developing social and emotional skills

l Explicit–targeting specific social and emotional skills.

Durlaket al.17_{concluded that SEL programmes tended to have a significant and positive impact on}

students’social and emotional competence, increase prosocial behaviour, reduce conduct and internalising problems, and improve academic performance. They also reported that in those studies that followed up participants, these effects remained statistically significant for at least 6 months post intervention.

However, only a small number of studies in this review (15%) reported follow-up data that met the inclusion criteria, and so little is known about the long-term effects of SEL programmes. Adiet al.,18 whose review informed the NICE guidelines, reinforced this view and observed that, although programmes teaching social skills and emotional literacy show promise, there remains a need for good-quality trials to assess these programmes’long-term effectiveness.

More recently, the Early Intervention Foundation’s second of three reports on social and emotional skills in childhood focused on what works in the UK to promote such skills in childhood and adolescence.19 The authors found that school-based and targeted programmes were most effective, along with those interventions that adopted a‘whole school’approach, involving staff, parents and the wider community. The evidence of the effectiveness of UK out-of-school programmes was less clear-cut.

Existing impact evaluations of the Roots of Empathy programme

The ROE website (www.rootsofempathy.org) reports a number of evaluations that have been conducted to date. To provide further context for this present study, an attempt has been made to identify these existing

(34)

studies and to synthesise the data. Full details of the studies identified and of the methods used for the meta-analysis are provided inAppendix 1.

In total, 10 studies20–29_{were identified that had reports that provided sufficient information to assess their} eligibility for the meta-analysis. A further six studies were referenced but it was not possible to locate the full report or the data for these. Although the authors of these were contacted directly, the research team had not received a response at the time of writing. Of the 10 reports found, three were excluded because they did not meet the eligibility criteria (i.e. employed an experimental or quasi-experimental design, quantitatively measured at least teacher-rated prosocial and aggressive/difficult behaviour, and collected outcome data at both pre and post test).

Of the seven eligible studies,20–26_{only one}20_{was a (cluster) randomised controlled trial that also tracked} children for 3 years following the end of the programme. The remaining six studies employed

quasi-experimental designs with pre- and immediately post-test data only. Four of the studies,20–23 including the cluster randomised controlled trial, were conducted in Canada, two were conducted in Scotland24,25_{and one was conducted in Australia.}26

A total of 4140 primary school aged children from 145 schools took part in the seven eligible studies.20–26 Sample sizes ranged between 132 and 785 children, with an average sample size of 591. All of the

evaluations measured teacher-rated prosocial and aggressive behaviour using valid and reliable instruments. A range of other teacher and child rated outcomes were also measured, but this meta-analysis focuses only on synthesising the effects for the most commonly measured outcomes:

l teacher-ratedprosocial behaviourimmediately post test (all seven previous studies)

l teacher-ratedaggressive behaviourimmediately post test (all seven previous studies)

l child-reportedempathyimmediately post test (five studies)

l child-reportedemotional regulationimmediately post test (two studies).

Full details of the studies included and excluded, and also of the methods used for the meta-analysis, are provided inAppendix 1. The findings are summarised inTable 1. As can be seen, when the available data from the seven studies are pooled there is evidence that ROE is effective in leading to small improvements in prosocial behaviour [standardised mean difference (SMD) 0.13] and reductions in aggressive behaviour (SMD–0.18). However, and interestingly, there is no evidence to suggest that it is effective in improving other SEL outcomes among children, in this case empathy and emotional regulation.

As noted, only one evaluation20_{studied the longer-term impact of the programme. This is the only} pre-existing randomised controlled trial for which there are data, and it suggests that after 3 years the intervention group hadpoorerprosocial behaviour than the control group [SMD–0.12, 95% confidence interval (CI)–0.17 to–0.07]. With respect to aggressive behaviour 3 years post intervention, the intervention group were displaying only slightly less aggressive behaviour than the control group (SMD–0.06, 95%

–0.09 to–0.03) and, although statistically significant, this effect was much reduced compared with that observed immediately post test (SMD–0.25).

TABLE 1 Summary of meta-analyses of previous evaluations (n=7) of the ROE programme

Outcome Pooled sample Pooled SMD (95% CI)

Teacher-rated prosocial behaviour 1895 ROE, 1617 control (seven studies) 0.13 (0.06 to 0.19)

Teacher-rated aggressive behaviour 1897 ROE, 1626 control (seven studies) –0.18 (–0.33 to–0.03)

Child-reported empathy 1186 ROE, 861 control (five studies) 0.10 (–0.05 to 0.25)

Child-reported emotional regulation 699 ROE, 655 control (two studies) 0.03 (–0.08 to 0.14)

(35)

Overall, therefore, although the findings from existing evaluations of ROE are promising, they raise interesting questions regarding the apparently mixed effects of the programme immediately post test and also about whether or not such effects are sustained in the longer term. Moreover, the current evidence base is limited to only one randomised trial that is also the only study to date that has considered the longer-term effects of the programme. In addition, no study to date has included a cost-effectiveness analysis. This, then, provides the rationale for the present evaluation.

Objectives

The aims of the current evaluation are to:

l evaluate the immediate and longer-term impact of the ROE programme on social and emotional

well-being outcomes among pupils aged 8–9 years

l evaluate the cost-effectiveness of the programme.

The purpose of the research is to answer the following research questions.

1. What is the impact of the programme post test and up to 3 years following the end of the programme on a number of specific social and emotional well-being outcomes for participating children?

2. Does the programme have a differential impact on children depending on their gender, the number of siblings they have and their socioeconomic status and/or the socioeconomic profile of the school? 3. Does the impact of the programme differ significantly according to variations in implementation

fidelity found?

4. What is the cost-effectiveness of the programme in reducing cases of aggressive behaviour and increasing prosocial behaviour among school-aged children?

The full protocol for this trial, published in August 2011 before ethics approval was sought for the study and, thus, before the recruitment of schools and pre-testing, can be found at the National Institute for Health Research Evaluation, Trials and Studies website.30

A cluster randomised controlled trial and evaluation and cost-effectiveness analysis of the Roots of Empathy schools-based programme for improving social and emotional well-being outcomes among 8- to 9-year-olds in Northern Ireland

This is a repository copy of

A cluster randomised controlled trial and evaluation and

cost-effectiveness analysis of the Roots of Empathy schools-based programme for

improving social and emotional well-being outcomes among 8- to 9-year-olds in Northern

Ireland

.

White Rose Research Online URL for this paper:

http://eprints.whiterose.ac.uk/129099/

Version: Published Version

Article:

Connolly, Paul, Miller, Sarah, Kee, Frank et al. (4 more authors) (2018) A cluster

randomised controlled trial and evaluation and cost-effectiveness analysis of the Roots of

Empathy schools-based programme for improving social and emotional well-being

outcomes among 8- to 9-year-olds in Northern Ireland. Public Health Research. ISSN

2050-4381

https://doi.org/10.3310/phr06040

PUBLIC HEALTH RESEARCH

A cluster randomised controlled trial and evaluation

and cost-effectiveness analysis of the Roots of

Empathy schools-based programme for improving

social and emotional well-being outcomes among

8- to 9-year-olds in Northern Ireland

A cluster randomised controlled trial and

evaluation and cost-effectiveness analysis

of the Roots of Empathy schools-based

programme for improving social and

emotional well-being outcomes among

8- to 9-year-olds in Northern Ireland

Paul Connolly,

1

* Sarah Miller,

1

Frank Kee,

2

Seaneen Sloan,

1

Aideen Gildea,

1

Emma McIntosh,

3

Nicole Boyer

3

and Martin Bland

4

Centre for Evidence and Social Innovation, Queen

’

s University Belfast, Belfast, UK

Centre of Excellence for Public Health Research (Northern Ireland), Queen

’

s

University Belfast, Belfast, UK

Institute of Health and Wellbeing, Public Health and Health Policy, University of

Glasgow, Glasgow, UK

Department of Health Sciences, University of York, York, UK

Published March 2018

Public Health Research

Public Health Research

Editor-in-Chief

NIHR Journals Library Editor-in-Chief

NIHR Journals Library Editors

Abstract

A cluster randomised controlled trial and evaluation and

cost-effectiveness analysis of the Roots of Empathy

schools-based programme for improving social and

emotional well-being outcomes among 8- to 9-year-olds

in Northern Ireland

Paul Connolly,

* Sarah Miller,

Frank Kee,

Seaneen Sloan,

Aideen Gildea,

Emma McIntosh,

Nicole Boyer

and Martin Bland

Contents

List of tables

List of figures

List of supplementary material

List of abbreviations

_{* Sarah Miller,}

_{Frank Kee,}

_{Aideen Gildea,}

_{Emma McIntosh,}

_{and Martin Bland}

_{Centre for Evidence and Social Innovation, Queen}

_’

_{s University Belfast, Belfast, UK}

_{Centre of Excellence for Public Health Research (Northern Ireland), Queen}

_’

_s

_{Institute of Health and Wellbeing, Public Health and Health Policy, University of}

_{Department of Health Sciences, University of York, York, UK}

_{Editor-in-Chief}

_{* Sarah Miller,}

_{Frank Kee,}

_{Seaneen Sloan,}

_{Emma McIntosh,}

_{Nicole Boyer}

_{and Martin Bland}