9
Chapter 9 – Design & Methods (1)
Design of the Evaluation of the School Improvement Programme a. Introduction
In three chapters, section four presents issues relating to the design of the research. Chapter 10 discusses issues arising from the pilot programme. Chapter 11 discusses ethical considerations for the design and delivery of the programme. This chapter explains the design of the programme evaluation and the methods adopted for data collection and analysis. Issues relating to the implementation of the evaluation are discussed in the relevant results chapters.
b. Overview of the evaluation design
Aim of the evaluation
Both the design of the programme and the evaluation were informed by aims 4, 5 and 6 of this study, namely:
4 To obtain and assess evidence showing whether students’ performance had
improved by the end of the school improvement CPDL programme and at follow-up. 5 To identify problems and possibilities associated with well-motivated but largely
untrained teachers delivering school improvement programmes, and collecting data for an evaluation, with only arms-length supervision from the Country Director. & 6 To throw light on aspects of the programme that the teachers, pupils and other
stakeholders experienced as strengths and weaknesses of the programme.
Design of the Evaluation
A quasi-experimental design was the most suitable structure for an evaluation in the circumstances. The impact evaluation required data to be collected before and after the intervention in both the schools receiving the intervention and the comparison schools. Process evaluation data were collected throughout the period and at the end of the intervention.
1 Baseline data to be collected before the CPDL started (Time 1) and end-line data to be collected on completion of the programme at the 12-month point (Time 2). These data consisted largely of literacy assessments, attendance and lesson observations.
2 A comparison group of schools with similar student intakes and teachers who were not taking part in the CPDL, would have the same data collection.
3 The distinction between impact data and process data was explicit: Impact data were provided by the assessments at baseline (Time 1) and at 12 months (Time 2). With these data, it was possible to compare the performance of students at participating schools with the performance of pupils at non-participating schools.
Process data were collected during the follow-up component of the intervention. These data consisted of classroom visits looking for evidence of implementation, interviews with teachers and head teachers, focus groups with head teachers, teachers, students in Project Y schools and members of the School Management Committees (SMC,).
The underlying rationale for this approach was that despite enormous sums spent worldwide each year on CPDL for teachers, very few studies have found evidence of
sustained improvement in students’ performance (Coe, 2013). In low-income countries, the problem is aggravated when educational aid agencies are more concerned with their own philosophical or ideological agendas than in finding a common approach that teachers can use to raise standards (Galloway, 2006). It is further aggravated when aid agencies – or their donors – decide to focus on materials, for example provision of textbooks (Sabarwal, Evans, & Marshak, 2014), or important but relatively small topics such as lesson plans rather than on a broader “whole school” approach that integrates cognitive and social learning (Zhao, 2016).
While education is a stated government priority in Sierra Leone for each new regime as it comes into power, education improvement activities have largely focused on reconstruction and rehabilitation of school infrastructure, particularly in the SABABU project, the significant post war programme for rehabilitation of basic education. Only 13.4% of the overall project costs were spent on any form of teacher training. Of the small teacher training component the majority of the project went on peace education and HIV/AIDS prevention rather than on the foundational skills of literacy and numeracy (Bu-Buakei Jabbi, 2007). Further,
education INGOs have undertaken teacher training programmes but these have tended to have a continuing emphasis on HIV/AIDS awareness and peace education (Ministry of Education, Science & Technology, 2007). The post Ebola Presidential Recovery Programme undertook a national teacher training programme in English Language and Mathematics but the initial reviews, as yet unpublished, indicate that the brevity of the training has raised doubts about the possibility of sustained or meaningful impact (Government of Sierra Leone, 2015).
Impact Evaluation & Process Evaluation
The evaluation structure was characterised as having mixed outcomes because both quantified and qualitative data have been collected. Quantified data were collected to assess whether meaningful impact could be associated with the CPDL programme: the impact evaluation. Qualitative data were collected in order to explain why and how certain changes had, and other changes had not, taken place: the process evaluation.
The study followed a quasi-experimental design with an intervention group and a comparison group but the groups were not randomly selected. In a quasi-experiment limited causal claims can be made about the effectiveness of the intervention by comparing the outcomes from the intervention group with the outcomes from the control group.
Causality
Any causal claim can only be limited because in a quasi-experiment, even with careful matching, baseline equivalence cannot be assumed and therefore other factors could potentially explain any differences observed at post-test.
A quasi-experiment is like a randomised controlled trial but without the crucial
randomisation. A quasi-experiment can contribute to knowledge by providing limited causal evidence of impact. In this instance, the lack of randomisation is compounded by the small sample size and the sample was not randomly selected, so the argument in this study is not for generalisability to other contexts. The argument is that indications of possible impact would suggest value in conducting a larger scale experiment.
Tashakkori and Cresswell (2005) further outline four attributes of a strong mixed methods study as:
1. ‘demonstrate the need for mixed methods to answer research questions that include clearly interconnected qualitative and quantitative components,
2. present distinctly identifiable qualitative and quantitative data (or one transformed to the other) that are analysed and presented separately,
3. make identifiable inferences or conclusions on the basis of the results of appropriate qualitative and quantitative data analyses, and
4. clearly integrate the results of the two or more (qualitative and quantitative) strands of the study into coherent conclusions or inferences that are more comprehensive and meaningful than those of the qualitative or quantitative strands alone.’ (p207) Tashakkori and Cresswell (2005) provide a model for this study and enable us to show clearly how the study aligns with a mixed methods approach. Firstly, there is a need for a mixed-methods approach in this study because the quantified and qualitative components are completely interlinked. Understanding if there has been an impact or not is important, but insufficient. To understand which aspects of the CPDL participants think have had an impact and how they brought about change, according to the participants themselves provides further contextual information.
Research aim 4 required a design which can provide data on teacher behaviour before (Time 1) and after (Time 2) the intervention, in the intervention and control groups to facilitate comparison, and at follow-up (Time 3) in the intervention group. These data were obtained by observing lessons and recording observations against particular criteria. This aim also required a measure of students’ performance before and after the intervention in intervention and control schools in order for comparisons to be drawn. These data were obtained from a literacy test which measured students’ progress through the stages of literacy acquisition.
Research aims 5 & 6 required information that can illuminate how participants felt about, responded to, and were able to implement the different features of the CPDL. These data were obtained through semi-structured interviews and focus group discussions (see below for details). The two sets of data make it possible to respond to gaps in the literature about effective CPDL in Sierra Leone, in low-income countries and further afield.
In relation to Tashakkori & John W. Creswell's (2005) second point, in this study, there were clearly distinct sets of quantified and qualitative data performing different roles and
providing answers to different questions and these were collected, presented and analysed separately. There were quantified outcome data collected at pre and post intervention points answering the question of impact or lack of impact of the CPDL programme. There
were qualitative process data, collected throughout the implementation, that helped answer the question as to why, how and which components did or did not have had an impact.
In relation to the third attribute, in this study, it could be seen that there are identifiable inferences arising from these different data sets. The inference emerging from the impact data would be that there was or was not evidence that the CPDL had had an impact on teacher behaviour and student performance. The inference from the process data would be that element X or Y of the CPDL had been more or less easy to implement or respond to; that a particular element needed strengthening; that a particular element had no impact and so on. These inferences are identifiable and different and emerge separately from the different data sets.
In relation to the fourth attribute, in this study, it can be seen that the quantified and qualitative data do indeed complement and enrich each other.
If only the literacy test and lesson observation had been collected, there might have been evidence suggesting impact in those five schools which had participated in the CPDL. There would have been no means of understanding which components of the CPDL might have contributed to success elsewhere. Any aspect or aspects of the CPDL might have been responsible for the change: be it the intensive preliminary training, the follow up support to the head-teachers, the provision of teaching aids, the support to the teachers’ own literacy through the reading circles or any other aspect. Without the information gathered through the qualitative data sources, there would have been no way of understanding how this CPDL might best be rolled out or if, on the other hand, there would be no way of rolling it out because the contextual particularities were what had ensured success or failure. The quantified data collected were very important and do provide a basis for an understanding of change or no change associated with CPDL. For a more complete understanding of the subtleties of which component was most significant in supporting change and how different participants had experienced and responded to different
components, a process evaluation using qualitative data collection methods was absolutely necessary. It is this that provided the basis for an analysis of the teachers’ experiences of the various aspects of the project and how the different elements contributed to the outcomes. Showing that CPDL has some impact in one context, could potentially affect the 1800+ children who attend the Project Y schools. If the methodology could be applied in
other related, but not identical, contexts, for example the 1.2 million primary school
children in Sierra Leone, the research might indicate the possibility of wider significance with further research and larger sample sizes. Or, the process data might suggest ways for further testing of adapted versions of the QEP intervention elsewhere.
The literacy test and the lesson observations provided a means of measuring if change had occurred. In a context where there are numerous NGO activities purporting to support change and millions of pounds being spent on the endeavour but little robust data collected as to the impact, it is important, if there is a case to be made that a given intervention is different from the general provision in this field, that there are robust data with which to make it. As already argued, the evidence suggests strongly that most CPDL programmes fail to have significant impact on school effectiveness worldwide (Coe, 2013). Riddel and Niño- Zarazúa, (2016) also indicate the lack of efficacy of aid to education including teacher training in many other low-income country contexts and the Country Status Report for Sierra Leone (MEST, 2013) confirms the same struggle in Sierra Leone. The evidence strongly suggests that effective CPDL is still largely evading the education community in general, hence the need to identify, analyse and publicise effective interventions where they exist.
c. Evaluation Methods
This study uses a quasi- experimental design and can best be aligned with what Cohen, Manion, & Morrison, (2011. Kindle Location 14324) describe as ‘a quasi-experimental design: the pre-test-post-test non-equivalent group design’. In this design, as the names suggest, there is a pre-intervention test and a post-intervention test administered to an experimental group and to a control group and a comparison undertaken between the two results. This study is characterised as having non-equivalent groups because they were not randomly selected.
Outcome data were collected through a pre-intervention baseline study (Time 1), post- intervention data collection after 12 months (Time 2) and a follow up of the intervention sample six months later (Time 3). Data were collected in the five participating schools and concurrently from two groups of five comparison schools each.
Process data were collected throughout the intervention and in particular during the follow up implementation phase. Process data were used to explore and understand the outcome data and to find explanations for any change or lack of anticipated change resulting from the
intervention. It was also used to understand how much the schools had or had not been able to participate in the project and achieve change and improvements. The process data were largely qualitative.
Learning from discussions on validity of qualitative data, about the importance of
understanding the process as a means of explaining the outcome data, it was judged that integrating qualitative approaches with the quantified data also minimised any weaknesses (Cohen et al., 2007) in the quantified data by providing the opportunity for analysis of the results gathered through the outcome data collection. It also provided rich material for understanding the context and drawing inferences about what elements of the CPDL
programme might need adaptation for a different context, which elements were likely to be crucial in any other context and so on.
What structure is appropriate for the quantified data collection?
Within the conservative academic and policy maker communities in Sierra Leone qualitative data are taken less seriously (Nishimuko, 2007). This study can provide only indicative data at best because of the small sample size and lack of randomisation but should show whether or not further research using this intervention, or enhanced iterations of it, is worth
undertaking.
In the current study, equivalent groups were not possible to be formed through
randomisation as this was impractical. The most important reason for this was that the funding for the experimental group of schools was specifically targeted at the five schools in Project Y and no others. In order to eliminate as much interference of other potential contaminating factors as possible, and thus to achieve a control group that was as similar as possible to the experimental group, schools were selected from the same district, under the same district deputy director for education. Comparison groups were selected to match the intervention schools but baseline equivalence was not possible to achieve. Financial and time constraints made larger groups impossible either of intervention or control schools. Initial discussions with the teachers in Project Y prior to the CPDL, as well as in EducAid’s experience in other communities, informed a decision in favour of a researcher led design over an action research project. The constraints of time and commitment to action research by the participants was judged to be too onerous and with too high a risk of non-
Experience has taught that in attempts to improve personal professionalism and whole school performance, it takes time to build relationships for full buy-in even of senior staff. If with senior staff this engagement is difficult, it is more so with more junior staff and
particularly those not on the government payroll. Conservative estimates place the figures at 20% of the teaching population as not on the government payroll (P. Bennell, 2004) and outside of urban centres this proportion increases. In the Project Y schools, 39% of the teachers were ‘community teachers.’ Community teachers are not paid by government but by the community, or at times, not at all. This not only demotivates the individual teachers concerned but puts increasing financial pressure on the schools themselves and the school authorities who have to find alternative means to pay their staff or risk losing them
(Williams, 2014). Morale and commitment within the teaching profession in Sierra Leone, is low (Ngegba et al., 2016; O’Neill, 2014).
Researchers in all contexts regularly find that it takes time to build relationships and break down mistrust with teachers participating in CPDL and research studies (Walford, 2001). As Walford points out, it is not always obvious to a school what the benefits will be of
participating in a project or study. The benefits of participating in the more frequently seen model of CPDL in Sierra Leone, of a short one or two-day workshop with travel allowances, relatively luxurious food and a certificate of participation are more obvious and immediate. The beneficiaries do not always engage with the purported aims of the CPDL which should be the learning, and the change this learning will bring in the school. While there is a lack of literature examining the Sierra Leonean understanding of the potential negative impacts of some aid agency driven workshop programmes, EducAid’s experience aligns with
experiences elsewhere, exemplified by a study in Tanzania. Discussing an alternative, indigenous education programme associated with positive student outcomes, Welch (2012) identified corruption, inefficiencies, and a focus on extrinsic rewards for participation in CPDL in the foreign run education improvement programmes in Tanzania that she observed. Intrinsic motivations for learning are characterised as the motivation to learn for learning’s sake, for the joy of gaining new skills and being a better professional (Lepper, Henderlong, & Gingras, 1999). Extrinsic motivation is learning for some other reward. To a large extent, there has been an unfortunate tendency to move away from intrinsic motivations to learn and gain the skills required to be an excellent professional. To gain the trust and
associating with any workshop takes time and a significantly different approach from most NGO work.
Teacher buy-in for an action research process was judged to be unachievable in a first round