School Performance Measures for Testing Direct Effects
To assess school leadership’s direct effects on school performance, the team aimed to develop a school performance variable beyond the measures of student performance alone. The intent was to develop a robust indicator of school performance in the context of sustained educational reforms, reflecting the innovations introduced through the RTS project. To do so, the research team first had to determine the criteria to identify and select high- and low-performing schools for the pilot qualitative assessment. The team drew on existing RTS data sets to assign schools a rating per their performance on the following variables: changes in Early Grade Reading Assessment (EGRA) scores, submission of Zambia School Gateway reports, designation as guidance and counseling centers of excellence, and project staff monitoring and evaluation observations.
A brief note on the variables used to construct the school performance rating:
• The EGRA is a standardized oral assessment of a student’s reading skills, including oral reading fluency (ORF), reported as the number of correct words per minute a student can read. The EGRA has become established as an internationally accepted protocol for measuring foundational literacy skills (Dubeck & Gove, 2015). It has been used in many countries to produce indicators of literacy and fluency (see Gove & Wetterberg, 2011, for example), including in Zambia for the RTS
baseline, midline, and endline impact evaluation surveys (Rhodwell, 2013, 2015, 2017).
• As described in the Introduction, the Zambia School Gateway was a system by which head teachers reported twice yearly on student and teacher performance.
• The “guidance and counseling center of excellence” designation was a recognition of excellence bestowed upon high-performing schools. They served as model schools for implementing innovative reforms in relation to guidance and counseling services provided to their students. One component of the RTS project was to support schools to adopt and
implement guidance and counseling services to improve student health outcomes.
• Lastly, the RTS project monitoring and evaluation team maintained a database of school statistics including dropouts, participation in RTS-supported activities, and responsiveness to PLP requirements (teachers participate in training, materials are used in the
classroom, etc.).
Table 3-3 details the school performance rating construct that was assigned to all 1,200 plus RTS-supported schools. Schools could be scored as high as 4 points (if they scored positively for all four possible 1-point criteria) or as low as −3 points (if they scored negatively for all three possible −1-point criteria). This indexed school performance score attempted to balance field observations with characteristics of schools’ fidelity to RTS-introduced innovations.
Table 3-3. School performance rating criteria
Criterion Score = 1 point Score =−1 point
Change in EGRA score from baseline to midline
If a school’s midline ORF average was greater than five correct words per minute (the national average) and improved over its baseline score (n= 53)
If a school’s midline ORF average was one correct word per minute or worsened over its baseline score (n= 22)
Zambia School Gateway returns
If a school successfully submitted teacher and learner performance data for each cycle beginning in Term 1, 2014, through Term 1, 2015 (all six submissions) (n= 318)
If a school submitted teacher and learner performance data for no more than two or fewer cycles out of six expected submissions (n= 269) Designation as guidance and counseling center of excellence
Based on school list provided by guidance and counseling team, school was accorded 1 point for designation (n= 107)
No criterion given to negatively score school’s performance in relation to guidance and counseling implementation (n= 0) Field monitoring
and evaluation observations
The RTS project monitoring and evaluation team observed schools and collected monitoring and evaluation records for its statistical database; schools identified as high performers were given 1 point (n= 256)
Schools identified as low performers were given −1 point (n= 92)
On an 8-point scale, the 1,204 RTS-supported schools received the following scores:
• super = 4 points (2 schools, 0.2 percent) • excellent = 3 points (33 schools, 2.7 percent) • good = 2 points (111 schools, 9.2 percent) • notable = 1 point (297 schools, 24.7 percent) • neutral = 0 points (496 schools, 41.2 percent) • poor =−1 point (255 schools, 21.2 percent) • very poor =−2 points (9 schools, 0.7 percent) • extremely poor =−3 points (1 school, 0.1 percent)
Learning Outcome Measures for the Whole Model Test: Changes in ORF for Grade 2 and Grade 3 Learners from Baseline to Midline to Endline
The learning outcome measures were the changes in ORF, as measured by the number of correct words per minute, from baseline to endline and from midline to endline for grade 2 and grade 3 students. The EGRA was administered in 197 schools (approximately 4,000 learners) in 2013 for the baseline, 200 schools (4,000 learners) in 2015 for the midline, and 200 schools (4,000 learners) in 2017 for the endline of the RTS project. The EGRA
results—and in particular, the changes in ORF scores—were among the key indicators for evaluating the RTS project’s impact on literacy learning outcomes. The Cronbach’s alpha for the EGRA ORF subtask was 0.97, and the overall mean ORF scores for grade 2 and grade 3 were 7.4 correct words per minute and 12.5 correct words per minute respectively. However, an analysis of the endline EGRA ORF scores showed that significant differences were attributed to the language in which the students took the test.
Table 3-4 displays the summary statistics for the learning outcome variables. The values in the second column reflect the total number of schools that participated in the leadership survey and from which data were available from the baseline, midline, and endline EGRAs. The smaller the number of schools surveyed, the greater the standard error, and the larger the confidence intervals. The relatively large standard deviations and upper and lower 95 percent confidence means attest to the lack of precision due to the small sample size. The maximum and minimum scores intimate that outliers may have skewed the results of the analyses.