EVALUATING
THE
EFFECTIVENESS
OF
EDUCATIONAL
BLOCK
GRANTS
TO
ORPHANS
AND
VULNERABLE
CHILDREN
September 2011
Center for Global Health and Development
Boston University School of Public Health &
Boston University School of Education
Boston, Massachusetts, USA
The USAID | Project SEARCH, Orphans and Vulnerable Children Comprehensive Action Research (OVC‐CARE) Task Order, is funded by the U.S. Agency for International Development under Contract No. GHH‐I‐00‐07‐00023‐00, beginning August 1, 2008. OVC‐CARE Task Order is implemented by Boston University.
ORPHANS AND VULNERABLE CHILDREN
S
EPTEMBER2011
Funding Source: PEPFAR‐USAID IQC No. GHH‐I‐00‐07‐00023‐00 Task Order No. 01
Principal Investigator: Malcolm Bryant, MBBS, MPH
Center for Global Health and Development
Co‐investigators (BU): Mary Shann, PhD
Boston University School of Education
Bram Brooks, MPH
Center for Global Health and Development
Co‐investigators (Uganda): Paul Bukuluki, PhD
Makerere University
Denis Muhangi, PhD
Makerere University
Co‐investigators (Tanzania): Joe Lugalla, PhD
Centre for Strategic Research and Development (CESTRE)
Gideon Kwesigabo, MD
Centre for Strategic Research and Development (CESTRE)
Contact information: Malcolm Bryant
Center for Global Health and Development, Boston University 801 Massachusetts Ave, Crosstown Center 3rd Floor,
Boston, MA 02118, USA
Table of Contents
LISTOFACRONYMS...1
EXECUTIVESUMMARY...2
INTRODUCTIONANDOVERVIEW...22
BACKGROUNDANDRATIONALE...22
METHODOLOGYANDSTUDYSITES...31
SAMPLEDESIGNANDSIZE...34
STUDYPROCEDURES...35
RESULTS...37
DISCUSSION...88
REFERENCES...97
APPENDECES...100
1. INSTRUMENT:QUANT1(STUDENT COMPARISON) ... 100
2. INSTRUMENT:QUANT2(SCHOOL COMPARISON) ... 104
3. INSTRUMENT:FGD(COMMUNITY REPRESENTATIVES) ... 106
4. INSTRUMENT:FGD(OVCSTUDENTS) ... 109
5. INSTRUMENT:SSI(NGOS) ... 112
6. INSTRUMENT:SSI(HEADMASTER /DEPUTY HEADMASTER) ... 116
7. CAREGIVER CONSENT FORM ... 121
8. CHILD ASSENT FORM ... 123
9. OTHER CONSENT FORM... 125
List of Acronyms
AIDS Acquired Immune Deficiency Syndrome
AVSI Association of Volunteers in International Service
BG Block Grant
BU Boston University
CBO Community Based Organization
CGHD Center for Global Health and Development
COPE Community‐Based Orphan‐Care, Protection & Empowerment
CRS Catholic Relief Service
FBO Faith Based Organization
ICOBI Integrated Community Based Initiatives
IRB Institutional Review Board
NGO Non‐Governmental Organization
OVC Orphans and Vulnerable Children
PEPFAR President’s Emergency Plan for AIDS Relief
S Scholarship Program
SSN Social Safety Net
USAID United States Agency for International Development
USE Universal Secondary Education
EXECUTIVE SUMMARY
Introduction
Boston University, in collaboration with Makerere University in Uganda and the Center for
Strategic Research and Development (CESTRE) in Tanzania, implemented a mixed methods
study to determine the effectiveness and cost of different approaches to providing educational
support to OVC at secondary schools. The study had four objectives: to compare block grants
(BG) versus individual scholarship (S) approaches; to explore the differences in effectiveness
among various block grant approaches; to calculate the costs per child of each approach, and to
measure the impact of each approach on educational outcomes for all children in the targeted
schools.
Four NGOs funded by PEPFAR through USAID Track 1 funds were studied. Two used block
grants as a means to support OVC in secondary school (Africare and ICOBI), while two used
scholarship approaches (AVSI and CRS). One NGO providing block grants operated in both
countries (Africare). Each of the five NGO programs studied was distinct in its characteristics,
administration, and adaptations to the local environment.
The study was a retrospective cohort study and used a mixed methods design adopting both
qualitative and quantitative approaches to research.
The qualitative components drew on interviews, focus groups, observational techniques, and
document analysis to learn about the intervention used by each NGO and its context from the
point of view of the primary recipients and other stakeholders who were knowledgeable about
the OVC block grants and scholarship programs.
The quantitative components of the study took three forms. The first (QUANT1) addressed how
well the OVC fared under different approaches, with a specific focus on comparing outcomes
for those in block grant and scholarship programs over time as compared to their OVC peers
and non‐OVC peers who did not receive any targeted benefit. The second quantitative
component (QUANT2) addressed how individual schools that have received financial assistance
through block grants have fared over time as compared with a matched set of schools that did
not receive any form of block grants. The third quantitative component of the study (QUANT3)
examined the relative cost effectiveness of different models of disbursing the funding to
schools to aid OVC by examining direct costs, indirect costs, and imputed costs of each NGO
program.
Results
A total of 5,738 students were recruited to the study in both countries. Table 1 outlines the
overall characteristics of the sample size.
Gender characteristics of the students were virtually identical in both countries with 52.5%
enrolment age in the two countries. In Uganda children begin primary school at 6 years of age,
whereas in Tanzania they begin primary school at 7 years of age.
Table 1: Study Sample Characteristics
Tanzania (n = 1930) Uganda (n = 3808)
Mean Student Age (years) F1/S1 Student (Std Dev) F2/S2 Student (Std Dev ) F3/S3 Student (Std Dev ) F4/S4 Student (Std Dev ) 15.8 (1.4) 16.8 (1.4) 17.6 (1.4) 18.3 (1.2) 14.5 (2.0) 15.6 (1.9) 16.6 (1.9) 17.5 (1.9) Gender Male (%) Female (%) 1013 (52.5%) 917 (47.5%) 2003 (52.6%) 1805 (47.4%) Student Status
OVC Ever Supported (%) OVC Never Supported (%) Non‐OVC (%) 498 (25.8%) 721 (37.4%) 711 (36.8%) 777 (20.4%) 1531 (40.2%) 1500 (39.4%)
Student NGO Support Africare (%) ICOBI (%) AVSI (%) CRS (%) None (%) 243 (12.6%) ‐‐ ‐‐ 255 (13.2%) 1432 (74.2%) 256 (6.7%) 257 (6.7%) 264 (6.9%) ‐‐ 3031 (79.6%)
Availability of Student Records 2006 (%) 2007 (%) 2008 (%) 2009 (%)* 232 (12.0%) 747 (38.7%) 1464 (75.9%) 1723 (89.3%) 662 (17.4%) 1130 (29.7%) 2670 (70.12%) 2605 (68.4%)
*Student records for ICOBI (Uganda) were captured up until 2010
Table 2 identifies the specific sub‐populations in the sample by country and type of approach:
Table 2: Sample Characteristics by Intervention
Uganda Tanzania
Total number of students 3,808
Block Grant (Africare) supported OVC – 256
Block Grant (ICOBI) supported OVC – 257
Scholarship (AVSI) supported OVC– 264
Control OVC (never supported) – 1,531
Non OVC – 1,500
Total number of students 1,930
Block Grant (Africare) supported OVC– 243
Scholarship (CRS) supported OVC– 255
Control OVC (never supported) – 721
QUANT1
Mean absenteeism rate was calculated by averaging absenteeism rates between the different
groups. The results, presented in Table 3, show a significant difference overall between groups,
with ever supported OVC (5.1%) having much better attendance rates than never supported
OVC (6.4%). Also, never supported OVC (6.4%) had much worse attendance rates than non OVC
(5.5%). However, we found marked differences between Tanzania and Uganda. In Tanzania,
children who were ever supported (6.5%) demonstrated worse attendance rates than their
never supported counterparts (5.7%) and no significant difference as compared to non OVC. On
the other hand, in Uganda, ever‐supported OVC (3.6%) showed much better attendance rates
than either their never supported OVC (6.8%) or non OVC (6.1%) counterparts.
Table 3: Mean Student Absenteeism Rate, Stratified by Country
OVC Ever
Supported
OVC Never
Supported
Non‐OVC
Mean Student Absenteeism Rate
(2006‐2009) Tanzania (n=1783) Uganda (n=2792) Total (n=4575) 6.5%* 3.6%* 5.1%* 5.7%* 6.8%* 6.4%*+ 6.1% 5.1%* 5.5%+
Paired t‐test compares the means of two groups of observations: OVC Ever Supported vs. OVC Never Supported; OVC Never Supported vs. Non‐OVC; OVC Ever Supported vs. Non‐OVC. Paired significance is noted accordingly.
* Student status significantly different at p < 0.05 + Student status significantly different at p < 0.05
When we examined absenteeism results across NGOs to compare block grants and
scholarships, we found some interesting differences (Table 4). Firstly, in Tanzania there is no
significant difference in absenteeism in the aggregate or when broken down by gender
between block grant or scholarships, although it is interesting to note that absenteeism among
male OVC at scholarship schools (supported or not) is much higher than females. Secondly, in
Uganda, all three NGOs show dramatic reductions in absenteeism over both comparison
groups, however children supported through Africare’s block grant approach did not fare
statistically better than OVC who attended the same schools and were not supported.
Table 4: Mean Student Absenteeism Rate, Stratified by NGO Support and Gender
OVC Ever Supported OVC Never Supported Non‐OVC
Mean Absenteeism Rate
(2006‐2009) Africare (BG) UG (n=721) Male (n=404) Female (n=317) 3.2%* 2.5%*+ 4.0% 5.6% 5.7%* 5.5% 6.1%* 5.8%+ 6.4% ICOBI (BG) UG (n=1010) Male (n=515) Female (n=495) 3.7%* 4.0%* 3.6%* 8.4%*+ 8.9%*+ 7.9%*+ 4.5%+ 4.9%+ 4.1%+ AVSI (S) UG (n=1059) Male (n=542) Female (n=517) 3.6%* 4.3%* 2.7%*+ 6.1%* 7.2%*+ 5.2%* 5.0%* 5.3%+ 4.7%+ Africare (BG) TZ (n=1152) Male (n=614) Female (538) 6.2% 6.0% 6.3% 5.8% 5.6% 6.0% 6.3% 6.5% 6.0% CRS (S) TZ (n=631) Male (n=338) Female (n=293 7.0% 8.1% 5.6% 5.4% 6.2% 4.4% 5.7% 5.7% 5.7%
Paired t‐test compares the means of two groups of observations: OVC Ever Supported vs. OVC Never Supported; OVC Never Supported vs. Non‐OVC; OVC Ever Supported vs. Non‐OVC. Paired significance is noted accordingly.
* Student status significantly different at p < 0.05 + Student status significantly different at p < 0.05
Drop out rates were significantly different by country and showed considerable variation
among NGO and approaches to supporting students.
It is striking in Table 5 that the drop out rates for supported OVC are significantly worse for
Africare supported schools in both countries. Further investigation is required to understand
this, but the same pattern is not reflected in the other block grant program, so that it is most
likely an effect of Africare’s approach rather than the block grant approach as a whole.
A multivariate model was created to predict student drop out (Table 6). Secondary school
students in Uganda are eleven times more like to drop out compared to students in Tanzania.
This is not surprising since dropout rates in Uganda are considerably higher than dropout rates
in Tanzania. When analyzing the effect of gender, there seems to be no difference in dropout
rates between male and female students. Finally, when comparing the impact of educational
support, we see that never supported OVC students are almost twice as likely to drop out when
compared to non‐OVC students – OR 1.83 (95% CI 1.46‐2.30). The adjusted odds ratio is slightly
lower for ever supported OVC students compared to non‐OVC students – OR 1.46 (95% CI 1.10‐
1.92) – suggesting that the support they receive may facilitate a slight reduction in the
likelihood of dropping out of school.
Table 5: Drop Out Rate by Student Status and Approach 2006‐2009
Proportion (%) Odds Ratio (95% CI)
NGO Support
Africare UG ‐ Block Grant
Non‐OVC [Ref.] 48/491 (9.8%) 1.00
OVC Never‐Supported 88/493 (17.9%) 2.01 (1.38‐2.92)* OVC Ever‐Supported 39/247 (15.8%) 1.73 (1.10‐2.72)*
ICOBI UG ‐ Block Grant
Non‐OVC [Ref.] 43/475 (9.1%) 1.00
OVC Never‐Supported 98/517 (19.0%) 2.35 (1.60‐3.45)* OVC Ever‐Supported 31/255 (12.2%) 1.39 (0.85‐2.27)
AVSI UG ‐ Scholarship
Non‐OVC [Ref.] 35/508 (6.9%) 1.00
OVC Never‐Supported 36/501 (7.2%) 1.05 (0.65‐1.70) OVC Ever‐Supported 14/264 (5.3%) 0.76 (0.40‐1.43)
Africare TZ ‐ Block Grant
Non‐OVC [Ref.] 3/466 (0.6%) 1.00
OVC Never‐Supported 4/469 (0.9%) 1.33 (0.30‐5.96) OVC Ever‐Supported 7/242 (2.9%) 4.60 (1.18‐17.94)*
CRS TZ ‐ Scholarship
Non‐OVC [Ref.] 1/244 (0.4%) 1.00
OVC Never‐Supported 3/250 (1.2%) 2.95 (0.30‐28.57) OVC Ever‐Supported 5/524 (2.0%) 2.34 (0.27‐20.15)
* Student status significantly different at p < 0.05
Table 6: Multivariate Analysis to Predict Drop Out (N=5679)
Adjusted Odds RatioƗ (95% CI)
Country Tanzania [Ref.] 1.00 Uganda 10.87 (7.11‐16.60)* Gender Male [Ref.] 1.00 Female 1.068 (0.88‐1.30) Student Status
Non‐OVC [Ref.] 1.00
Never Supported OVC 1.83 (1.46‐2.30)* Ever Supported OVC 1.46 (1.10‐1.92)*
Ɨ Adjusted odds ratio by logistic regression analysis * Significantly different at p < 0.05
A second multivariate model looked at predictors of passing the national Form IV exams (Table
7). The observation that students in Uganda are eleven times more likely to pass their From IV
exams compared to their Tanzanian counterparts is largely an artifact of several things. Firstly,
the two countries use different assessment criteria to determine pass/fail; secondly, the
ministry of education advised NGOs to support the schools with the greatest problems and
poorest performance; finally, because of the later start of schooling in Tanzania, large numbers
of students “aged out” of the program, leaving a very small proportion (less than 10%) of
students taking Form IV exams. However the most plausible reason behind the difference in
passing rates on Form IV exams between the two countries is rooted in the difference in
language policy. In both countries, Form IV exams are given in English. In Uganda where many
tribal languages are spoken and no one language dominates, English is widely used and is the
language of instruction throughout elementary and secondary schools. In Tanzania, Swahili has
been adopted as the national language, and it is the language of instruction in elementary
schools. However, in secondary school in Tanzania, there is an abrupt change to the use of
English as the language of instruction. Students take their final Form IV examinations in English,
a language they have only used for four years.
In assessing gender as a predictor of passing From IV exams, stark differences can be seen
between male and females students. Girls are half as likely to pass their Form IV exams
compared to boys – OR 0.55 (95% CI 0.45‐0.67). The impact of educational support can be seen
as having a positive effect on passing the Form IV exams. Whereas never supported OVC are
20% less likely to pass their exam – OR 0.78 (95% CI 0.62‐0.99), there is no statistical
significance between ever supported OVC and non‐OVC students. This finding suggests that the
educational support received by OVC students puts them almost on the same passing rate
compared to non‐OVC students.
Table 7: Multivariate Analysis to Predict Passing of National Form IV Examination (N=1752)
There were indications that children in block grant supported programs performed better than
scholarship schools, but none of the differences were significant at the alpha 0.05 level.
Adjusted Odds RatioƗ (95% CI)
Country Tanzania [Ref.] 1.00 Uganda 11.47 (6.67‐19.74)* Gender Male [Ref.] 1.00 Female 0.55 (0.45‐0.67)* Student Status
Non‐OVC [Ref.] 1.00
Never Supported OVC 0.78 (0.62‐0.99)* Ever Supported OVC 1.10 (0.85‐1.42) Ɨ
Table 8 – Table 10 explore the Form II examination results (Tanzania) and the Form II and Form
IV examination results (Uganda and Tanzania) by subject stratified by gender. With the
exception of Form IV results from Tanzania, these tables emphasize the gender discrepancies
and confirm that they exist for every subject and exist between the two countries.
Table 8: Form II National Examination Scores between Male and Female Students, Tanzania
Female Male
Mean Exam Scores Ɨ
(2006‐2009) English (n=1340) Civics (n=1341) Geography (n=613) Math (n=1341) Physics (n=1275) Chemistry (n=1294) Biology (n=613) Kiswahili (n=1341) 26.7 (13.5) 30.2 (12.4)* 44.8 (14.6)* 14.7 (13.9)* 33.7 (14.1) 32.6 (13.7)* 35.0 (12.5)* 46.9 (19.4) 28.1 (16.1) 33.0 (14.2)* 47.8 (16.3)* 17.9 (17.1)* 34.8 (16.7) 34.3 (15.3)* 38.9 (14.8)* 47.0 (17.1) * Student status significantly different at p < 0.05
Ɨ Exam scores are out of 100 points (higher points indicate better exam performance)
Note: Form II National Examinations are not conducted in Uganda
Table 9: Form IV National Examination Scores between Male and Female Students, Tanzania
Female Male
Mean Exam Scores Ɨ
(2006‐2009) English (n=152) Civics (n=152) Math (n=153) Physics (n=45) Chemistry (n=45) Kiswahili (n=151) 4.2 (0.8) 4.2 (0.8) 4.8 (0.5) 4.2 (0.8) 4.3 (0.7) 4.2 (0.8) 4.3 (0.8) 4.2 (0.8) 4.6 (0.9) 4.2 (0.8) 4.1 (0.9) 4.2 (0.7) * Student status significantly different at p < 0.05
Ɨ
Exam scores are out of 5 points: 1=A 2=B 3=C 4=D 5=F (lower points indicate better exam performance)
Table 10: Form IV National Examination Scores between Male and Female Students, Uganda
Female Male
Mean Exam Scores Ɨ
(2006‐2009) English (n=1667) Geography (n=841) Math (n=1667) Physics (n=1664) Chemistry (n=1661) Biology (n=838) 6.0 (1.8) 6.0 (1.6)* 7.1 (1.7)* 8.1 (1.5)* 8.3 (1.3)* 7.4 (1.6)* 6.1 (1.7) 5.5 (1.7)* 6.6 (1.9)* 7.5 (1.7)* 7.9 (1.6)* 6.8 (1.8)* * Student status significantly different at p < 0.05
Ɨ
Exam scores are out of 9 points: 1=D1 2=D2 3=C3 4=C4 5=C5 6=C6 7=P7 8=P8 9=F (lower points indicate better exam performance)
QUANT2
This part of the study was conducted in response to the repeated observation by school
teachers, head teachers, and NGO staff, that the improvements that block grants brought to
the school improved the quality of the school grounds, increased the number of class rooms,
brought physical improvements such as water, electricity, latrines, and also purchased essential
laboratory and other equipment. It was frequently observed that these changes had improved
the standing of the school, attracted new students, and improved overall school performance.
We hoped to be able to observe if the interventions had made any noticeable difference to the
overall functioning of the school. In order to study whether these changes had occurred,
observations were taken for four years prior to when the interventions began. Because of the
small sample size and large standard deviations, no statistical significance tests were
conducted, and only trends can be observed.
Anecdotal stories of increases in enrolment because of block grants are not born out by the
data presented in QUANT2 and do not show variance with the national trends. Increases noted
by staff at assisted schools, are therefore more likely to be as a result of pre‐existing trends
augmented by Universal Secondary Education (USE). Overall, with the exception of AVSI
supported schools, there was a steady rise in student to teacher ratios, with the ratios
noticeably higher for Africare and ICOBI. The similarities between Africare and ICOBI are not
surprising as they both serve rural areas in the Southwest part of the country. The National data
represent both urban and rural schools, but the majority of schools in Uganda are still rural,
thus trends are more likely to resemble those seen with Africa and ICOBI.
In addition to the rise in student to teacher ratios, national examination scores fell dramatically
over the observation period in national schools, once again being paralleled in Africare and
ICOBI schools. Observations from Western settings have demonstrated a direct relationship
between student to teacher ratios and the quality of education. We speculate that the
increasing ratios were at least in part, responsible for the decreasing exam scores. Most
importantly, block grants did not seem to have influenced this downward trend line.
QUANT3
Each of the NGOs kept financial data in a different format, which made analysis of the cost data
complex, and impossible to provide identical categorization of financial expenditures and costs
for each organization. There was wide variation in the total number of children served by each
program, and even pinning this number down was difficult because of the PEPFAR reporting
requirements which call for both direct and indirect support to be noted. Each NGO had a
slightly different way of calculating the percentage of resources that go to direct vs. indirectly
supported OVC.
Table 11 shows the comparative costs per child of block grants vs. scholarships. Costs of
scholarship programs are significantly and consistently higher than the block grant programs in
both countries.
Table 11: Comparative Costs of Block Grants vs. Scholarships
Uganda Tanzania
Block Grant average cost per child per year $324 $293
Scholarship average cost per child per year $412 $356
Difference between approaches per child per year $88 $63
When examined by NGO (Table 12) there is considerable variation between NGOs with some
obvious inefficiencies being present. One means of reducing the inefficiencies would be to
increase the number of children served as this would appear to gain significant economies of
scale. However, even with large numbers of children, 40% of the total expenditure per child is
going to administration costs.
Table 12: Breakdown of Program Costs by NGO
Africare TZ USD (% of total) CRS TZ USD (% of total) Africare UG USD (% of total) AVSI UG USD (% of total) ICOBI UG USD (% of total) Total children served 1,150 1,117 700 3,352 3,108 Direct Benefit to Child $109 (37%) $130 (37%) $127 (37%) $199 (61%) $184 (60%) Program costs $127 (43%) $140 (39%) $150 (44%) $87 (27%) $76 (25%) Indirect costs and donations $56 (19%) $86 (24%) $66 (19%) $42 (13%) $45 (15% Total $293 (100%) $356 (100%) $343 (100%) $412 (100%) $305 (100%)
Finally, Table 13 compares the cost effectiveness of the two approaches. With the exception of
the variation in drop‐out rates there is little difference between the approaches in general in
terms of their ability to produce educational outcomes. The qualitative data suggests the
variation in drop‐out rates is directly related to the amount of money given as direct benefit to
the child rather than the difference between block grant or scholarship (e.g. Africare with a
higher drop‐out rate in Uganda only provides partial funding and spends between $110 ‐ $130
per child, while AVSI provides far in excess of the actual school fees and spends $199 per child).
Table 13: Comparison of Effectiveness of Different Approaches to Providing Support to OVC vs. No Support by
Country
UGANDA
Cost per child
per year
Attendance Drop out Academic Achievements
Block Grants $324 Significantly Improved No difference No difference to non OVC Scholarship $412 Significantly Improved Significantly improved No difference to non OVC TANZANIA
Block Grants $293 No difference Significantly higher rates Notably better performance
Based on the above, we come to the conclusion that block grants are more cost‐effective than
scholarships, although we observed in the qualitative data that as the number of children
included in a school supported by a block grant becomes small (less than ten), the cost
effectiveness approaches that of scholarships.
QUALITATIVE
Individual interviews or focus group discussions were conducted with district education
officers, NGO chiefs and their staff, headmasters, community representatives, and both
supported and non‐supported OVCs separated by gender. A descriptive summary of the
qualitative summary is presented in Table 14. Some of the questions yielded brief responses
that could be tallied. Others yielded more detailed explanations. In the latter cases themes
were abstracted and direct quotes presented that anchored the range and common ground of
the sentiments.
Table 14: Descriptive Summary of Participants in the Qualitative Study
Headmasters (IDI) Community Reps. (FGD) District Education Officers (IDI) NGO Reps. (IDI) OVC (FGD) Tanzania Gender Male Female 32 12 18 3 1 2 3 0 27 27 Total 44 21 3 3 54 Uganda Gender Male Female 35 10 18 11 2 0 2 0 42 43 Total 45 29 2 2 85 Grand Total 89 50 5 5 139
Notes: IDI = In‐Depth Interview FGD = Focus Group Discussion
Focus Groups with OVC Students
Although there are varying perceptions of how they felt as an OVC, in general, most students
felt negatively about being an OVC. Students in Tanzania and Uganda listed different problems
faced by OVC, however, issues such as discrimination, stigma, lack of educational materials, and
shortages in food were commonly raised. When asked the question of how OVC students fared
compared to their non‐OVC counterpart, it’s interesting to note that the majority of OVC felt as
though they either performed equally or studied harder than non‐OVC. (More importantly, the
quantitative achievement data bear up this claim.) Furthermore, in both countries, many of the
supported OVC who claimed they had done well academically responded with lofty goals when
asked about their aspirations for the future, including wanting to be a pilot, accountant, doctor,
Key Informant Interviews with Headmasters
Headmasters noted that the main issues faced by schools included insufficient classroom and
facilities, lack of electricity and water, and teen pregnancy. In regards to the selection process
associated for educational support, different NGOs had different criteria for the selection of
schools. Some schools applied for block grant support while other schools were asked to
participate and were not part of the decision process. When asked about the strengths of the
block grant support, most headmasters responded favorably, especially when support was
translated into material goods such as desks, tables, and laboratory supplies. A wide range of
weaknesses of the block grant were also highlighted by the headmasters, including delays in
payments, the small number of OVC students that can be supported, and the school
administration’s limited ability to prioritize their needs. Overall, headmasters noted that the
benefits of the block grant were mostly positive and that block grant support benefitted the
entire school.
Key Informant Interviews with NGO Personnel
Some of the positive benefits for the school as a result of the various educational support
included increased school reputation and the ability for the school to invest on capital
developments that they would otherwise be unable to afford. In regards to positive benefits for
children, NGO personnel mentioned that support increased school attendance and that it was
beneficial to students who were unable to pay for school fees. Various strengths were
mentioned of the different educational support that NGOs provided, from improved school
infrastructure, increased enrollment, and the ability of block grants to support OVC and non‐
OVC students. Although optimistic about the positive impact of the support that they have
provided, NGO personnel were realistic of the limitations of their intervention. Educational
support does not address the other needs of a child that influence retention and academic
performance. In addition, it was noted that some caregivers show reduced responsibilities
towards OVC as they believe that the NGOs will provide all the necessary assistance to a
supported child.
Key Informant Interviews with District Education Officers
DEOs highlighted some of the main challenges faced by the district, from the material
deficiencies in schools to the issue of high poverty and illiteracy rates within their district. In
terms of specific problems faced by OVC, DEOs reiterated the stigma and negative attitudes
towards OVC and the educational issues faced by OVC. In regards to the process to select
schools and children for educational support, DEOs noted the involvement of local communities
in selecting schools and children to be supported. When asked about ways to improve
educational support in their respective districts, it was interesting to note that one DEO
requested more evaluation and follow‐up of different support programs. As OVC support is
targeted for children under 18 and programs have a limited implementation timeline, it was
common to hear complaints associated with the fact that supported students were unable to
finish the four year O‐Level education cycle due to these issues.
Focus Group Discussion with Community Representatives
Community representatives repeated some of the main OVC problems that were already
highlighted by earlier interviews, including lack of food and education issues. When asked
about the process for identifying OVC for selection of educational support, there was no
consistent way in which this was done – reflecting the different approaches that NGOs took in
selecting OVC. OVC identification is usually done at the community level, either through
community leaders or village governments. It was interesting to note that factors such as
poverty were considered as a selection criteria as preference was usually given to children who
are “needy” and “badly off”. Finally, when asked about ways to improve OVC support
programs, community representatives stressed the importance of close collaboration with local
governments. In addition, they thought that it was important to follow up supported OVC to
monitor their progress and find effective ways to select OVC for educational support.
Discussion
While there were two main approaches for the delivery of services to OVC ‐‐block grants and
scholarships‐‐ the five NGOs delivering the services had distinctive programs for targeting the
aid to needy students. Therefore we examined them individually. Each of the five programs
supporting OVC to attend secondary school that we studied was successful in allowing limited
numbers of OVC who would not have been able to receive secondary education, go to school.
From that perspective, every organization attained its programmatic goals. However, there are
limitations presented by each approach, particularly in relation to the costs of the programs
and the inability to address the underlying challenges faced by the educational systems in both
countries we studied.
The original question that this study was tasked with was to compare block grants to schools
with the provision of individual scholarships. We look at the comparison of block grants versus
scholarships as follows:
Block Grants Scholarship
Did the program succeed in increasing access to secondary education?
Yes.
Every child sponsored by Africare and ICOBI
were verified as attending school and their
records examined.
Partly.
In Tanzania, significant numbers of children
registered in the scholarship program could not
be verified as attending school and their
records could not be found. In Uganda, the
same administrative issues were encountered
at first leading to significant modification of the
program.
Did the program provide equity in access to secondary education?
No
Limited funds were available and priorities had
to be made about who would be chosen to
receive secondary education and who would
not. However, because of the process used to
select OVC the community was generally
aware of the number of OVC in need and
additional sources were sought out for OVC
not supported by the block grants. Most
importantly, the schools were aware of who
was sponsored and who was not and could
pay increased attention to them.
No
As with block grants, limited funds were
available and priorities had to be made. The
process of selection was not perceived to be
transparent. Schools often were unaware of
children selected, and thus could not provide
attention to special needs or follow‐up on
absenteeism or drop‐ outs.
Did the program result in improved attendance at school?
Partially
Two of the three block grant programs
showed significant improvements of
supported OVC over their non OVC and non‐
supported OVC counterparts.
Partially
One of two scholarship programs demonstrated
significant improvements of supported OVC
over their non‐OVC and supported OVC
counterparts.
Did the program result in a decrease in dropout rates?
Mixed
Two of the three block grant programs saw
significantly increased drop‐out rates when
compared to non‐OVC, while the third saw
lower drop‐out rates.
No
There were no observed differences in either
scholarship approach between supported OVC
and their non‐OVC counterparts.
Did the program result in better academic performance?
Possibly
Supported OVC in Tanzania performed better
on their aggregate scores for Form II, but the
result was not statistically significant. There
were no measurable differences on aggregate
scores for form IV and individual subject
scores showed that OVC performed at the
same level as non‐OVC.
No
There were no differences on Form II or Form
IV aggregate scores. Numbers were too small to
Block Grants Scholarship
Did the program result in measurable changes in the school’s infrastructure?
Yes
Schools documented construction of new
classrooms, latrines, water tanks, electric
generators, and purchase of laboratory
equipment and other school materials for all
block grant approaches.
No
There were no known benefits to the schools.
Interviews with teachers showed that in the
majority of cases teachers were unaware that
assistance was being provided to the school.
Did the program result in measurable improvements in the school’s performance
No
Overall school performance decreased over
time, but this is explained by the rapidly rising
enrolments and increasing student to teacher
ratios. There were no differences observed
between intervention schools and control
schools.
No
This was not part of the design or intent of the
program.
Did the program result in improvements in psychosocial wellbeing of OVC?
Yes
Students reported being able to use school
resources made available as a result of the
conditionality of block grants.
Partially
Students reported receiving support at home,
but few students were aware of there being a
teacher trained in psychosocial care at the
school.
Was the program cost effective?
More effective
Costs of block grant programs were between
$60 and $90 dollars less per child than
scholarships. However, it was noted that cost
effectiveness is directly related to the number
of students enrolled in the block grant
program.
Less effective
Scholarship programs were more expensive
than block grants, with no consistent difference
in educational performance, absenteeism, or
continuation for children supported by block
grants and scholarships.
Accountability
In addition to the comments above, we made several observations about accountability and the
potential for diversion of funds.
1. Africare’s BG approach was the most regimented approach, leaving little or no freedom
for schools to make their own decisions about the use of funds beyond the relatively
narrow criteria that Africare set for use of the block grant money. This approach was
criticized by several local key informants in other NGOs and the Ministry of Education
because it was perceived to be disempowering of the schools and exhibited a mistrust
of schools capacity to procure goods and manage money. However, the approach was
never criticized by staff or informants the schools that were studied.
2. AVSI’s S approach was less regimented than Africare, but still limited what funds could
be spent on and conducted detailed financial audits on expenditures. No complaints
3. ICOBI’s BG approach was designed to contrast with Africare’s regimented approach and
provide much more freedom for schools to choose and be free of administrative
burden. This approach was extremely popular with schools, but it was observed that
ICOBI is now considering moving to a more rigid model aligned with Africare’s approach
because of concerns about accountability.
4. CRS’s S approach was the most open of all. However, it was also observed to be the
most open to abuse, and several cases were observed which raised concerns. Firstly,
when schools were visited to collect data from supported student’s records there were
several instances where the student had never attended the school and could not be
traced. This led the local investigation team to coin the term “Ghost students” (Watoto
yatima hewa in Swahili), coming to the conclusion that families had received funding,
but money had not been used for school. Secondly, there were several reports by
teachers that students only had part of their funds paid by the local FBO, with the
student forced to leave school partway through the term because of non‐payment.
Finally, in several instances local FBO collaborators simply disappeared after questions
were raised about funding, leading to concern that they feared that the research team
were “auditors”.
Absenteeism and gender
There is a great deal of discussion in both the popular and the scientific literature about the
effect of menstruation on girls’ ability to maintain school attendance and perform adequately in
school. The argument is made that the lack of sanitary napkins prevents girls from attending
school during menstruation, and that even when sanitary napkins are available, lack of privacy
and adequate sanitation makes it difficult for girls to change napkins. Some authors have
argued that this may result in between 18 – 36 days of school lost in a 180 day school year,
Mooijman et al (2005). Many NGOs (including all the NGOs in this study) provide sanitary
napkins to OVC girls they are supporting as part of their program. Our results are therefore
somewhat surprising in that for every group – supported OVC, non‐supported OVC, and non
OVC, girls have a lower rate of absenteeism than boys. Menstruation appears to be less of an
issue for girls than previously thought (some estimates put girls absenteeism as high as 18%,
with the cause being an inability to deal with menses). This is the first study to report
quantitative data on girl’s absenteeism in Africa, and indicates that further study is required to
clarify the issue. Further conclusions cannot be drawn as this was not a specific focus of the
research and questions to differentiate the reasons for girls and boys absenteeism were not
asked.
The findings of reduced absenteeism of girls when compared to boys were consistent in control
schools where there was no intervention to provide sanitary napkins. Our findings are in line
with the recent publication by Oster and Thornton (2011), which concluded that actual lost
time at school due to menses was very small (approximately half a day a year in Nepal). This is
not to suggest that provision of sanitary napkins, good hygiene and privacy are not an
Drop out and gender
There was no difference in the drop‐out rates between boys and girls in any of the schools in
our study (6% for boys and 6% for girls). However, OVC who were supported did not have a
significantly higher rate of drop‐out than non OVC, while non‐supported OVC of both genders
had significantly higher dropout rates. We can conclude therefore that the support being
provided to OVC was protective against dropping out.
Academic performance and gender
Results indicate that on average, female students are half as likely to pass their National Form II
and Form IV exams compared to male students. Across the different support groups, the same
pattern holds true, female students are less likely to pass their national exams. Male OVC ever
supported students are more likely to pass their Form IV exams compared to non‐OVC
students, while female OVC ever supported students are less likely to pass the equivalent exam
compared to their non‐OVC counterpart. Data suggests that educational support to OVC seems
to have better impact on the academic performance of male students as opposed to female
students.
Costs and efficiencies
The wide variation in costs per program largely reflects the variation in administrative oversight
and follow‐up across the organizational approaches, as well as the fact that each organization
contributes different percentages of the total education costs (ranging from 50% for Africare’s
block grant approach in Uganda to 90% for AVSI’s scholarship program in Uganda). It is not
clear how decision‐making should be made about what proportion of the overall costs should
be covered by NGOs and this requires further research.
It was observed that the larger the number of children in the program, the lower the proportion
of administrative costs – suggesting noticeable economies of scale and that large programs are
more cost effective. It was also noted during qualitative interviews that block grant programs
with small numbers of students did not provide sufficient funds to the school to provide the
leverage required to enable the entire school to benefit from the grant. It was concluded that
the larger the number of students enrolled in a block grant program the more cost effective the
approach, while block grant programs with smaller numbers rapidly become as inefficient as
scholarship programs.
Limitations of the Study
Lack of knowledge of the households from which the supported OVC came.
Many contextual factors which would help to explain issues raised in this study cannot be
information does not affect the relevance or quality of the data collected at schools or in focus
groups, but it does limit the ability to interpret some of the findings around absenteeism and
drop out. It also limits the ability to provide direct comparisons between schools as the details
of the populations served are not well known.
Children “aging out” of the program.
Because of early statements about who constituted an OVC by PEPFAR, decisions were made
early on by the various programs that educational support would not be continued beyond the
age of 18 years (in the last year of project implementation this was reversed, but did not affect
the cohort of students being studied). The consequence of this is that children who initiated the
program found themselves losing their educational support in Form IV (and sometimes earlier).
In Tanzania this resulted in the majority of Form IV students being lost to follow‐up and this in‐
turn affected the results for the final year of the cohort – particularly for Form IV examinations.
Lack of concentrated populations of supported OVC.
The finding that both of the scholarship programs (CRS and AVSI) did not have populations of
50 or more supported OVC in any individual secondary school only became clear as individual
recipient lists were examined. This required that a much larger sample size of schools be made
for both NGOs. In turn, directly matched control schools could not be chosen for all schools in
the CRS and AVSI samples. At CRS schools there were often only five or six study students, and
therefore the control students were picked from the same schools.
Differences in the educational systems between Uganda and Tanzania.
Children start school one year younger in Uganda than in Tanzania, with the result that the OVC
in our sample in Tanzania were all one year older than their equivalent grade in Uganda.
Fortunately, while a year’s age difference is significant in younger age groups, it has little
bearing on the ability to perform academically after mid‐adolescence and we do not anticipate
that age would result in any differences in pass rates between the two countries. However, the
later start date in Tanzania meant that many students in Form IV were over 18 and therefore
no‐longer eligible for OVC support. This is reflected in the very low numbers of exam results for
students in Form IV in Tanzania which in turn limited our ability to draw significant conclusions
on Form IV academic performance in this environment.
In 2007, Uganda introduced Universal Secondary Education, eliminating tuition fees for
secondary school. In contrast, fees for secondary education are still charged in Tanzania. The
fees in Tanzania are 20,000 TZS per term, which is then matched by the government. However,
secondary education in Uganda is not free, and fees in Tanzania are not limited to the official
government fees. The reality is that in both countries, the cost of actually attending secondary
school is approximately the same (in excess of 100,000 Shillings). The actual fees charged are
not standardized and may differ from school to school, and even from year to year in the same
school. We therefore do not believe that USE in Uganda presents a problem in making country‐
to‐country comparisons.
Lack of detailed information on country‐wide school performance.
The information needed to conduct performance comparisons of all intervention schools
(regardless of NGO) in the country versus matching schools has not been made available from
the Ministry of Education. This has severely limited the conclusions we can draw from the
QUANT2 study; we can only observe trends rather than make clear statements about the
broader impact of assistance to secondary schools. This constraint does not affect the other
conclusions and recommendations being made in this report.
Conclusions
We can conclude from our study that providing support to children to attend secondary school
can be successful whether through block grants or scholarships, and that in general, supported
performed as well as their non‐OVC peers and sometimes better.
We also concluded that of the two approaches (block grant and scholarship), block grants are
simpler to administer, require less administrative costs and oversight to ensure that students
receive the benefits, and are more cost effective than scholarships per student. However, we
did observe that block grants become less effective with smaller numbers of students enrolled
as not only are there losses in economies of scale, the leverage on the school is much reduced
when only small amounts are provided. We were unable to determine the exact point at which
block grants become most efficient.
It was clear throughout our observations that provision of free access to schools alone is
insufficient to ensure that OVC attend regularly or perform well. Additional support was
required that addressed physical health, nutrition, social setting, economic well‐being, and
other aspects of the child's overall well‐being.
Tanzania specific:
It is obvious that the Government of Tanzania's focus on increasing access to secondary
education in rural areas through the building of ward level schools has been successful.
However, the extremely poor performance at certificate level is of enormous concern reflecting
serious problems with the quality of education, which needs to be addressed urgently.
We noted that Tanzania's use of a national examination at Form II provides an extremely
important benchmark for early decision‐making about appropriate interventions to improve
student performance. It was observed to the research team on several occasions that this
examination is no‐longer may even be phased out which would remove this excellent tool.
Uganda specific:
Because Uganda only collects national standardized achievement data at Form IV, the Ministry
has no way of knowing how the system is performing over time, as a whole, and earlier decision
making about appropriate interventions to improve student performance is problematic.
Recommendations
General Recommendations:
Investment in secondary school education for OVC is of value and is a program that PEPFAR and
other stakeholders should consider supporting while encouraging local Ministries of Education
to take on this role.
In designing secondary school education programs for OVC, Governments, PEPFAR and other
development partners should give a high priority to block grant programs. However, these
should be done within a comprehensive approach that addresses the physical, mental and
social wellbeing of children.
To design their OVC block grant programs more efficiently, PEPFAR and other development
partners should focus on areas with concentrations of OVC (especially in rural settings), and
within those populations seek to serve larger numbers of students in smaller numbers of
schools.
Selection criteria for selecting OVC when resources are limited, should be clarified and
standardized to ensure equity and transparency; should OVC be chosen on the basis of the
most needy, or the most likely to succeed?
Tanzania Specific Recommendations:
In Tanzania access to education for OVC needs to be expanded; the Government of
Tanzania, civil society, and the private sector, should work together to eliminate specific
financial and other barriers to education.
The Government of Tanzania should continue to use Form II examinations as a means of
determining educational effectiveness. If this is impossible, then sampling of students
and content areas can be used making the evaluation of effectiveness less costly.
The Ministry of Education needs to standardize the collection, storage, and maintenance
of school records (including attendance registers and grade reports), and these
standards should be applied to private and church schools as well.
Uganda Specific Recommendations:
In Uganda, in addition to providing Universal Secondary Education, the Government of
Uganda should eliminate financial and other barriers to education for OVC.
The Government of Uganda should give serious consideration to establishing additional
standardized national examinations for earlier grades at secondary school to enable
improved decision‐making about the quality of education. If this is impossible, then
sampling of students and content areas can be used making the evaluation of
effectiveness less costly.
The Ministry of Education needs to standardize the collection, storage, and maintenance
of school records (including attendance registers and grade reports), and these
Recommendations for Further Research:
Research is required to determine what proportion of the total cost of education is the
optimal amount to assure the best outcomes from support programs.
Future research should employ comprehensive outcome measures and longitudinal
approaches to study improvements in student access, teacher effectiveness, and
student performance over time.
INTRODUCTION AND OVERVIEW
At the request of the PEFPAR Technical Working Group for Orphans and Vulnerable Children,
Boston University (BU) undertook a study to provide evidence to facilitate PEPFAR decision‐
makers to make informed choices as new investments are made in educational support to
Orphans and Vulnerable Children (OVC). This work was conducted through the Orphans and
Vulnerable Children Comprehensive Action Research Project (OVC‐CARE), funded by USAID
through Project SEARCH (Contract No. GHH‐I‐00‐07‐00023‐00, Task Order 1)
In early 2011, in collaboration with research partners Makerere University in Uganda and the
Center for Center for Strategic Research and Development (CESTRE) in Tanzania, BU
implemented a mixed methods study to evaluate the effectiveness of educational support
provided to secondary school children through the President’s Emergency Fund for AIDS Relief
(PEPFAR) by a variety of Non Governmental Organizations (NGOs), using different approaches,
in Uganda and Tanzania. The design was a retrospective, longitudinal study, and used a mixed
methods approach, adopting both qualitative and quantitative methodologies.
The qualitative components drew on interviews, focus groups, observational techniques, and
document analysis to learn about the intervention used by each NGO and its context from the
point of view of the primary recipients and other stakeholders who are knowledgeable about
the OVC block grants programs.
The quantitative components of the study took three forms. The first addressed how well the
OVC fared on attendance, drop‐out, and achievement measures under different approaches,
with a specific focus on comparing block grant and scholarship programs over time as
compared to their OVC peers and non‐OVC peers who did not receive any targeted benefit. The
second quantitative component addressed how individual schools that have received financial
assistance through block grants have fared over time as compared with a matched set of
schools that did not receive any form of block grants. The matches were based on school size,
class grade level, locale, and SES characteristics. The third quantitative component of the study
examined the relative cost effectiveness of different models of disbursing the funding to
schools to aid OVC by examining direct costs, indirect costs, and imputed costs of each NGO
program.
BACKGROUND AND RATIONALE
In addition to the intrinsic importance of education and its important role in economic growth,
a causal link has long been established between education and a range of health outcomes
(Lynch 2003, Chou 2010). More recently it has been shown that for every year of increase in the
education of women of reproductive age, child mortality decreased by 9.5% (Gakidou 2010).
Therefore increasing access to education and retention at school is important not only to the
Many factors contribute to children’s access to education in developing countries, including:
gender, health status, family structure, poverty, distance to school, and perceived value of
education (Goodsmith, 2004). Being orphaned or made vulnerable can also play a role in
whether a child goes to school. Studies have indicated that where national school attendance is
high, there is little difference between OVC and non OVC in terms of school attendance.
However, in 38 of 59 surveys reviewed (most in Sub‐Saharan Africa), low school attendance was
the norm, and in these settings orphans were significantly less likely to attend school (Akwara,
2010). Equally troubling is that Africa has one of the lowest education completion rates in the
world. In one out of four African countries, half the children enrolled at the end of primary
school do not continue to the secondary level (UNESCO, 2006). The low transition rate into
secondary schools is partially a result of caregivers not being able to afford to pay for a child’s
education. Secondary schools are rarely free and students must pay for tuition as well as other
school related expenditure such as books, uniforms, and examination fees.
Various mechanisms of funding are applied by USAID and other development partners to
support the education of orphans and vulnerable children. These include scholarships and block
grants. Scholarships are individual payments made to, or on behalf of, a child to pay for part or
all of their education. Block grants are fixed‐sum grants to local or regional educational systems
or institutions that give the recipient organization broad flexibility to design and implement
designated programs. Scholarships are usually given to the family or a trustee on behalf of the
child, to pay for school fees. Block grants are paid in‐advance to the school and use the
attraction of a lump sum of money to leverage admission to school for less than the regular
admission fee. In most cases, with the payment of a block grant, a certain number of children
are guaranteed admission to the school under a contractual basis.
While both mechanisms have been used in various developing countries, their relative impact
on both the quantity and quality of education for OVC has not been well evaluated. It is also not
clear when it is appropriate to apply one or the other. The goal of this research was to compare
educational access that results from implementation of the two approaches in secondary
schools among OVC. The research also provides information on the ability of each approach to
influence educational outcomes.
Beginning in 2005, many large Non Governmental Organizations (NGOs) were awarded grants
by the United States Agency for International Development (USAID) under so‐called “Track
One” funding to provide core