ealigning
griculture
to
mprove
utrition
RAIN PROJECT: IMPACT
EVALUATION REPORT
An impact evaluation report prepared by the International Food
Policy Research Institute of the Realigning Agriculture to Improve
Nutrition (RAIN) project in Zambia
May 2016
Jody Harris, Phuong Hong Nguyen, John Maluccio,
Adam Rosenberg, Lan Tran Mai, Wahid Quabili, Rahul Rawat
Acknowledgements
A large community-based study like this could not be completed without the coordinated efforts of many people.
The International Food Policy Research Institute (IFPRI) team thanks all those who facilitated every step of this process:
Concern Worldwide staff in Dublin, particularly the Strategy Advocacy and Learning (SAL) advisors and the Zambia liaisons within the International Programmes Department, for their support and commitment to the RAIN project and its evaluation.
Concern Worldwide staff in Zambia for their constant support for our work. In particular, Gudrun Stallkamp who provided critical overall support to this evaluation, and who provided inputs in selecting sites for the evaluation, commented on several pieces of this survey including the evaluation design, ethics approval application, questionnaire revisions, and for connecting the evaluation team with other key personnel at Concern Worldwide; this survey could not have been completed with her support
Mumbwa Child Development Agency (MCDA) for their support in implementing the project
Palm Associates for facilitation of administrative and logistical support to the survey fieldwork.
The Government of Zambia health and agriculture officials in Mumbwa District and Central Province who provided support to this survey and to the RAIN project
The National Food and Nutrition Commission for loaning the survey team anthropometric equipment, and for invaluable support to anthropometric training of enumerators
The entire survey team from Palm Associates from both survey rounds, including the research assistants, field supervisors, fieldworkers, and data entry and management team
Finally, to the women and children that belong to the households who participated in the surveys, we thank you for your role in making this a success. Your contributions of time and information, and your dreams for a more healthy future for your children are at the very heart of this project.
The RAIN project was funded by Irish Aid and the Kerry Group with additional support from the Bank of Ireland. Funding for the evaluation was provided by Concern Worldwide, through grants received from Irish Aid, Kerry Group, and PATH through
support provided by the UK Government’s Department for International Development. Additional support for the evaluation was provided by the CGIAR research program on
Agriculture for Nutrition and Health, led by IFPRI.
The views expressed herein are those of the authors and in no way can be taken to reflect the official opinions of the funding organisations.
Acronyms
BMI Body Mass Index
CSA Census Supervisory Area
CSO Central Statistics Office
CWW Concern Worldwide
DHS Demographic and Health Survey
DPT Diphtheria/Polio/Tetanus
ENA Essential Nutrition Actions
FANTA Food and Nutrition Technical Assistance HAZ Height-for-age z-score
HDDS Household Dietary Diversity Score
HHS Household Hunger Scale
IFPRI International Food Policy Research Institute IYCF Infant and Young Child Feeding
NFNC National Food and Nutrition Commission
NGO Non-governmental Organization
OPV Oral Polio vaccine
RAIN Realigning Agriculture to Improve Nutrition
SEA Supervisory Enumeration Area
WAZ Weight-for-age z-score
WHO World Health Organization
Table of contents
EXECUTIVE SUMMARY IX
1 INTRODUCTION 20
1.1 OVERVIEW OF THE NUTRITION SITUATION IN ZAMBIA 20
1.2 DESCRIPTION OF THE RAINPROJECT 21
1.3 EVALUATION OBJECTIVES 22
1.4 STRUCTURE OF THE REPORT 23
2 METHODS 24
2.1 EVALUATION DESIGN 24
2.1.1 RATIONALE FOR AGE RANGE TO DETECT IMPACTS ON STUNTING 25
2.1.2 RANDOMIZATION PROCESS 26
2.1.3 SAMPLE SIZE ESTIMATE 27
2.1.4 SAMPLING METHODOLOGY 28
2.2 SURVEY INSTRUMENTS 29
2.2.1 CONCEPTUAL BASIS FOR THE IMPACT EVALUATION QUESTIONNAIRE 29
2.2.2 HOUSEHOLD QUESTIONNAIRE 30
2.2.3 ANTHROPOMETRIC MEASUREMENT 30
2.3 DATA COLLECTION AND MANAGEMENT 31
2.3.1 TRAINING OF PERSONNEL 31
2.3.2 SURVEY TEAM COMPOSITION 32
2.3.3 DATA MANAGEMENT 32
2.3.4 SURVEY INFORMATION 33
2.4 DATA ANALYSIS 33
2.4.1 EXPOSURE TO THE PROGRAM 36
2.4.2 IMPACT ESTIMATES ON MAIN AND SECONDARY OUTCOMES 36
2.4.3 DECOMPOSITION ANALYSIS 37
2.5 ETHICAL APPROVAL 37
3 RESULTS: BASELINE CHARACTERISTICS 39
3.1 KEY BASELINE INDICATORS 39
4 RESULTS: RAIN INTERVENTION EXPOSURE 41
4.1 KEY PROGRAM EXPOSURE INDICATORS 41
5 RESULTS: IMPACT OF RAIN INTERVENTIONS ON ANTHROPOMETRIC OUTCOMES 47 5.1 MAIN IMPACT ANALYSIS OF RAIN INTERVENTIONS ON ANTHROPOMETRIC OUTCOMES 47
5.1.1 INTENT-TO-TREAT OUTCOMES FOR ANTHROPOMETRIC OUTCOMES AMONG CHILDREN 24-59.9 MONTHS OF AGE 48 5.1.2 INTENT-TO-TREAT OUTCOMES FOR ANTHROPOMETRIC OUTCOMES AMONG CHILDREN 6-23.9 MONTHS OF AGE 50 5.2 PLAUSIBILITY ANALYSIS 1:HIGH POTENTIAL FOR EXPOSURE AGE GROUPS (24-47.9 MONTHS) 52 5.3 PLAUSIBILITY ANALYSIS 2:DOSE-RESPONSE RELATIONSHIP BETWEEN PROGRAM EXPOSURE CHILD NUTRITION 56 5.4 PLAUSIBILITY ANALYSIS 3:CHANGES IN THE UNDERLYING DETERMINANTS OF CHILD GROWTH AND NUTRITION 60
5.4.1 CHILD CHARACTERISTICS 60
5.4.2 MATERNAL CHARACTERISTICS:DEMOGRAPHIC, HEALTH SEEKING BEHAVIOR, AND NUTRITIONAL STATUS 64 5.4.3 HOUSEHOLD CHARACTERISTICS:FOOD SECURITY, DIETARY DIVERSITY, SOCIOECONOMIC STATUS, AND ACCESS TO
SERVICES 72
6 RESULTS: IMPACT OF RAIN INTERVENTIONS ON IYCF PRACTICES 79
6.1 MAIN IMPACT ANALYSIS OF RAIN INTERVENTIONS ON IYCF OUTCOMES 79
6.2 PLAUSIBILITY ANALYSIS:DOSE-RESPONSE RELATIONSHIP BETWEEN PROGRAM EXPOSURES AND IYCF PRACTICES 86 7 RESULTS: IMPACT OF RAIN INTERVENTIONS ON KNOWLEDGE AMONG CAREGIVERS 89 7.1 IMPACT RAIN PACKAGE OF INTERVENTION ON NUTRITION AND HYGIENE KNOWLEDGE 89 8 RESULTS: IMPACTS OF RAIN INTERVENTIONS ON THE WOMEN’S EMPOWERMENT 96
8.1 WOMEN’S SOCIAL EMPOWERMENT 97
8.2 WOMEN’S ECONOMIC EMPOWERMENT 102
8.3 WOMEN’S EMPOWERMENT IN AGRICULTURE AT ENDLINE 107
9 RESULTS: IMPACT OF RAIN INTERVENTIONS ON PRODUCTION OF NUTRIENT-RICH FOODS 110
9.1 FOOD PRODUCTION 110
10 RESULTS: DECOMPOSITION ANALYSIS OF THE DETERMINANTS OF CHANGES IN CHILD GROWTH
OUTCOMES OVER TIME 118
10.1 DETERMINANTS OF CHILD GROWTH 118
11 DISCUSSION 134
12 APPENDICES 140
APPENDIX 1:DETAILED SAMPLE SIZE CALCULATIONS 140
APPENDIX 2:VARIABLES INCLUDED IN THE DECOMPOSITION ANALYSIS 145
APPENDIX 3:DETAIL ON THE RAIN PROJECT 147
APPENDIX 4: COMPARING RAIN AND DHS 150
APPENDIX 5:EXPLAINING THE IMPACT OF RAIN:PROGRAM IMPACT PATHWAYS 151
List of tables
TABLE 2.1.1.SAMPLE SIZES ... 28
TABLE 2.1.1.SAMPLE SIZES USED FOR ANALYSES ... 34
TABLE 2.4.1:ANALYSIS METHODS- USE AND INTERPRETATION ... 35
TABLE 3.1.1BASELINE CORE IMPACT INDICATORS BY PROGRAM GROUP ... 39
TABLE 3.1.2.SELECTED BASELINE MATERNAL UNDERLYING FACTORS BY PROGRAM GROUP ... 40
TABLE 3.1.3.SELECTED BASELINE HOUSEHOLD UNDERLYING FACTORS BY PROGRAM GROUP ... 40
TABLE 4.1.1PROGRAM EXPOSURE:PARTICIPATION AND DELIVERY ... 42
TABLE 4.1.2 PROGRAM PARTICIPATION ... 43
TABLE 4.1.3. PROGRAM PARTICIPATION (AMONG MOTHERS WITH CHILDREN <2 YEARS OLD) ... 43
TABLE 4.1.4. SMF AND CHV INTERACTION ... 44
TABLE 4.1.5. SMF AND CHV INTERACTION (AMONG MOTHERS WITH CHILDREN <2 YEARS OLD) ... 44
TABLE 4.1.6.EXPOSURE TO BROADER RAIN PROJECT EVENTS AND MATERIALS ... 44
TABLE 4.1.7.EXPOSURE TO BROADER RAIN PROJECT EVENTS AND MATERIALS (AMONG MOTHERS WITH CHILDREN <2 YEARS OLD) ... 44
TABLE 4.2.1NGO PRESENCE, BY PROGRAM GROUP ... 45
TABLE 4.2.2SOCIAL CAPITAL CLUBS SUPPORTED BY NGOS, BY PROGRAM GROUP ... 46
TABLE 4.2.3INCOME GENERATING ACTIVITIES SUPPORTED BY NGOS, BY PROGRAM GROUP ... 46
TABLE 4.2.4TRAINING ACTIVITIES SUPPORTED BY NGOS, BY PROGRAM GROUP ... 46
TABLE 5.1.1.ANTHROPOMETRIC INDICATORS AMONG CHILDREN 24-59.9 MONTHS OF AGE, BY PROGRAM GROUP AND SURVEY ROUND . 49 TABLE 5.1.2ANTHROPOMETRIC INDICATORS AMONG CHILDREN 6-23.9 MONTHS OF AGE BY PROGRAM GROUP AND SURVEY ROUND ... 51
TABLE 5.2.1.ANTHROPOMETRIC INDICATORS AMONG CHILDREN 24-47.9 MONTHS OF AGE BY PROGRAM GROUP AND SURVEY ROUND .. 54
TABLE 5.2.2.ANTHROPOMETRIC INDICATORS AMONG BENEFICIARIES WITH CHILDREN 24-47.9 MONTHS, BY PROGRAM GROUP AND SURVEY ROUND... 55
TABLE 5.3.1.ANTHROPOMETRIC INDICATORS AMONG BENEFICERIES WITH CHILDREN 24-59.9 MONTHS OF AGE BY PROGRAM GROUP AND SURVEY ROUND... 58
TABLE 5.3.2.ASSOCIATION BETWEEN PROGRAM EXPOSURE WITH CHILD STUNTING AMONG CHILDREN 24-59.9 M ... 59
TABLE 5.4.1.CHILD IMMUNIZATION AND SUPPLEMENTATION STATUS AMONG CHILDREN 0-23.9 MONTHS, BY PROGRAM GROUP AND SURVEY ROUND... 61
TABLE 5.4.2.CHILD IMMUNIZATION AND SUPPLEMENTATION STATUS AMONG CHILDREN 24-59.9 MONTHS, BY PROGRAM GROUP AND SURVEY ROUND... 62
TABLE 5.4.3.CHILD MORBIDITY, BY PROGRAM GROUP AND SURVEY ROUND ... 63
TABLE 5.4.4.SELECTED BASELINE MATERNAL UNDERLYING FACTORS, BY PROGRAM GROUP AND SURVEY ROUND ... 65
TABLE 5.4.5TIME ALLOCATION, BY PROGRAM GROUP ... 66
TABLE 5.4.6TIME ALLOCATION AMONG BENEFICIARIES, BY PROGRAM GROUP ... 67
TABLE 5.4.7USE OF PRENATAL CARE, BY PROGRAM GROUP AND SURVEY ROUND ... 68
TABLE 5.4.8MATERNAL DIETARY DIVERSITY, BY PROGRAM GROUP AND SURVEY ROUND (<24 MO) ... 69
TABLE 5.4.9MATERNAL DIETARY DIVERSITY AMONG BENEFICIARIES, BY PROGRAM GROUP AND SURVEY ROUND ... 70
TABLE 5.4.11WOMEN’S NUTRITIONAL STATUS AMONG BENEFICIARIES, BY PROGRAM GROUP AND SURVEY ROUND ... 71
TABLE 5.4.12HOUSEHOLD PERCEPTIONS OF FOOD INSECURITY, BY PROGRAM GROUP AND SURVEY ROUND ... 73
TABLE 5.4.13HOUSEHOLD PERCEPTIONS OF FOOD INSECURITY AMONG BENEFICIARIES, BY PROGRAM GROUP AND SURVEY ROUND ... 74
TABLE 5.4.14HOUSEHOLD DIETARY DIVERSITY, BY PROGRAM GROUP AND SURVEY ROUND ... 75
TABLE 5.4.15HOUSEHOLD DIETARY DIVERSITY AMONG BENEFICIARIES, BY PROGRAM GROUP AND SURVEY ROUND ... 76
TABLE 5.4.16HOUSEHOLD ASSET OWNERSHIP, BY PROGRAM GROUP AND SURVEY ROUND ... 77
TABLE 5.4.17HOUSEHOLD ACCESS TO SERVICES, BY PROGRAM GROUP AND SURVEY ROUND ... 78
TABLE 5.4.18HOUSEHOLD SOCIOECONOMIC STATUS, BY PROGRAM GROUP AND SURVEY ROUND ... 78
TABLE 6.1.1.WHO RECOMMENDED IYCF INDICATORS, BY PROGRAM GROUP AND SURVEY ROUND ... 81
TABLE 6.1.2.DIFFERENCE IN DIFFERENCE OF WHO RECOMMENDED IYCF INDICATORS, BY PROGRAM GROUP AND SURVEY ROUND ... 82
TABLE 6.1.3.PRE-LACTEAL FEEDING AMONG CHILDREN 0-23 MONTHS OLD, BY PROGRAM GROUP AND SURVEY ROUND ... 83
TABLE 6.1.4.FOOD GROUPS CONSUMED IN THE PAST 24 HOURS AMONG CHILDREN 6-23 MONTHS OLD, BY PROGRAM GROUP AND SURVEY ROUND ... 84
TABLE 6.1.5.TIMELINESS OF INTRODUCTION OF COMPLEMENTARY FOODS AMONG CHILDREN 6-23 MONTHS OLD, BY PROGRAM GROUP AND SURVEY ROUND ... 85
TABLE 6.1.6.REPORTED MEAL FREQUENCY AMONG CHILDREN 6-23 MONTHS OLD, BY PROGRAM GROUP AND SURVEY ROUND ... 85
TABLE 6.2.1 WHO RECOMMENDED IYCF INDICATORS AMONG BENEFICIARIES (THOSE ARE THE MEMBER OF A RAIN’S WOMEN’S GROUP) BY PROGRAM GROUP AND SURVEY ROUND ... 87
TABLE 6.2.2 DIFFERENCE IN DIFFERENCE OF WHO RECOMMENDED IYCF INDICATORS AMONG BENEFICIARIES (THOSE ARE THE MEMBER OF A RAIN’S WOMEN’S GROUP) ... 88
TABLE 7.1.1.KNOWLEDGE ABOUT BF AMONG MOTHERS OF CHILDREN 0-23.9 MONTHS OF AGE, BY PROGRAM AREA ... 90
TABLE 7.1.2.KNOWLEDGE ABOUT BF AMONG MOTHERS WHO ARE MEMBERS OF RAIN GROUP WITH CHILDREN 0-23.9 MONTHS OF AGE, BY PROGRAM AND SURVEY ROUND... 91
TABLE 7.1.3.REPORTED KNOWLEDGE ON TIMELINESS OF INTRODUCTION OF COMPLEMENTARY FOODS AMONG MOTHERS WITH CHILDREN 0-23.9 MONTHS, BY PROGRAM GROUP AND SURVEY ROUND ... 92
TABLE 7.1.4.REPORTED KNOWLEDGE ON FEEDING DURING ILLNESS AMONG MOTHERS WITH CHILDREN 0-23.9 MONTHS, BY PROGRAM GROUP AND SURVEY ROUND ... 93
TABLE 7.1.5REPORTED EXPOSURE TO INFORMATION, BY PROGRAM GROUP... 94
TABLE 7.1.6.REPORTED KNOWLEDGE OF HYGIENE PRACTICES, BY PROGRAM GROUP ... 95
TABLE 8.1.1.RELATIONSHIP WITH SPOUSES, BY PROGRAM GROUP AND SURVEY ROUND ... 99
TABLE 8.1.2.PERCEPTION OF EQUALITY, BY PROGRAM GROUP AND SURVEY ROUND ... 99
TABLE 8.1.3.DECISION MAKING POWER, BY PROGRAM GROUP AND SURVEY ROUND ... 100
TABLE 8.1.4.WOMEN’S SOCIAL CAPITAL, BY PROGRAM GROUP AND SURVEY ROUND ... 101
TABLE 8.2.1.FINANCIAL EMPOWERMENT, BY PROGRAM GROUP AND SURVEY ROUND ... 102
TABLE 8.2.2.ACCESS TO ASSETS AND ABILITY TO SELL ASSETS, BY PROGRAM GROUP AND SURVEY ROUND ... 103
TABLE 8.2.3.PURCHASING DECISIONS, BY PROGRAM GROUP AND SURVEY ROUND ... 104
TABLE 8.2.4.SUMMARY KEY DOMAINS OF WOMEN EMPOWERMENT (ALL WOMEN), BY PROGRAM GROUP AND SURVEY ROUND ... 105
TABLE 8.2.5.SUMMARY KEY DOMAINS OF WOMEN EMPOWERMENT (AMONG RAIN BENEFICERIES), BY PROGRAM GROUP AND SURVEY ROUND ... 106
TABLE 8.3.1WOMEN'S EMPOWERMENT IN AGRICULTURE, BY PROGRAM GROUP AND SURVEY ROUND ... 108
TABLE 8.3.2WOMEN'S EMPOWERMENT IN AGRICULTURE AMONG RAIN BENEFICIARIES, BY PROGRAM GROUP AND SURVEY ROUND .... 109
TABLE 9.1.1NUMBER OF FIELD CROPS CULTIVATED, BY PROGRAM GROUP AND SURVEY ROUND ... 112
TABLE 9.1.2NUMBER OF VEGETABLES OR FRUITS CULTIVATED, BY STUDY ARM AND SURVEY ROUND ... 113
TABLE 9.1.3REARING ANIMALS AND PRODUCTION OF ANIMAL SOURCE FOODS, BY PROGRAM GROUP AND SURVEY ROUND ... 114
TABLE 9.1.4PRODUCTION OF SEVEN DIFFERENT FOOD GROUPS, BY PROGRAM GROUP AND SURVEY ROUND ... 115
TABLE 9.1.5DIFFERENCE IN DIFFERENCE OF FOOD PRODUCTION INDICATORS, BY PROGRAM GROUP AND SURVEY ROUND ... 116
TABLE 9.1.6DIFFERENCE IN DIFFERENCE OF FOOD PRODUCTION INDICATORS IN HOUSEHOLDS PARTICIPATING IN RAIN, BY PROGRAM GROUP AND SURVEY ROUND ... 117
TABLE 10.1.2CHANGES IN THE MEANS OR PERCENT OF KEY VARIABLES IN ZAMBIA 2011-2015(SAMPLES OF CHILDREN 24-59.9
MONTHS) ... 124
TABLE 10.1.3HEIGHT-FOR-AGE Z REGRESSIONS BY ROUND WITH TEST FOR COEFFICIENT DIFFERENCES OVER TIME AMONG CHILDREN 24– 59.9 MONTHS ... 125
TABLE 10.1.4STUNTING REGRESSIONS BY ROUND WITH TEST FOR COEFFICIENT DIFFERENCES OVER TIME AMONG CHILDREN 24–59.9 MONTHS IN ZAMBIA ... 126
TABLE 10.1.5WEIGHT-FOR-HIGHT Z REGRESSIONS BY ROUND WITH TEST FOR COEFFICIENT DIFFERENCES OVER TIME AMONG CHILDREN 24 –59.9 MONTHS ... 127
TABLE 10.1.6WASTING REGRESSIONS BY ROUND WITH TEST FOR COEFFICIENT DIFFERENCES OVER TIME AMONG CHILDREN 24–59.9 MONTHS ... 128
TABLE 10.1.7DECOMPOSING SOURCES OF HAZ AND STUNTING CHANGE BY SURVEY ROUND AMONG CHILDREN 24-59.9 MONTHS IN ZAMBIA ... 130
TABLE 10.1.8DECOMPOSING SOURCES OF WHZ AND WASTING CHANGE BY SURVEY ROUND AMONG CHILDREN 24-59.9 MONTHS IN ZAMBIA ... 130
TABLE 10.1.9DECOMPOSING SOURCES OF HAZ AND STUNTING CHANGE BY SURVEY ROUND AMONG CHILDREN 0-59.9 MONTHS IN ZAMBIA ... 132
TABLE 10.1.10DECOMPOSING SOURCES OF WHZ AND WASTING CHANGE BY SURVEY ROUND AMONG CHILDREN 0-59.9 MONTHS IN ZAMBIA ... 133
TABLE 10.1.1DEFINITIONS OF VARIABLES USING IN THE ANALYSES ... 145
TABLE 1PROCESS EVALUATION SAMPLING ... 151
List of figures
FIGURE 2.1.RAINEVALUATION DESIGN ... 25FIGURE 2.2.MAP OF RANDOMIZED INTERVENTION AREAS (CONTROL NOT SHOWN) ... 27
FIGURE 5.1.PREVALENCE OF STUNTING, UNDERWEIGHT AND WASTING AMONG CHILDREN 24-59.9 MONTHS BY PROGRAM GROUP AND SURVEY ROUND... 48
FIGURE 5.2.PREVALENCE OF STUNTING, UNDERWEIGHT AND WASTING AMONG CHILDREN 6-23.9 MONTHS BY PROGRAM GROUP AND SURVEY ROUND... 50
FIGURE 5.3.PREVALENCE OF STUNTING, UNDERWEIGHT AND WASTING AMONG CHILDREN 24-47.9 MONTHS BY PROGRAM GROUP AND SURVEY ROUND... 53
FIGURE 5.4.PREVALENCE OF STUNTING, UNDERWEIGHT AND WASTING AMONG BENEFICERIES WITH CHILDREN 24-59.9 MONTHS BY PROGRAM GROUP AND SURVEY ROUND ... 57
FIGURE 10.1HAZ BY CHILD AGE,RAIN PROGRAM AREAS 2011 AND 2015 ... 119
FIGURE 10.2WHZ BY CHILD AGE,RAIN PROGRAM AREAS 2011 AND 2015 ... 119
FIGURE 10.3TRENDS IN STUNTING AND WASTING PREVALENCE, BY CHILD AGE AND SURVEY TIME IN ZAMBIA ... 121
FIGURE 10.4DISTRIBUTIONS OF CHILD HAZ SCORES AMONG CHILDREN 24-59.9 MONTHS IN ZAMBIA BY SURVEY ROUND ... 122
FIGURE 10.5DISTRIBUTIONS OF CHILD WHZ SCORES AMONG CHILDREN 24-59.9 MONTHS IN ZAMBIA BY SURVEY ROUND ... 122
FIGURE 12.1COMPARISON OF STUNTING AND WASTING BETWEEN RAIN AND DHS ... 150
Executive summary
RAIN project model
Almost half of children in Zambia are stunted, and reduction of chronic malnutrition is prioritized in nutrition policies and programs in the country. The Realigning Agriculture to Improve Nutrition (RAIN) project was a partnership between Concern Worldwide (CWW), Mumbwa Child Development Agency (MCDA) and the International Food Policy Research Institute (IFPRI), aiming to design, implement and evaluate a model of multi-sectoral integration of interventions to reduce the prevalence of chronic malnutrition in Mumbwa district, in the Central Province of Zambia. A key component of the RAIN project was to document evidence of both impact and process for application in other contexts and at scale, through a rigorous evaluation design. Project partners over the course of the 5-year project have included the Mumbwa District Child Development Agency (MCDA), Women for Change (WFC), the Zambian Ministry of Agriculture and Livestock (MAL), the Zambian Ministry of Health (MoH), and the Zambian Ministry of Community Development Maternal and Child Health (MCDMCH).
The project targeted children during the critical period from conception through 24 months of age, roughly equivalent to the first 1,000 days of life, through integrated agriculture, nutrition and health community based interventions. The overall approach focused on addressing the multi-sectoral causes of malnutrition and on learning how to effectively tackle the challenges of inter-sectoral collaboration. The RAIN project comprised of a district-level agriculture intervention to increase year round availability of, and access to, nutrient rich foods at the household level, in some areas accompanied by promotion of optimal health, nutrition, and care seeking behavior through the delivery of social behavior change communication. Most program elements were delivered through local women’s groups created by the program, led by a female Smallholder Model Farmer (SMF) nominated by her group to receive
agricultural training and inputs and pass these on to the group during monthly meetings. In the nutrition and health areas, groups were also linked to an existing Community Health Volunteer (CHV) who
received additional training in nutrition topics to pass on to the group. In addition, some community-wide gender sensitization and information activities were undertaken.
The RAIN project and its impact evaluation were designed to help address a critical gap in the evidence base regarding the degree to which agricultural interventions, either alone or when combined with nutrition and health interventions, can improve child nutrition, and ultimately reduce the prevalence of
x
stunting in young children. It aimed to establish ‘proof of concept’ for an intervention model that can then be replicated and scaled up within Zambia and beyond.
RAIN evaluation
Objectives
The primary evaluation objective of this evaluation was to:
1) Assess the impact of the two different RAIN intervention packages on stunting, among children 24 -59 months of age.
In addition to the primary evaluation objective, the secondary evaluation objectives were to assess the impact of the RAIN package of interventions on:
2) Core WHO infant and young child feeding (IYCF) indicators among children 0-23 months of age
3) Health and nutrition knowledge among caregivers of young children
4) Different domains of women’s empowerment
5) Agricultural production, and in particular, the availability of, and access to, a year-round supply of diverse and nutritious foods
Evaluation Design
A fully randomized evaluation design to evaluate the impact of RAIN was determined to be not feasible, primarily for practical project implementation reasons, and therefore a hybrid design was adopted that combines a cluster randomized probability design comparing the two RAIN intervention packages, with a plausibility design that compares the RAIN intervention arms to a non-randomized control group. Randomization was carried out at the level of the census supervisory area (CSA), a sampling unit used by the Central Statistics Office of Zambia. This design yielded 3 different study arms:
1. The Agriculture only (Ag-only) group, which included agricultural interventions implemented by Concern Worldwide and its partners
2. The Agriculture-Nutrition (Ag-Nutrition) group, which included both agriculture and nutrition/ health interventions implemented by Concern Worldwide and its partners
3. The Control group, which had access to standard government agriculture and health services, and where Concern Worldwide carried out no implementation activities
xi
Sample size estimates were powered to detect a reduction in the prevalence of stunting (the primary impact indicator) of 8 percentage points, and a 0.2 Z-score difference in the mean height-for-age Z-score (HAZ) between any two study groups. The study was also powered to detect appreciable impacts in the secondary impact indicators. At baseline, we sampled approximately 1000 households per arm with a child aged 24-59 months of age, for a total sample size of 3044 households. At endline, we oversampled the two RAIN interventions arms, by approximately 20% (for an approximately sample size of 1200 households per arm), to account for potential limited intervention exposure at the household level, for a total sample size of 3536 households.
Repeated cross-sectional baseline (2011) and endline (2015) surveys were conducted in the same communities over time, at the same time of the year (July-August). Data collection included a household questionnaire and anthropometric measurements, collecting information on nutrition outcomes and determinants of malnutrition at the child, maternal and household level.
Evaluation Analyses
Three broad sets of analyses were conducted:
1) Estimation of impact of the RAIN intervention arms (Ag-only, and Ag-Nutrition), compared to the control, and to each other, on the primary impact indicator i.e. stunting. This included:
a. Estimation of main impact of RAIN interventions on prevalence of stunting, and mean HAZ scores, using difference-in-difference (DID) estimates
b. A series of additional analyses to assess the plausibility of identified impacts. These additional analyses included the following:
Plausibility analysis 1: Analysis of the change in stunting prevalence among children in high potential-for-impact age group i.e. among children 24-47.9 months of age
Plausibility analysis 2: Dose-response analysis between program exposure and child growth outcomes, thereby creating an internal comparison group to test program effects with greater degree of confidence.
Plausibility analysis 3: Analysis of change in determinants of stunting over time. 2) Estimation of impact of RAIN interventions on secondary outcomes: Infant and young child feeding
practices; maternal nutrition and health knowledge; women’s empowerment; and agricultural production.
xii
3) Decomposition analysis to examine various social, behavioral and economic factors as potential drivers of change in linear growth and stunting over time.
All major impact analyses were conducted using: 1) intent-to-treat analysis, where all sampled households are included in the analysis, regardless of whether they were actually exposed to the RAIN project; as well as by 2) per-protocol analysis, where analysis is restricted to those households
confirmed to be RAIN project beneficiaries in the two intervention arms.
Results
Baseline randomization
At baseline, there were no statistically significant differences among the three study arms on
anthropometric outcomes (HAZ, WHZ, stunting and wasting). Most IYCF indicators were comparable across the three study groups; the exceptions were for early initiation of breastfeeding and meal frequency, which were lower in the Ag-Nutrition group. Additionally, maternal and household characteristics, were also comparable among the three groups at baseline, with no major statistically significant difference between groups. These results (presented in detail in the RAIN baseline report) indicate that randomization at baseline was successful.
Household exposure to RAIN interventions
Key to interpreting the impact results is understanding intended exposure to the project interventions. This comprises participation (whether and to what extent a household was participating in RAIN project components) and delivery (RAIN implementation being received as planned). Overall participation was 31 percent in the Ag-only group, and 34 percent in the Ag-nutrition group. Of this third of all eligible households participating, the intensity of program delivery varied, with approximately 50% of all households receiving medium or high levels of program delivery. There was no difference in reported participation when restricting the exposure analyses to households with a child < 2 years of age, when sampled at endline. In terms of intensity of delivery, SMF attendance at RAIN groups was high
(approximately 90%), but CHV attendance was low (38-45%); group members therefore had more opportunity to interact with trainers from the agriculture side than from the health side. The additional home visits, aimed at providing one-to-one support for gardening and IYCF counselling and support, were more limited still; SMF home visits were not happening as often as planned (45%, and 53% of households were visited by an SMF in 2015, in the Ag-Nutrition and Ag-only arms, respectively), and CHV visits even less often (13% in the Ag - Nutrition arm). In terms of spillover between study arms, leakage
xiii
of CHV participation to the Ag-only group was fairly common as some clinics where RAIN-trained CHVs were based were sited on the border between project areas, but leakage of the project to control areas was not common, with very few women in this area reporting participation. The overall pattern and magnitude of these results did not differ between all households sampled, and households with a child < 2 years of age, when sampled at endline.
There was no difference between intervention and control areas in presence of other non-governmental organization (NGO)-supported projects or activities. We can therefore conclude that the control group was functioning as a valid comparator to the intervention arms. We do not have data on the presence of government services in these areas, although there is no reason to believe that there was any difference in government services across study groups.
PROGRAM IMPACTS:
I.
Nutritional status
Over time, between baseline and endline, the prevalence of stunting decreased significantly in all three study groups in the impact evaluation age range of children 24-59 months of age. However, there was a differential decline in stunting in the three study groups, in favor of the control group, with a
significantly greater decrease, compared to the two RAIN interventions groups. DID impact estimates suggest a negative impact of the RAIN intervention groups on stunting, ranging from +7pp for the Ag-only group, to +9pp for the Ag-Nutrition group, compared to the control group, respectively. Similar negative impacts on mean HAZ scores were seen for the Ag-only group (-0.35 HAZ, compared to the control group).
Due to the aging of children throughout the duration of the RAIN project, there was greater potential for certain age groups of children to be exposed to the RAIN interventions for longer than others. Caregivers of children 24 - 48 months of age at the time of the endline survey had the opportunity to have been enrolled in the program for the entire 1000 days period between conception and age two. Therefore, we conducted impact analyses in this age group of children. Similar declines were observed in stunting, over time, for all three study group, but no differential reductions in the prevalence of stunting were observed. This suggests a null impact of the RAIN project interventions on stunting, among those with greatest potential for exposure during the 1000 day window of opportunity.
xiv
Since fewer households than expected participated in the RAIN project, we undertook “per-protocol” analyses, specifically restricting the impact analyses to households who reported a member having joined a RAIN women’s group.We observe similar negative impacts on stunting.
Levels of wasting increased significantly over time, for all three study groups, in pure intent to treat analyses among: 1) children 24-59 months of age at endline, 2) children with greater potential for exposure (24-47 months of age, at endline), and in per-protocol analyses for: 3) children 24-59 months of age at endline, and 4) children with greatest potential for exposure. In DID analyses, there was a consistent positive impact on the prevalence of wasting, in the Ag-only arm, compared to the control arm, of about -4 pp. This suggests an overall protective effect on wasting, of the Ag-only study arm.
Maternal level underlying determinants of child nutrition
We assessed caregiver demographics, health seeking behavior, time use, and nutritional status as underlying determinants of child nutritional status. Overall, maternal characteristics were similar across program groups, at both baseline, and endline. At endline, women in the Ag-Nutrition study group had a higher prevalence of >4 prenatal visits and also had higher number of prenatal visits compared to control group. However, these differences were present at baseline as well. There were significant improvements in maternal dietary diversity (measured out of a total of 7 food groups) within study arms over time, in both intent-to-treat, and per-protocol analysis, but there were no differential changes over time, in favor of any study group, in impact analyses. Maternal Body Mass Index (BMI) increased over time in both the Ag + Nutrition and the Ag-only study groups, but these increases were not differential. Overall, in DID impact analyses, there was an increase of approximately 10pp in the proportion of overweight women in the Ag-only arm, compared to the control arm. There was a commensurate decrease in the proportion of women classified as of normal weight, in the Ag-only arm. In per-protocol analysis, there was a marginally significant positive program impact on maternal BMI of 0.57 kg/m2 in the Ag-only arm, compared to the control arm.
Household level underlying determinants of child nutrition
We assessed household food security, dietary diversity, socio-economic status, and access to services as underlying determinants of child nutrition. Overall, respondents’ perception of their household food security (using the household hunger scale) decreased significantly over time for all groups, in both the full sample, and among confirmed RAIN beneficiaries. In impact analyses, there was a significant decrease in the prevalence of “little to no hunger” in the Ag-Nutrition group, compared to the control group, and a significant increase in the level of “moderate hunger” in this group. This was the case in
xv
both intent-to-treat analysis, and among confirmed RAIN beneficiaries. There was a significant positive program impact on the level of “severe hunger”; DID impact estimates indicate reduction of between 4-7 pp in the Ag-only group, compared to the control, in both intent-to-treat, and per-protocol analyses. At the same time, the Ag-Nutrition group had significant positive impacts on household dietary diversity, with an increase of about 1 food group, based on a 12 food group scale, in both intent-to-treat, and per-protocol analyses. Overall, there were no significant differences in socio-economic status or access to services between groups over time.
II.
Infant and young child feeding
The second evaluation objective was to assess the impact of the RAIN package of interventions on core WHO infant and young child feeding (IYCF) indicators among children 0-23 months of age. At endline, all breastfeeding-related IYCF indicators were high across all three study groups, but complementary feeding practices were sub-optimal ranging from approximately 25-30% for the minimum acceptable diet, to 60% for the minimum meal frequency. Several of these IYCF indicators improved over time, within groups, such as early initiation of breastfeeding (ranging from a 24-31 pp increase over time), or complementary feeding indicators (with increases ranging from 6 to 12 pp for different indicators, in the RAIN intervention groups), but these increases were not differential in favor of any group. Of note, consumption of iron rich foods decreased over time (ranging from 13-15pp), in all groups. Overall, there was no attributable program impact on improving IYCF practices in both intent-to-treat and per protocol analyses. The only impact we found was the consumption of legumes/nuts which was higher in both intervention arms compared to control.
III.
Caregiver Health and Nutrition Knowledge
The third evaluation objective was to assess the impact of the RAIN package of intervention on health and nutrition knowledge among caregivers. Overall, IYCF knowledge increased over time, except for breastfeeding based on child’s demand and continue breastfeeing if mother is ill. The overall
improvements in breastfeeding knowledge over time was lower in the Ag-only group, when compared to the control group. As with complementary feeding, the knowledge of timely introduction of
xvi
was most notable for animal source foods (flesh foods and eggs, where there was an approximate 20pp greater knowledge regarding the appropriateness of feeding these foods to children 6-8 months of age).
Hygiene knowledge was significantly different among study arms at endline for many aspects relating to hand washing; protecting children from worms; and making drinking water safer, but the direction of difference was not consistently favorable to a single study arm, or to the RAIN intervention arms.
IV.
Women’s Empowerment
The fourth evaluation objective was to assess the impact of the RAIN package of intervention on women’s empowerment. We assessed impacts on eight social and economic domains of women’s empowerment, plus empowerment in agriculture. There were clear impacts of the RAIN interventions on different domains of women’s empowerment. In DID impact analyses, we observe significant program impacts in the Ag-only group, compared to the control group, on social capital, asset access, financial empowerment, perception of equality. The Ag—Nutrition group had significant program impacts, when compared to the control group on social capital only. Overall, the direction of impacts was similar in intent-to-treat, and per-protocol analyses, with slightly higher impacts in per-protocol analyses.
There was a clear shift over time in women’s involvement in decision making in agriculture. This shift occurred across all study groups for different aspects of decision making. The change was greater in the RAIN intervention groups compared to the control group, suggesting a clear impact of the RAIN
interventions on improving women’s empowerment in agriculture in both intent-to-treat, and per-protocol analysis.
V.
Agriculture Production
The fifth evaluation objective was to assess the impact of the RAIN agriculture package of interventions on the availability of and access to a year-round supply of diverse and micronutrient-rich plant and animal source foods at household level. Overall the RAIN interventions had a consistent significant attributable impact on several different dimensions of agricultural production and consequent
availability during the year of nutritious foods. Both the Ag-Nutrition and the Ag-only arms, had greater increases over time, compared to the control group, on the total number of foods produced , the total number of agricultural activities engaged in by the households, and the number of months producing Vitamin A rich foods, and dairy. Consistently, program effect sizes were approximately two-fold larger in per-protocol analyses.
xvii
Drivers of nutrition change
Finally, we conducted decomposition analysis to explain factors that may have contributed to the large reductions in stunting over time, and increase in wasting. Our analysis however explain only a small proportion of these changes. The model explains 8.4% and 6.3% of the actual change in HAZ scores and stunting prevalence, respectively, in our key age group (24-59 months) during this period. Among the sources of predicted change, receipt of nutrition counseling and reductions in child morbidity stand out as the largest factors, explaining the largest proportion of the predicted change in HAZ scores.
Unexpectedly, household hunger and agriculture production variables did not predict reductions in stunting prevalence or increases in HAZ scores. The model only explains 1% actual change for WHZ and 13% actual change for wasting. For children aged 0-59 months, the models explain 4-6% of actual change for HAZ/stunting and around 16% for WHZ and wasting. Similar analyses with several rounds of the Demographic and Health Survey (DHS) data from Zambia showed similar results.
Discussion
This report presents findings from a large, complex, multi-year, inter-sectoral project that combined agriculture and nutrition interventions to impact child nutrition. The RAIN project is one of a handful of projects that includes a rigorous randomized design to examine impact, and responds directly to recent calls for stronger evaluation designs of agriculture and nutrition interventions to strengthen the
evidence base on these links.
Overall, the RAIN project had mixed impacts. The project had: 1) consistently positive impacts on agricultural production, 2) impacts on different domains of women’s social and economic
empowerment, as well as women’s empowerment in agriculture, 3) impacts on household food security as measured by household dietary diversity, and 4) a potential protective effect on child wasting. In general, where there were significant program impacts, the magnitude of these impacts was larger in per-protocol analyses, among confirmed RAIN beneficiaries. There were however no discernable impacts on reducing the prevalence of stunting, on improving IYCF practices among young children, or on improving caregiver health and nutrition knowledge. There appears to be little to no additional benefit of the Ag-Nutrition arm, compared to the Ag-only intervention arm for the impacts achieved. This is further evidenced by greater exposure to the agricultural intervention components of the RAIN project, compared to the nutrition intervention components.
xviii
The low coverage achieved by the program is one possible factor contributing to the lack of consistent impacts across project objectives. The larger effect sizes in per-protocol analyses support this
hypothesis. As noted previously, only a third of eligible households surveyed had participated in the project through joining a women’s group—the main point of entry into the RAIN project. In addition, home visits by SMFs and CHVs, a critical component of the intervention delivery by the program, did not materialize as envisaged, losing a key one-to-one element that would bolster the project. Across both the women’s groups and home visits, it was clear that the agriculture frontline workers (SMFs) were more active than the health-side workers (CHVs), which plays out in the improved agriculture outcomes compared to nutrition and health outcomes in our evaluation. Only 12-16 percent of households had both high participation and high levels of program delivery; around half of households were
categoriezed as receiving medium or high level of delivery. The process evaluation (conducted in 2013 & 2014) suggested that the SMF, a position created specifically for the RAIN project, were initially more incentivized to act for RAIN through RAIN’s provision of incentives (agricultural inputs) than the CHV positions that already existed in the community and which did not receive similar incentives until after 2014, though both groups received additional training. In addition, CHVs serviced the entire community, whereas SMFs were working specifically with RAIN groups; also SMFs are a newly created positions, and the information they provide was new, whilst the CHVs were existing positions so women should have been exposed to many of the messages before. However, the project strengthened the knowledge of the CHVs and increased the number of CHVs in the area. It is clear from the evaluation that by and large, the Ag-only group fared better than the Ag-Nutrition group, suggesting no additional value of nutrition interventions in this project, implemented primarily by CHVs. As noted in the results section, per-protocol analysis (whereby analysis was restricted to those individuals that were confirmed RAIN beneficiaries) did not alter the main findings, though almost two-thirds of the sample was lost in this process. As such it is hard to make definitive claims about lack of impact being solely driven by low participation.
The impact evaluation design attempts to attribute changes in key impact indicators of interest to the RAIN interventions. As such, it is important to note a limitation of the design is its inability to accurately account for elements beyond the RAIN project interventions. For several indicators of interest, we observe changes over time in both the RAIN intervention groups, as well as the control group. While we are able to document the lack of leakage of formal program delivery of RAIN interventions to the control group, we are unable to document informal leakage of secondary intervention components (knowledge, agricultural inputs etc. provided by non-project staff) among peers or relatives across program study
xix
arms. Additionally, we are unable to adequately account for general improvements in government health services across the district, regardless of RAIN study group. There are clear improvements over time, between baseline and endline, in access and use of government-run health services across all study groups, as documented in this report. This is evident in increased access and use of prenatal care, as well as receipt of health and nutrition counseling services at government clinics. As such, it is
plausible that this increase in access and use directly impacts health and nutrition knowledge, and IYCF practices, components of prenatal counseling and under 5 clinics at government health centers. This increase may be sufficiently large to prevent detection and attribution of RAIN interventions, over and beyond general increases and secular trends.
There is limited evidence from other evaluations of similar programs. An evaluation of Helen Keller International’s (HKI) Enhanced Homestead Food Production (EHFP) intervention on Burkina Faso (one of only a very few rigorous impact evaluations of a similar agriculture-nutrition programs designed to improve child nutrition) demonstrated results that were similar for many, but not all, of the outcomes examined for the RAIN project (Olney, Pedehombga et al. 2015). The evaluation of the EHFP program in Burkina Faso found 1) positive impacts on wasting among children, 2) small positive impacts on anemia, 3) small positive impacts on maternal dietary diversity, 4) positive impacts on maternal underweight, and 5) positive impacts on several dimensions of women’s empowerment. Of note, the EHFP program had no impact on reducing the level of stunting, and no impact on food security.
Overall, the results from the RAIN evaluation contribute to alleviating some of the dearth of evidence from rigorous impact evaluations of integrated agriculture and nutrition programs. There are clear and important maternal and household level benefits of this program, which may be achieved in other, similar programs, and lessons learnt can be used to to scale-up the existing project. The clear and consistent impact of the RAIN intervention on agriculture production, and on women’s empowerment, two core objectives of the program, are noteworthy and consistent with the limited evidence to date from similar interventions.
20
1
Introduction
1.1
Overview of the nutrition situation in Zambia
Zambia ranks low on human development metrics, national income per capita, Gini coefficient, and other indices and measures of poverty and inequality. With a young, rural and very sparsely spread population, with low life expectancy and poor gender equity (Central Statistical Office, Ministry of Health et al. 2009; UNDP 2010), and fairly low agricultural productivity largely due to poverty and poor infrastructure, the Government sees little financial revenue. Production of maize is heavily promoted in Government policy and programs, and is the predominant cash and subsistence crop, with food security in Zambia generally equated to ‘maize security’ (Smale and Jayne 2009). The Global Hunger Index (K. von Grebmer, J. Bernstein et al. 2015) ranks Zambia as having an ‘extremely alarming’ hunger situation, highlighting major deficits in nutrition and child survival.
Zambia suffers from a high rate of under-five mortality at 75 per 1000 live births , though this is a substantial improvement since the last Demographic and Health Survey (DHS) and more than halved over the past 15 years (Central Statistical Office, Ministry of Health et al. 2014). An estimated 13 percent of adults 15-49 years old are HIV-positive (Central Statistical Office, Ministry of Health et al. 2014). Fifty-eight percent of children are fully vaccinated at 12 months, and only around half of all children who suffer from diarrhoea, fever or pneumonia received appropriate treatment (Central Statistical Office, Ministry of Health et al. 2014). Use of improved drinking-water sources is low (85% urban, 49% rural), as is access to improved sanitation (56% urban; 34% rural) (UNICEF 2015). Vitamin A supplementation is relatively high nationally (80 – 90 percent for children under one year), but pockets of poor coverage persist. Despite endemic malaria, only 40 percent of under-fives sleep under insecticide-treated nets. Primary health care is free in Zambia for pregnant women and children under five, although barriers to access still exist; fertility rate (6.2 per woman) is very high (Central Statistical Office, Ministry of Health et al. 2014).
Prevalence of undernutrition (stunting) in Zambia had increased since 1990, remaining stable at around 45 percent from 2000 to 2011 (WFP and FAO 2010); the most recent DHS survey shows a reduction in stunting to 40% nationally (Central Statistical Office, Ministry of Health et al. 2014). Six percent of children in Zambia are wasted and 15 percent are underweight; about 10 percent of women have a low body mass index indicating maternal underweight, and 23 percent are overweight (Central Statistical
21
Office, Ministry of Health et al. 2014). Infant and young child feeding practices are variable, with 73 percent of children under six months exclusively breastfed, but only 11 percent fed appropriately for their age (Central Statistical Office, Ministry of Health et al. 2014). In Central Province, in which Mumbwa district is located, 42.5 percent of children below five years of age are stunted, 15.3 percent are underweight, and 4.6 percent are wasted (Central Statistical Office, Ministry of Health et al. 2014). Chronic malnutrition is prioritized in the National Food and Nutrition Strategic Plan (2011), and the Sixth National Development Plan explicitly mentions nutrition as an essential cross-cutting issue for achieving the country’s socio-economic development. The Government of Zambia is an ‘early riser’ country within the global Scaling Up Nutrition (SUN) movement and is a Feed the Future focal country.
Mumbwa district, where the RAIN project is being implemented, is a rural district in Central Province in the middle of Zambia, around a two hour drive from the capital Lusaka, with a good trunk road
connection but little in the way of local roads, transport, or energy infrastructure. Mumbwa is classed by the Famine Early Warning System as the Central Maize-Cotton Zone, with maize and cotton growing widespread; this area is not prone to drought as rainfall is normally adequate and the area has moderate access to the market, though this varies between Wards (ZVAC 2004). Stunting in Central Province was slightly higher than the national average at the last survey, at 42.5%, and wasting slightly lower, at 4.6% (Central Statistical Office, Ministry of Health et al. 2014).
1.2
Description of the RAIN Project
The Realigning Agriculture to Improve Nutrition (RAIN) project was a partnership between Concern Worldwide (CWW) and the International Food Policy Research Institute (IFPRI), aiming to design, implement and evaluate a model of multi-sectoral integration to improve stunting rates in Mumbwa district, Zambia, and to document evidence of both impact and process for application in other contexts and at scale. The project targeted children during the critical period from conception through 23 months of age, roughly equivalent to the first 1,000 days of life, through integrated agriculture, nutrition and health interventions. The overall approach focused on addressing the multi-sectoral causes of
malnutrition and on learning how to effectively address the challenges of multi-sectoral collaboration; the RAIN project was rooted in literature suggesting that integrated actions by several sectors can provide more effective and sustainable processes through which improved nutrition outcomes can be achieved.
22
The RAIN project comprised of a district-level agriculture intervention to increase year round availability of, and access to, nutrient rich foods at the household level, in some areas accompanied by promotion of optimal health, nutrition, and care seeking behavior through the delivery of social behavior change communication. Most program elements were delivered through local women’s groups created by the program, led by a female Smallholder Model Farmer (SMF) nominated by her group to receive
agricultural training and inputs and pass these on to the group during monthly meetings. In the nutrition and health areas groups were also linked to an existing Community Health Volunteer (CHV) who
received additional training in nutrition topics to pass on to the group. In addition, some community-wide social marketing and information activities were undertaken. A critical characteristic of the RAIN model was hoped to be its ability to be brought to scale and contribute to the achievement of
development goals on hunger and poverty. The RAIN project enrolled 4,437 beneficiaries over the four years of implementation, in staged enrolment, with no phase-out strategy for women once they were recruited. Project partners over the lifetime of the project included the Mumbwa District Child Development Agency (MCDA); Women for Change (WFC); the Zambian Ministry of Agriculture and Livestock (MAL), and the Zambian Ministry of Health (MoH). For more information see Appendix 3: Detail on the RAIN project.
The RAIN project was designed to help address a critical gap in the evidence base regarding the degree to which agricultural interventions, either alone or when combined with nutrition and health activities, can reduce the prevalence of stunting in young children. It aimed to establish ‘proof of concept’ for an intervention model that could then be replicated and scaled up within Zambia and beyond. The project was designed to detect and attribute impact to this RAIN intervention model, and was implemented, monitored and evaluated using a design involving two different intervention groups vs. a comparison group: Intervention Group One was supported to participate both in agriculture and nutrition/ health activities, while Intervention Group Two was supported to participate in agricultural activities only, and the results compared to a control group receiving standard Government services. In line with Concern’s core commitment to eradicate extreme poverty, the RAIN project ensured that extremely poor and vulnerable households were included in the project.
1.3
Evaluation Objectives
The objectives of this impact evaluation are to assess the impact of two different packages of RAIN interventions delivered through community-based agriculture and health platforms.
23 The primary objective of this evaluation is:
1. To assess the impact of the two different RAIN intervention packages on nutrition outcomes among children 24 months and older
In addition to the primary evaluation objective, there are several secondary objectives. These include: 2. To assess the impact of the RAIN package of interventions on core WHO infant and young child
feeding (IYCF) indicators among children 0-23 months of age
3. To assess the impact of the RAIN package of intervention on health and nutrition knowledge among caregivers
4. To assess the impact of the RAIN package of intervention on women’s empowerment
5. To assess the impact of the RAIN agriculture package of interventions on the availability of and access to a year-round supply of diverse and micronutrient-rich plant and animal source foods at household level
1.4
Structure of the Report
This report is structured as follows. Chapter 2 presents the evaluation design, sampling methodology, the main components of the survey questionnaire, and the logistics of fieldwork. Chapter 3 describes sample characteristics at baseline. Chapter 4 presents data on exposure to the different components of the RAIN intervention. Chapter 5 presents findings on the core anthropometric impact indicators by program group and survey round, including descriptive statistics and difference-in-difference analysis to look at changes between groups over time. This chapter also presents three different plausibility analyses to dig deeper into the findings, looking at age groups most likely to have benefited from the program; at confirmed beneficiaries of the program; and at changes in the underlying determinants of nutrition that might explain the results. Chapter 6 presents findings on the impact of RAIN on infant and young child feeding (IYCF) practices by program group and survey round, as well as plausibility analysis looking at confirmed beneficiaries of the program. Chapter 7 presents findings on the impact of RAIN on nutrition-related knowledge among caregivers. Chapter 8 presents findings on the impact of RAIN on women’s empowerment, and chapter 9 on the impact of RAIN on access to diverse foods. Chapter 10
then presents a decomposition analysis to examine various social, behavioral and economic factors as potential drivers of child growth over time, before chapters 11 and 12 discuss the findings in light of
24
process evaluation data and existing literature, and provide some conclusions. The Appendix provides more detailed results for some chapters, where necessary.
2
Methods
2.1
Evaluation Design
To evaluate the impact of RAIN activities, two different evaluation designs were considered, both employing 2 different intervention groups (community based agriculture interventions alone; and agriculture + health/nutrition interventions) and a control group, but differing in how these groups are assigned. A fully randomized cluster evaluation design to evaluate the impact of RAIN was determined to be infeasible primarily for practical project implementation reasons, and therefore a hybrid design was adopted that combines a cluster randomized probability design of the RAIN interventions, with a plausibility design that compares RAIN interventions to a control group (Figure 2.1.1).
This cluster randomized design yields 3 different study arms:
1. Agriculture interventions only
2. Agriculture AND nutrition/health interventions 3. Comparison group with no RAIN project interventions
Repeated cross-sectional surveys were administered in the same communities and at the same time of the year in project years 1 and 5 (baseline 2011 and endline 2015, respectively). This design will allow us to 1) document changes in key impact and process indicators over 4 years of program implementation in the two RAIN project intervention areas and the control area; and 2) determine the impact of agriculture and health and nutrition interventions relative to agricultural interventions alone and relative to the control group.
25
Figure 2.1. RAIN Evaluation Design
2.1.1 Rationale for Age Range to Detect Impacts on Stunting
The assessment of the impact of nutritional interventions on child anthropometry (as well as other outcomes) should consider the age at which assessments should be made to detect the greatest difference between intervention and comparison areas. Evidence suggests that (1) the longer children are exposed to early nutrition inputs like improved IYCF practices and nutrition supplementation in the under-two age period, the greater the impact will be; and (2) the earlier children are exposed, the greater the impact will be (Martorell 1995).
The logic of the RAIN project targeting this vulnerable age period is that the investments in the first two years of life are progressive and cumulative. Therefore, our ability to detect a significant impact on anthropometric status, particularly, will be greatest among those children who were exposed to RAIN interventions in the entire period preceding the age of 18-24 months, which signifies the peak age of growth faltering in Zambia. At that point in time, the differences between RAIN intervention and control children will be much larger than in the preceding period because RAIN interventions, if received early enough and long enough, should protect children and prevent the deterioration of their height-for-age
26
measurements. Measuring the anthropometric impacts of early nutrition interventions in a younger age group (i.e., <18 months of age) will likely underestimate the total impact of the inputs.
For the RAIN project, this means that the age at which impact on anthropometry should be assessed is dependent on (1) the child’s age at the onset of exposure to RAIN interventions, (2) the total duration of exposure to the RAIN interventions within the 0-23 month target age-focus for the RAIN project, and (3) the age of peak growth faltering. We apply these principles with the impact evaluation design to define the exact age group on whom to assess impact on anthropometry outcomes, below.
2.1.2 Randomization Process
During an early exercise with district government officials, two control wards were randomly chosen from a set of three ward pairs; the two other ward pairs were selected to receive RAIN project
interventions. Within the 4 intervention wards there were a total of 29 survey areas previously defined for the Zambian census (areas known as CSAs). From an evaluation perspective, it was desirable to randomly assign these 29 CSAs to the two different RAIN intervention groups. From an operational point of view, however, it was desirable to keep as few as possible smaller units. This would reduce the risk of spill-over effects and would support staff working on the project to select and maintain properly the specific areas that are assigned to them over a period of several years. Considering both statistical and operational aspects, six CSAs were merged into three. This resulted in a total of 26 CSAs that would be available for randomisation. A map was prepared that showed the location of all 26 areas and their boundaries within the four intervention wards, and a process of ‘drawing lots without replacement’ was used during a community meeting in Mumbwa with representation from the District MOH, MACO, MOLFD, MCDA and Concern Worldwide. The resulting randomization is shown in Figure 2.2.
27
Figure 2.2. Map of randomized intervention areas (control not shown)
2.1.3 Sample Size Estimate
For both Baseline and Endline, all sample size calculations were carried out using STATA 11 software.
Two different sample size estimates were obtained. The first was to detect changes in stunting, and the second was to detect changes in height-for-age Z scores.
At baseline, sample size of 1000 children aged 24-59 months per study arm, for a total of 3000 children, was determined to be sufficient to allow us to detect, at endline, a minimum difference in any two groups of a:
I. 8 percentage point difference in the prevalence of stunting
II. 0.2 Z-score difference in the mean height-for-age Z-score (HAZ), assuming a standard deviation of 1.3.
This estimate was based on an estimated baseline prevalence of stunting of 53 percent, with a mean HAZ of -1.8 as reported by the 2009 Demographic and Health Survey (Central Statistical Office, Ministry of Health et al. 2009), a power of 80, a one-sided test, a minimum of 13 sampling clusters per study arm, and considers a relatively high level of clustering (ρ=0.01) of undernutrition within each of the 3 study groups (see Appendices
Agriculture only
28
Appendix 1: Detailed sample size calculations). At baseline, we sampled approximately 1000 households with a child aged 24-59 months of age, for a total sample size of 3044 households. At endline, we oversampled the two RAIN interventions arms, by approximately 20% (for an approximately sample size of 1200 households per arm), to account for potential limited intervention exposure at the household level, for a total sample size of 3536 households. 1.
Table 2.1.1. Sample sizes
Agriculture + Nutrition Agriculture only Control All
Baseline 978 1025 1041 3044
Endline 1212 1244 1080 3536
2.1.4 Sampling Methodology
Sampling of children in these surveys was specific to households that had at least one child aged 24-59 months of age, the age range for detecting impacts on stunting i.e. the primary RAIN project impact indicator. However, to capture impacts of the RAIN project on key IYCF indicators, children between 0-23 months of age were also sampled.
Logistical considerations prevented us from sampling children 0-23 months and 24-59 months of age from unique households; sampling unique household would substantially increase our sample size which was not feasible. Therefore, preference was given to households that had both a child 24-59 months of age as well as a child 0-23 months of age. Where multiple children within these age ranges were present in a household, the youngest child within the household was selected.
A listing exercise was undertaken in the 6 wards to determine eligible households i.e. households with the presence of at least one child aged 24-59 months of age. A child aged 24-59 months old was identified as the INDEX child. A child aged 0-23 months old was identified as the NON-INDEX child. All households were categorized into 3 possible categories:
1. Households with an INDEX child only
2. Households with both an INDEX and a NON-INDEX child
3. Households with a NON-INDEX child only
29
Because the CSA is the unit of randomization, it is therefore the unit of most interest from an evaluation perspective as it is the unit by which treatments are applied. However for sampling purposes the CSA covers a very large geographic area and a large number of households and is therefore not practical as a unit for sampling. For sampling purposes the unit that is most practical is the Supervisory Enumeration Area (SEA), which is a clearly identified geographic area as per the Government Central Statistics Office (CSO). The majority of CSAs have 3 SEAs within them, though within the 6 wards from which the
baseline survey sample was drawn, there was a range of 2-5 SEAs per CSA, as per the 2010 CSO sampling frame.
The sample size calculations required that a total of 69 households be sampled in the endline survey per unit of randomization-the CSA. The majority of CSAs had 3 SEAs, and within these SEAs 23 households were sampled. In CSAs that had 4 or 5 SEAs, 3 SEAs were randomly sampled and 23 households were sampled from each SEA. In CSAs that had only 2 SEAs, 35 households were sampled from the first SEA, and 34 households from the second. Therefore, based on this sampling procedure, a total of 69 households were sampled per CSA as required by the sample size calculations.
Within each SEA, households were sampled based on the household listing exercise conducted at the start of field work, using a systematic random sampling procedure that used a random number table. A total of 3044 households were sampled for this baseline survey. Only 34 households that were selected as part of the systematic random sampling procedure refused to be interviewed.
2.2
Survey Instruments
Several instruments were created for the collection of field data in this survey, based on a conceptual framework designed to capture the various factors known to affect child nutrition as described in the baseline report (Harris, Quabili et al. 2011). The conceptual framework and survey instruments are described below. Data was collected through a multi-module household questionnaire, and anthropometric measurements of children and caregivers.
2.2.1 Conceptual basis for the impact evaluation questionnaire
The UNICEF conceptual framework (UNICEF 1990) formed the basis for the questionnaires, which were designed to capture information on all levels of influence that contribute to child undernutrition. The framework identifies the causes as immediate, underlying and basic, with each level of factors having influence on the other. The framework identifies inadequate dietary intake and diseases as the
30
household and family level. These are mainly insufficient or lack of access to food, clean water and sanitation, health services, inadequate maternal and child care, and maternal undernutrition. The immediate causes are embedded within the larger societal factors or basic causes which range from human and social capital, women status in the society, political and social atmosphere, etc.
In addition to the above key impact indicators, because the household is the unit of analysis, and the focus of our data collection, we captured information from sampled households on the impact of the RAIN project on key agriculture, health, and nutrition indicators as outlined in the log frame (e.g. household food security, dietary diversity, and IYCF practices). In addition, information was collected on other factors that influence the uptake, adoption and impact of RAIN interventions, such as
socioeconomic status; local food production; household market access and food purchasing behaviour; household parental characteristics; maternal knowledge of essential nutrition actions (ENA); exposure to other agriculture, health, and nutrition interventions; exposure to media; and household gender
relationships. At endline, we also included questions on women's time use, and exposure to the RAIN intervention.
2.2.2 Household questionnaire
The household questionnaire was written in English and translated simultaneously by data collectors into one of the several local languages; responses were noted in English. It was administered to the mother of the index child chosen for the survey (or the child’s primary caregiver, if this was not the child’s biological mother; the word ‘caregiver’ is used for the rest of this report, apart from those instances where questions relate to the biological mother of a child only, such as with pre- and post-natal care).
The household questionnaire was based on previous nutrition evaluation questionnaires used by IFPRI, which were grounded in the questionnaire model used for Demographic and Health Surveys and incorporated several validated instruments to measure different determinants of nutrition. The
questionnaire was adapted substantially for the purposes of the RAIN evaluation, particularly in terms of local foods, agricultural techniques, and indicators of socio-economic status. Feedback was sought from the implementing NGO, and the questionnaire was piloted at baseline and revised into the final version.
2.2.3 Anthropometric measurement
Anthropometric measurements (height and weight) were taken for eligible children in each household (less than 5 years of age) and their caregivers. Anthropometric data was collected using a standardized
