CHAPTER 3. METHODOLOGY
3.3 Data collection and sample characteristics
84. This section describes the characteristics of the sample and its suitability for assessing project impact.
The validity of a DD approach rests on the assumption that project and comparison groups are similar. Differential trends in the outcomes and covariate shocks were discussed in the previous second round analysis reports and were not found to be major threats. Here, we focus on changes in the composition of the project and comparison groups produced by attrition and migration.
85. The baseline survey targeted a sample of 755 households in the MV villages and 1,496 households in the CV villages. These sample sizes were identified to detect impacts of an acceptable size through power calculations. The size of the comparison group is twice the size of the project group for the reasons outlined above: to stratify the impact of the intervention by distance thus identifying spill over effects, and to be able to perform matching of observations at the household level to further improve the comparability of the two samples. In every survey round, the same households were targeted for the interview, though at each round not all targeted households were found. As a result, the samples vary at each survey round while the sample of panel households decreases over time.
We decided to follow this approach, rather than only following panel households over time, because a number of impacts such as mortality, nutrition and education tests are estimated over cross-sections and therefore benefit from larger samples.
3.3.1 Sample characteristics
86. The number of interviews conducted in project and comparison villages in all rounds is shown in Table 4. The largest number of interviews was conducted in the second round, while the smallest was conducted during the baseline. There is no obvious pattern in these numbers. There are no differences in the percentage of households interviewed in MV and CV areas suggesting that the absence of project benefits did not act as a deterrent to responding at the interviews in the CV areas.
On the contrary, these numbers suggest that a proportionally larger number of households was interviewed in CV areas than in MV areas.
Itad in association with IDS, LSHTM and PDA Ghana Back to contents Page | 39
Table 4. Completed household interviews in MV and CV areas
Sample Target 2012 2013 2014 2015 2016
87. The number of households interviewed across rounds is reported in the flow diagram, Figure 4. The diagram follows the style of CONSORT diagrams36 used in reporting the results of randomised control trials and also includes the number of households included at the analysis stage. This number is restricted to households interviewed (and matched) at baseline that were interviewed again in the following rounds.
Figure 4. Flow diagram of MV and CV households included in the study
88. For completeness we also report a flow diagram for comparison households from the ‘near’ (CN) and
‘far’ (CF) comparison groups, because by design they represent two separate samples. The two samples are very similar though it appears that fewer households were followed up in the ‘far’
comparison areas. A smaller number of ‘far’ comparison households is also included in the final analysis because trimming of the sample following matching removed a larger number of households that are distant from the MV area.
36 A CONSORT diagram (CONsolidated Standards Of Reporting Trials) is a flow diagram that displays the progress of all participants through a trial, such as a randomised control trial.
Household interviews
Itad in association with IDS, LSHTM and PDA Ghana Back to contents Page | 40
Figure 5. Flow diagram of ‘near’ and ‘far’ comparison households included in the study
Table 5. Panel households in MV and CV areas
Sample Target 2012 2013 2014 2015 2016
MV panel interviews 755 711 707 697 689 684
% 94.2 93.6 92.3 91.3 90.6
CV panel interviews 1,496 1,461 1,454 1,424 1,391 1,389
% 97.7 97.2 95.2 93.0 92.8
All panel interviews 2,251 2,172 2,161 2,121 2,080 2,073
% 96.5 96.0 94.2 92.4 92.1
3.3.2 Attrition rates
89. Attrition rates in the study area were very low. Less than 8% of the original target sample was lost over time. Note, however, that the baseline did not interview all target households. The attrition rate for the households interviewed at baseline is only 4.6% for the whole sample ((2073–2172)/2172).
More importantly, the attrition rates in MV and CV areas are very similar, being 3.8% and 4.9%
respectively.
Household interviews
• Baseline=725
• 2nd round=744
• Midterm=725
• 3rd round=716
• Endline=724
• Full panel=683
Matched households
• Baseline=634
• 2nd round=632
• Midterm=617
• 3rd round=609
• Endline=617
• Full panel=596 Matched households
• Baseline=675
• 2nd round=670
• Midterm=661
• 3rd round=663
• Endline=664
• Full panel=651 Household interviews
• Baseline=736
• 2nd round=743
• Midterm=731
• 3rd round=730
• Endline=733
• Full panel=706 Control group “near”
• Villages=34
• Households=748
Control group “far”
• Villages=64
• Households=748 Selection
Analysis Follow-Up
Itad in association with IDS, LSHTM and PDA Ghana Back to contents Page | 41
Table 6. Panel of individuals in MV and CV areas
Sample 2012 2013 2014 2015 2016
MV individuals 5,231 5,576 5,854 6,021 6,338
MV panel 4,930 4,654 4,550 4,474
% 94.2 89.0 87.0 85.5
CV individuals 10,337 10,649 11,023 11,255 11,750
CV panel 9,869 9,378 9,072 8,875
% 95.5 90.7 87.8 85.9
All individuals 15,568 16,225 16,877 17,276 18,088
All panels 14,799 14,032 13,622 13,349
% 95.1 90.1 87.5 85.7
90. Attrition rates were relatively small also for individual household members (see Table 6). More than 85% of the original target individuals were enumerated in the last survey round. The reduction in the sample is more the result of changes in household composition and errors in reporting household membership than of actual dropping out of the study.
Table 7. Reasons for not completing the interviews
Reason 2012 2013 2014 2015 2016
No competent household member at home 21 1 8 13 12
Entire household absent 22 11 20 14
Interview postponed 10
Interview refused 1
Partly completed
Dwelling vacant or destroyed 4 2 20 18
Dwelling not found 19 9 13 5 22
Household has relocated 6 15 8
Household dissolved or deceased 1 6
Other 6 4 7
All 79 21 59 73 66
91. The number of households lost at each round and the reason for not completing the interview at each round are reported in Table 7. These numbers are very small and do not allow us to investigate whether the ‘attriters’ are different from the rest of the sample, much less if there are differences in characteristics between MV and CV attriters. The reasons for not completing the interviews vary over time, but the absence of a competent household member at the time of the visit, absence of the entire household or the inability to find the dwelling were the predominant reasons.
92. Enumerators were instructed to enquire about the whereabouts of the households from neighbours in cases when it was assumed that the household had relocated. Few households appeared to have relocated and, oddly, not a single household had reportedly relocated in the year before the last survey round. Favourite locations for relocation appear to be Kumasi and surrounding areas and Accra (see Table 8). Overall these data suggest that, despite some challenges, the survey teams were able to enumerate households with the accuracy expected by similar surveys in the same settings and that there are no large differences between MV and CV areas driven by natural population change, movements in and out of households or reporting errors.
Itad in association with IDS, LSHTM and PDA Ghana Back to contents Page | 42
Table 8. Whereabouts of households reported as ‘relocated’
Location 2013 2014 2015 2016
Kumasi, Kumasi, Ashanti 1 2 4
Jagsi, Kumasi, Ashanti 1
Delaasa, Kumasi, Ashanti 1
Ejisu, Ejisu-Juaben, Ashanti 1
Sariba, Northern, West Mamprusi 1
Obuasi, Obuasi, Ashanti 2
Luisa, Builsa, Upper East 1
Accra, Greater Accra 7
Kentasi, Ashanti 1
Presetia 1
Eastern Region 1
Missing 0 5 0
All 6 15 8
93. As discussed in Annex A, attrition rates were also relatively small among individual household members. More than 85% of the individuals originally selected for the interview were enumerated in the last survey round. Attrition among individuals was mostly the result of errors in reporting household membership and, to a much lower extent, of changes in household composition. Changes in household composition are a potential threat to the validity of the comparison of outcomes in MV and CV areas. For example, comparisons could be biased if a considerable portion of the control population were to migrate to MV areas to access project benefits. The project and control groups could have changed in two ways: by a natural increase (the difference between births and deaths) or by movements of individuals in and out of households. We tested the impact of the intervention on household size and per adult equivalent and we found none. This result is important because it ensures that per capita and per adult equivalent figures in the project and control group are comparable. As for migration, the project did not have an impact on ‘permanent’ movements outside the Northern Region and reduced ‘temporary’ seasonal migration. The latter result being only temporary, however, does not lead to a change in the composition of the household.
3.3.3 Seasonal bias
94. The potential seasonal bias resulting from conducting the household surveys at different times at baseline, was thoroughly investigated and the results can be found in Appendix H of the Baseline report. Our analysis of the seasonality of the project outcome variables using secondary data showed that seasonality would not affect the estimation of impact on most variables and the large similarity of the same variables at baseline in project and control areas seems to confirm this position. We stress however that the use of mosquito bed nets and malaria incidence in particular could be affected. We also concluded that any seasonal bias could not be either ‘estimated’ or ‘adjusted’.
Nonetheless, the direction of bias is known and can be used in the interpretation of results. Of the two outcomes that could be affected by the bias: malaria prevalence and use of bed nets, only use of bed nets shows baseline values in project and control areas that are clearly different and could suggest the presence of bias. The bias would underestimate impact on malaria prevalence in MV areas and overestimate the impact on bed net use in MV areas.