Methodological issues

Study design

The main limitation with the evaluation presented in this thesis is the weakness of the arguments attributing causality. Randomised controlled trials are considered the gold standard to establish a causal relationship but it is not always possible to randomise people and results are typically not very generalisable since they offer an indication of controlled efficacy rather than real life effectiveness [188]. In the case of the ACCESS programme it would not have been practical or politically acceptable to randomise communities. It would have also excluded the possibility of monitoring the community impact of the programme since too few communities would have been in the DSS area. Hence a plausibility

assessment [90] was the only feasible way to assess the impact of ACCESS interventions. With this type of design improvements are attributed to the programmes if improvements are found in every step of the causal pathway between intervention and impact and all other explanations can be formally discarded. The comprehensive M&E plan of the ACCESS programme enabled to reach a high level of plausibility but it was not possible to rule out that improvements could have happened independently from the project, especially as a result of a secular trend.

There would have been two ways to reach a higher level of plausibility: 1) by documenting changes in the understanding of malaria and its treatment and 2) by choosing a comparison group to control for the influence of confounding factors. The main health outcome of the programme was the increase in the proportion of fever cases treated with an antimalarial within 24 hours. This improvement was ascribed to the ADDO intervention and the social marketing campaigns, on the basis of improved access outputs, (i.e. better availability and accessibility of treatment) and better knowledge. The access outputs were measured, and indeed improvements were noted [102] (cf. Chapter 4) but better knowledge was only

7. Discussion 116

assumed as a necessary consequence of exposure to social marketing campaigns. Although it might be a perfectly reasonable assumption, measuring improvements in understanding within the frame of the EMIC treatment seeking surveys or with a household survey before and after the implementation of the interventions would have added strength to the argument that the social marketing campaigns were able to promote behaviour change.

As far as the choice of a control is concerned, the Rufiji Demographic Surveillance Site (DSS) could have been an ideal comparison group. Two other large scale studies have evaluated interventions by comparing the Ifakara and the Rufiji DSS. The Interdisciplinary Monitoring Project for Antimalarial Combination Therapy in Tanzania (IMPACT-Tz) [105] [150] and the Integrated Management of Childhood Illness Multi-Country Evaluation (IMCI- MCE) [34, 166] have made use of this design. The choice of an external control would have enabled, for example, to rule out the possibility that improved access outputs were just the result of a natural growth in the retail sector for drugs, and that better treatment outcomes are just a consequence of that. It would have also enabled to quantify better the impact of the programme on morbidity and mortality.

Choice of location

There were numerous advantages to conducting this study in the Ifakara DSS. The DSS provided an excellent epidemiological framework for this study. The constant demographic surveillance enabled the assessment of the mortality and morbidity impact and facilitated the selection of random sample of households for the treatment seeking surveys [102] (Chapter 4). The findings from qualitative investigations of people’s perception and understanding of disease could guide the adaptation of the questionnaire used for the treatment seeking surveys [81, 128]. Furthermore, there was a wealth quantitative data to support the interpretation of the findings of this thesis. This includes entomological and parasitological data [77, 173, 189-190] (Chapters 5 and 6) collected since the 1990s. This data was further complemented by in depth descriptions of the agricultural practices of the farmers of the Kilombero Valley [74, 76, 167] and comprehensive agricultural production data from the Kilombero District Agricultural and Livestock Development Office (DALDO), most likely due to the proximity to the Kilombero Agricultural Research and Training Institute (KATRIN). In addition KATRIN itself had been collecting rainfall data since the nineties which it kindly agreed to share.

However, studies conducted in DSS areas are not necessarily very generalisable to the rest of the country. The Kilombero and Ulanga Districts in general, and their DSS areas in particular, have been the object of continued donor and research attention for decades. The

use of antimalarials to treat fever was already much higher in the study area (73% feverish children treated within 24 hours) compared to the rest of the country (51%) [26]. The continued visits by DSS staff could be causing respondent fatigue and biasing our results. The possibility of a Hawthorne effect also needs to be considered [191-192], whereby people may be adapting their responses (and behaviour) simply because they are being studied, not necessarily in response to a particular intervention.

Interventions

Unfortunately this evaluation was not able to fully assess the effect of interventions at health facility level. A refresher course training was conducted between 2004 and 2005 in the health facilities of the Kilombero and Ulanga District to improve quality of care. A follow-up was planned with a Quality Improvement and Recognition Initiative (QIRI). QIRI offers an

integrated approach for the evaluation of quality of care combined with a strategy to establish the root causes of performance gaps and to develop implementable strategies to address them. A central element of this component is capacity building for joint supportive

supervision and quality management, conducted by the regional and district health

management teams together with community representatives. Due to logistic constraints the implementation of QIRI excercises had to be postponed to 2007 and hence their effect could not be evaluated in this thesis.