Of the 140 studies included in the Cochrane review by Ivers et al., only four were from low-or middle-income countries. To our knowledge, there have been no systematic reviews of the effect of AF in low-resource settings. Two review papers however, both from 2005, examine the effectiveness of health worker performance improvement interventions in low-resource settings, and both evaluate AF among other performance improvement interventions.
Rowe et al. reviewed systematic reviews and other reviews of performance improvement interventions in low resource settings.3 The authors identified 11 literature reviews about performance improvement interventions that mostly address low-and-middle-income coun-tries. Of the 11 reviews, three address AF in combination with supervision. The authors concluded that, based on the systematic reviews they reviewed, they observed a trend indi-cating that supervision combined with AF is “generally quite effective.” The authors note that multi-faceted interventions are more likely to improve guideline adherence than isolated interventions. However, this conclusion is nearly exclusively based on use of AF for prescrib-ing, rather that other contexts and tasks. While the findings may be relevant for prescribing tasks within ART, they may be less applicable to other types of tasks, such as patient referral or group task performance. The review calls for high-quality research investigating perfor-mance improvement in developing countries, noting a lack of rigorous evaluations. Rowe et al. emphasize the importance of understanding when contextual factors may interfere with the generalizability of a study. The authors recognize the importance of connecting interventions to relevant theories in order to build common frameworks that can be used to organize and promote high quality research. They include lists of domains of theories and then specify the interventions that are based on theories within each domain, but list these only at a high-level that does not attribute specific psychological theories to AF. This review
is valuable in its recognition of a need for the use of conceptual frameworks to inform the evaluation of performance improvement interventions.
Another review from 2005, by Siddiqi et al., evaluates the effectiveness of guideline imple-mentation interventions in low-resource settings, specifically addressing AF. In the review, Siddiqi et al. find 15 studies evaluating AF, with one of those studies being an RCT. While almost all of the 15 studies demonstrated an improvement in guideline adherence or patient outcomes, most had design flaws, the authors noted. One important consideration put forth by Siddiqi et al. is the fact that the literature evaluating AF from developing countries is mostly positive, which is not reflected in the conclusions of systematic reviews of RCTs done primarily in high-income countries. The authors speculate that publication bias against negative findings may contribute to this relative lack of ineffective AF evaluations from low-resource settings. Nevertheless, Siddiqi et al. ascribe great potential to AF for low-cost performance improvement in developing countries. Notably, the authors do not question the issue of atheoretical approaches to designing AF interventions, nor do they acknowledge the variability of activities that are attributed to AF.128
The majority of publications about AF in low-resource settings over the past decade ad-dress either prescribing behavior or critical incident audit for obstetric care, aimed to reduce rates of perinatal mortality and morbidity. Within the critical incident audit literature, few if any studies are RCTs. The fact that a 2005 Cochrane review was initiated to evaluate the effectiveness of critical incident AF in reducing maternal mortality and morbidity in low-income countries is indicative of the growing interest in this intervention technique for developing countries. However, the authors found no suitable trials to include in the review and none have been found in updates of the review through 2011.129
Specifically within Malawi, three recent studies from Thyolo District in Malawi’s southern region are indicative of a broader trend towards the use of critical incident AF in low-resource settings. Two observational studies measured reduction in negative clinical outcomes using before-and-after studies of obstetric outcomes for critical incident AF. A 2009 study observed a reduction in the incidence of uterine rupture from 19.2 per 1000 births prior to the study, down to 6.1 per 1000 births at the conclusion of the study period.119A 2011 study at the same district hospital observed a reduction in the incidence of severe maternal complications from
13.5 per 1000 births to 10.5 per 1000 births at the end of the study period. This study noted significant reductions in mortality, hemorrhage, and uterine rupture, and non-significant reduction trends in eclampsia and peripartum infections.130 Another 2011 study conducted in Thyolo District used qualitative methods to evaluate health worker perceptions of critical incident AF. The authors concluded that, while a minority of staff had expressed fear about the audit sessions where critical incidents were discussed, most health workers viewed the technique positively and perceived the purpose of AF to be about learning, improving the quality of care, and improving focus and motivation within the clinic.131 Critical incident AF contrasts with the more standard practice of providing feedback to individual providers about their prescribing or test ordering behavior by actively engaging HCWs in the audit process. The recent publications describing clinical AF in Thyolo district were performed largely in a participatory manner lead by local health workers, rather than as a governmental initiative or other national program to improve the quality of care in Malawi, which may have further contributed to their success. Studies of AF in low-resource settings appear to have generally more positive effects than studies conducted in industrialized nations, but few evaluations of AF in low-resource settings are RCTs, preventing observation and analysis of effect. Observational studies in Malawi in particular have demonstrated significant impact of AF for critical incident audit within the domain of obstetric care.
Evidence addressing the effect of AF on health worker performance shows modest im-provement resulting from the use of AF, but with wide variation in effect that includes studies showing performance decreases. This variable evidence belies two issues of note:
1) the practice of AF is heterogeneous, including activities that vary in approach, targeted behavior, professional role, and context and 2) the mechanisms by which performance feed-back impacts behavior change are not well understood. The synthesis of evidence about the effectiveness of AF is hindered by its frequent evaluation within multi-faceted performance improvement interventions that do not evaluate the effect of AF alone within a controlled setting. A consensus is forming around the position that future studies of AF should not evaluate its efficacy in general, but rather should investigate the features of feedback as moderators of the effect of feedback on performance within head-to-head comparisons in controlled contexts.103,132 Furthermore, increasingly, researchers are turning to psychological
theories of behavior change inform research questions about the underlying mechanisms by which AF impacts behavior.
6.4 CONCLUSION
In summary, evidence shows that AF interventions appear to have a wide range of effects on performance, including large positive effects and null or negative effects for a significant proportion of studies. This large variance in effects on performance may be due in part to the heterogeneity of targeted behaviors, barriers to change, performance measures, AF components, clinical settings, and healthcare provider roles in AF intervention contexts that have been studied. Furthermore, evidence shows that recent clinical trials of AF are not adding to our insight into how and when AF is effective. To improve our understanding of how and when AF works, researchers are increasingly looking towards the application of psychological and behavioral theory to AF approaches.