While the report represents a consensus of the Working Group and its overall conclusions are broadly supported by all the members in their individual capacities, two members requested that their dissenting opinions be reg- istered on issues of particular importance that they feel are not adequately treated in the document.
François Bourguignon and Paul Gertler
The report makes a compelling case that filling the knowledge gap is a criti- cal next step for the development community to take to improve the lives of people living in developing countries. Filling the gap will require a sustained effort to increase the number of impact evaluations of development interven- tions. Moreover, it will require that similar interventions be evaluated in various settings, with the consequent need for a concerted evaluation effort.
We fully support this message and believe that the report and the consulta- tion process that produced it have served a major role in bringing attention to this issue. We also agree with most of the report’s conclusions and recom- mendations. However, we believe that the report’s recommendations need to be strengthened in two areas. First, closing the evaluation gap should not only be about providing more and better information, but also about making sure that the new knowledge is used in ways that improve the lives of people living in developing countries. Second, the success of any effort to improve impact evaluation will depend on the nature of the institutional arrangements created. Below we propose a number of recommendations along these two lines:
• We believe that one of the keys to filling the knowledge gap is strong part- nerships with developing country governments, NGOs, and researchers. Countries should both benefit from the new knowledge generated and be full partners in its generation. This requires an effort to build capac- ity within developing countries. Countries must have both the interest and capacity for quality evaluation to be systematically conducted and institutionalized.
• In our view one of the most critical challenges requiring collective action is related to the global public goods aspects of the evaluation gap. These concern mainly the need for systematic information shar- ing and dissemination within developing countries and development agencies, as well as coordination in the identification of priority topics. We are concerned that these issues have not received the attention they deserve in the recommendations of the report—and indeed have been left out of the core functions of the proposed council.
• We strongly recommend that any proposals for international collective action should encourage and not stifle or otherwise create undue bur- dens to initiatives to conduct impact evaluations by individual organiza- tions (national or international). National and international groups are heterogeneous in their interest in impact evaluations and their ability to conduct them. Participation in the council could prove burden- some for organizations that have capacity and have already scaled up quality evaluations. This would be the case, for example, for the
Improving Lives through Impact Evaluation
establishment of “quality standards” and the associated function of “reviewing” evaluation designs and studies. We believe that it would be better to let individual organization decide which services to use as needed.
• We also recommend that any proposal for establishing a new body for the purpose of allocating funds ensure that the body actually increase the total amount of funding available for evaluation and not increase the transaction costs in the allocation of such funding. In particular, we remain worried about creating a new bureaucracy with its own interests in managing funds that should be directed to global public goods.
Reply by Co-Chairs William Savedoff, Ruth Levine, and Nancy Birdsall to Bourguignon and Gertler
We are puzzled by this reservation because the points it makes do not differ from the Working Group consensus and are incorporated into the report. First, we state that the purpose of generating knowledge is to improve policy decisions that can make real differences to the well-being of people in developing countries (for example, pp. 9–10). Second, we explicitly address the issue of different institutional designs (pp. 40–42). Third, we are clear that improving impact evaluations and their use requires full partnership with developing country governments, NGOs, and researchers (throughout the report; see, for example, pp. 31, 40, and 42), noting that such partner- ships ensure good design and local relevance. We also highlight the need to strengthen capacity building (p. 33) and propose it as a function for the council (p. 38). Fourth, we agree that information sharing and dissemination are important (pp. 29–30). Indeed, identifying priority topics is included as a proposed core function (p. 37). Fifth, we agree that new resources should be additional to and complementary to existing initiatives (p. 34) and support fully the logic of reinforcing existing activities within member organizations (pp. 32–33). Finally, we nowhere recommend that any member have any obligation to submit any evaluation designs or reports to the proposed council. Membership itself is envisioned as voluntary. Members would decide on how the council would be governed and how such functions as “establishing quality standards” would be made operational (p. 35).