core functions and
elaborated ideas
on funding and
institutional design
When Will We Ever Learn?
Table 2 Make collective commitments
To be signed by several social ministries of developing countries, bilateral agencies, multilateral development banks, research institutions, philanthropic foun- dations, and nongovernmental organizations and executed by an international council
Function The council would: My organization would: How my organization would benefit: Establish quality standards for
rigorous evaluations • Convene experts to establish or endorse standards for method- ological rigor and evidence. • Support advances in evaluation
methodologies.
• Participate in establishing interna- tionally agreed standards for rigor- ous impact evaluations by which we will abide.
• Our studies will gain external legitimacy and we will find it easier to evaluate the quality of evidence coming from other organizations.
• We will be informed of method- ological advances.
Administer a review process for
evaluation designs and studies • Administer a review process with rotating panels of experts for impact evaluation proposals and completed studies.
• Assess whether submitted impact evaluations meet agreed standards of reliability and validity through endorsing existing peer review processes or, when necessary, submitting for separate expert review.
• Publicize clear standards for qual- ity of evidence.
• Promote production of systematic reviews.
• Offer prizes for high-quality research.
• Contract the council to conduct reviews of proposals and studies when we do not have in-house capacity.
• Provide staff to act as reviewers in areas of their expertise.
• Facilitate linking researchers and project managers when opportuni- ties for impact evaluation arise.
• Participate in establishing stan- dards.
• Submit impact evaluations for review by council or some other accredited independent process.
• This will reduce our require- ments for in-house expertise and improve our ability to identify rig- orous evidence in decisionmaking and evaluation design.
• We can more readily distinguish between strong and weak evi- dence.
• There will be greater incentives for our staff to produce or supervise rigorous studies.
• We will get external legitimacy, credibility, and prestige for the studies we finance or contract.
Identify priority topics • Convene committees of stakehold- ers to identify enduring questions for social program decisionmak- ing to guide members and the council’s own programs toward policy-relevant research.
• Participate in committees to iden- tify enduring questions and prior- ity topics.
• We will have the opportunity to influence the focus of impact evaluations in other organizations in light of our own demands for information.
• We will learn about the most pressing concerns of other organi- zations.
• We will be more likely to find impact evaluations on issues that are of importance to us.
Provide grants for impact evaluation
design • Manage funds that would be used for assessing whether an impact evaluation is feasible and relevant.
• Contribute to a collective pool of funds for impact evaluation designs.
• Our staff and managers will have access to flexible, timely funding to seize opportunities for initiating rigorous impact evaluations.
Organize and disseminate
information • Maintain a comprehensive data-base of impact studies, with associated information about the quality of evidence; prospective registry of studies.
• Provide open access to primary data.
• Document evaluation experiences.
• Disseminate training materials.
• Send our documents, prospective studies, training materials, and data to the council.
• We will have access to knowledge generated by other organizations, learning materials, and data.
• We will be able to distinguish between strong and weak evi- dence.
Build capacity to produce, interpret,
and use knowledge • Facilitate access to training materi-als, programs, and experts.
• Encourage institutional collabora- tions and use of local researchers.
• Optionally, finance fellow- ships, courses, and activities to strengthen local research institu- tions
• Engage in or provide technical support for capacity building activities for local research.
• We will be more likely to find local researchers with requisite skills to conduct impact evaluations.
• Resulting studies are more likely to be of good quality.
• Partners will be better informed about research quality, interpreta- tion, and uses.
Create a directory of researchers • Maintain and provide access to a
directory of qualified researchers. • Submit information regarding qualified researchers.
• Make use of references as required.
• We will gain ready access to a list of qualified researchers
Improving Lives through Impact Evaluation
By reviewing evaluation designs and assessing completed evaluations according to clear and transparent standards of methodological rigor, the council can help members distinguish between stronger and weaker evidence. By rating the quality of proposals and research, the council would enhance the generation of knowledge from impact evaluations in several ways. Researchers would have greater incentives to do rigorous studies, knowing that the effort would be recognized. Project managers, policymakers, and the public could more easily direct their attention to better evidence. And efforts to build capacity could more easily identify models for emulation.
• Identifying priority topics. No government or agency can initiate studies
on every policy question that they would like answered. Nor is it neces- sary to evaluate every program. A collective effort to identify the most pressing and enduring policy questions would help governments and agencies to cluster evaluations around common topics and to focus efforts on programs that are most likely to yield useful information for future policymaking. By participating in such a collective effort, govern- ments and agencies can influence the questions being asked and benefit from studies done by other institutions on programs like their own.
• Providing grants for impact evaluation design. The window of oppor-
tunity to design a good impact evaluation on an important question is narrow, occurring just at the moment of program conception and design. Often, the missing ingredient is timely funding to contract an expert to meet with stakeholders to assess whether an impact evalu- ation would be appropriate and what methods would generate the best evidence and then to design the evaluation. With small amounts of money, the council could act as a powerful catalyst—in some cases
Communication and public education • Raise awareness and advocate for changes in legislation and institu- tions to establish strong incentives to produce and incorporate knowl- edge from impact evaluations into policymaking.
• Educate legislators, journalists, and civil society in proper uses and value of impact evaluation.
• Coordinate communication and public education activities with the council.
• We will have an external ally to encourage the production and use of quality research.
Administer funds for impact
evaluation on behalf of members • Provide services to members who request and finance them, such as prepare terms of reference, con- tract and monitor research teams, and convene external review pan- els to assess research designs and final reports.
• Choose whether to use the coun-
cil’s services. • We would have the option of contracting the council to man- age impact evaluations, taking advantage of its network of review panels, expertise, quality assur- ance mechanisms, and reputation for independence and credibility.
Administer a pooled fund for impact
evaluations • Solicit, contract, monitor, and ensure the quality of impact evalu- ations to answer questions identi- fied by members as of enduring importance and of high priority.
• Contribute funds to a pool man- aged by the council.
• Participate with other members in setting priorities for which programs and questions will be addressed by the impact evalua- tions financed by the pooled fund.
• Our contributions will leverage substantially more impact evalu- ation than we can achieve on our own, complementing and expand- ing the value of our impact evalua- tion work.
• We will have access to a growing body of rigorous impact evalua- tions on questions of importance to us.
When Will We Ever Learn?
making it possible to do impact evaluations that might not otherwise get done; in other cases increasing the likelihood that the money spent on evaluation generates reliable and valid conclusions.
Other functions. Other functions identified in this review are either less
critical to the council’s mission or require substantially more resources. These functions might be delegated to the council in the future, depend- ing on the council’s performance, its staffing, and members’ interest and financial support:
• Organizing and disseminating information. With rapid changes in tech-
nology a wide range of databases and search engines are available to find information about what works in social development. However, the sheer number of studies and data that appears in a Web search is daunting without the ability to easily identify which information is most relevant and rigorous. A council could collaborate with other organizations to set up a prospective registry of impact evaluations to address publication bias, maintain databases of completed qualified studies, and encourage the production of systematic reviews.
• Building capacity to produce, interpret, and use knowledge. Efforts
to build local research capacity and evaluation systems should con- tinue. The creation of a council could enhance these individual efforts by establishing a network of expert reviewers who can also serve as technical consultants and trainers; by rewarding proposals that are led by developing country evaluators or that incorporate genuine partnering with local research institutions; by encouraging new people to enter the field of evaluation with fellowships or involvement in proposals; by briefing public officials, journalists, and civil society organizations on the benefits and uses of impact evaluation; and by disseminating training materials and rigorous evidence.
• Creating a directory of researchers. Governments and agencies often
have difficulty finding qualified research partners and use the same consultants repeatedly because of the costs of identifying new ones. The council’s endorsement of standards for impact evaluation, its net- work of reviewers, and its database of rigorous studies could generate a directory of researchers with proven skills and expertise. With little additional effort, the council can make this information available to its members and actively encourage the use of qualified experts.
• Undertaking communication activities and public education. Politicians
and the public do not readily understand impact evaluations, yet impact evaluations are critical to informing public debate and policymaking. The council could explain the benefits and uses of impact evaluation, advo- cate for legislation and policies to support the production and application of such knowledge, and build public support. The council’s network of experts, representatives from member organizations, and its own staff can give impetus to domestic initiatives to strengthen evaluation systems.
• Administering funds on behalf of members. Some members might
choose to use the council’s services to commission studies on their