A number of basic steps are required for designing and implementing the process, whether at the ministry or agency level.
1. Establish a committee of key agency personnel to guide and over-see the implementation eff ort. The committee should include individuals from within the organization as well as representatives from the offi ces of the program, budget, planning, information technology, human resources, and an analytical offi ce such as statistics, if one exists, whose work might be expected to relate to M&E. It is also likely to be helpful to include in the com-mittee an “outside” person who can bring a broader perspective, such as someone from a higher-level budget or planning offi ce.
2. A talented leader should be selected, as well as a small cadre of support personnel who will be responsible for supporting the implementa-tion of the various administrative and technical elements of the M&E system (under the guidance of the committee).
3. Consider meeting in the early stages with members of the legisla-ture to fi nd out what information would be useful to them for annual budget deliberations or other policy or oversight purposes. Similar input should be sought from the national budget offi ce.
4. Develop a plan and schedule for implementation, keeping in mind it is likely to take three to fi ve years for full implementation. However, some initial useful performance information can probably be expected to become available by the end of the fi rst year. As part of this step, it is critical to iden-tify who will be needed to undertake each step of implementation. The peo-ple selected should be available and committed to providing the time neces-sary to perform their assignments.
5. Provide training and technical assistance to managers and key professional staff in the basics of M&E. This applies to initial training and training new people as turnover occurs. The training for managers should provide a thorough understanding of what M&E entails and recommenda-tions for using the information. The training for other staff should focus more on the details of how M&E should be conducted so as to make the in-formation obtained both valid and useful. Multilateral and bilateral funding organizations are potential sources of support for these activities.
6. Ask all program managers within the agency to identify a results-focused mission statement for their programs. These statements will necessarily be general, but they should set the tone and direction for the M&E eff ort in each program.
7. Ask each program within the agency to select a set of outcomes and outcome indicators1 that will enable the program to monitor progress in achieving the results identifi ed in its mission statement. Programs could establish their own working groups of key personnel to help select appropri-ate outcomes and outcome indicators.
The M&E working group might also seek input from representatives of the customers served by the program. For example, the program might spon-sor meetings (focus groups) with small numbers of clients to help identify outcomes of importance to them. A public welfare agency, for example, might hold such sessions with its clients, a health agency might hold such sessions with patients in health clinics, and an economic development agency might hold such sessions with representatives from businesses.
Outcome indicators should be reviewed at a higher level to ensure that they contain the important performance information needed. For example, a ministry should review the indicators selected by each of its agencies and an agency should review the indicators proposed by each of its programs.
8. Inventory existing databases within the agency to identify the output and outcome information already available (or readily avail-able) that can be used to monitor progress toward the agency’s mission.
The sources might be inside or outside the agency, such as data already regularly collected by a central planning offi ce. Many, if not most, countries and their agencies already regularly collect outcome data, such as incidence and prevalence of health problems (for example, malaria, tuberculosis, and HIV), road accident fatalities, drinking water and air quality, unemploy-ment rates, school absenteeism and completion rates, or economic develop-ment indicators.
9. For each performance indicator candidate, identify the source of the data, collection procedures, and how often the data will be col-lected and reported. Some outcomes will require new data collection. An
indicator should not be considered fi nal until a feasible data collection pro-cedure has been identifi ed. An agency and its programs should not limit it-self to considering only outcome indicators for which it currently has data.
Depending on the availability of resources, however, the agency may need to defer beginning new data collection procedures until later years. In addition to data obtainable through agency records, two other major data collection options to consider are customer surveys and trained observer ratings.
Surveys of citizens. Surveys are increasingly being used around the world to obtain feedback on outcomes, including the quality of government ser-vices. The feedback can be from samples of households or of customers that have been assisted by the agency (such as patients, students, or businesses).
These surveys can provide feedback on a variety of program outcomes, in-cluding the surveyed person’s housing, employment, or health condition sta-tus; ratings of various program quality characteristics such as timeliness, ac-cessibility, fairness, and ease of obtaining information on services; frequency of use and reasons for nonuse of the service; and extent of having to make inappropriate payments to obtain service. Citizen surveys can also obtain information on the demographic characteristics of respondents, which helps identify the distributional eff ects of the program.
Trained observer rating procedures. This option uses personnel to system-atically rate physically observable conditions, such as the condition of roads, hospitals, health clinics, school buildings, or other public facilities; proxim-ity of households to sources of drinking water; and the abilproxim-ity of persons with disabilities to undertake basic activities of daily living. Each of these conditions is rated by an observer trained in the use of a well-defi ned rating scale. The raters visit all, or samples of, the units to be observed. The out-come indicator would be the number and percent of those items rated as being in satisfactory condition. Ratings obtained at regular intervals enable the program to monitor changes in each condition. The procedure has a po-tentially very useful side benefi t of identifying where work is needed and the extent of that need so units with poor ratings can be prioritized for correc-tion as soon as possible.
10. Provide for basic analysis of the performance data. At the minis-try or agency level, top-level offi cials should assign an offi ce to be responsi-ble for examining the data collected, identifying unexpected fi ndings, and reporting these fi ndings to the appropriate offi cials in time to allow the agency to act on them. The basic analysis for performance monitoring is typically comparisons, including comparing the current reporting period to previous reporting periods, current data to targets, outcomes among diff er-ent customer groups, and to similar programs in other countries. While
more sophisticated statistical analysis would be desirable, this fi rst-level, basic analysis should be highly informative and make the monitoring consid-erably more useful.
11. Top-level offi cials in the agency need to emphasize the impor-tance of outcome information, particularly by using the M&E informa-tion in decision making. They also need to use informainforma-tion collected on outcomes to identify needed program improvements and training or techni-cal assistance needs, and to draw attention to successful programs. Using M&E data at all levels of the agency, such as planning, budgeting, and pro-gram review, highlights the importance of the M&E system.
12. Review monitoring data and data collection procedures periodi-cally. This review is an appropriate time to introduce improved measure-ments to ensure the continued reliability and credibility of performance measurement. Performance measurement should always be considered to be a work in process.
13. Use M&E information to lay the groundwork for positive incen-tives to improve the eff ectiveness of public services. From the beginning, emphasize that achieving good results will be rewarded. Emphasize that a major purpose of M&E information is to help offi cials and their staff identify what works and what does not.
14. Emphasize the message that outcome information is very useful for benchmarking, but does not provide the background information that tells why. The M&E system should not be used as a blame game. M&E information is often perceived by managers as threatening because it has been used to blame managers for poor outcomes without considering why the outcomes occurred. Outcomes are often aff ected by factors over which managers have limited or no control. (See Boxes 5.1 and 5.2 for examples of country service agencies that have made considerable progress in most of these steps.)