Plan for evaluations and special studies

Evaluation is a relatively structured analytical effort undertaken selectively to answer specific management questions regarding USAID-funded assistance programs or activities. Evaluation is also a management tool that plays a vital role in Agency decision-making, accountability reporting, and learning. It is an important source of information about the performance of USAID activities, programs and strategies.

Determine in advance if special studies can be identified (e.g., DHS, intermittent surveys) or if evaluation issues can be predicted. Plan to determine, at the time of the Portfolio Review, if there are performance deviations (positive or negative) that show need for evaluations or studies. Near the end of the SO life, plan to determine if something happened that requires a study to better document the results.

INFORMATION SERVICES:

Research and Reference Services (R&RS): R&RS staff can help you

determine if other evaluations or special studies have been conducted on similar topics. Access R&RS via CDIE Online at http://cdie.usaid.gov (available only within the USAID firewall; click 'Research' or 'Library' at the top of the homepage)

CONSIDER THIS – Situations that Prompt a Need for Evaluation: To complement ongoing performance monitoring, consider planning an evaluation when there is a distinct and clear management need, as in the following

situations:

! A key management decision should be made and there is inadequate information

! Performance information indicates an unexpected result (positive or negative) that should be explained

! Customer, partner, or other informed feedback suggests that there are implementation problems, unmet needs, or unintended consequences or impacts

! Validity of Results Framework hypotheses or critical assumptions is questioned

! Periodic Portfolio Reviews have identified key questions that need to be answered or on which consensus should be developed

! Extracting lessons is important for the benefit of other Operating Units or future programming

TECHNIQUE – Planning for Evaluations: Take into account the following key steps and questions, as listed in Table 2-11, when planning evaluations and special studies.

Table 2-11. Key Steps and Questions for Planning Evaluations

KEY STEPS KEY QUESTIONS

# Decide if and when to evaluate # Clarify the evaluation purpose # Use R&RS for research support # Identify the research questions

# Select appropriate evaluation methods # Plan for data collection and analysis # Form an evaluation team

# Plan procedures (e.g., schedule, logistics, reporting needs, budget)

# Who is likely to need information from or about the program?

# What do they need to know?

# How would they use the information if they had it?

# When do they need it? # How accurate should it be?

# When and how should the data be collected and analyzed?

# Who is responsible for data collection and analysis?

If an evaluation is likely, plan to select an evaluation method from among several options. Table 2-12 presents some evaluation method options. Helpful Hint 4: Rapid Low-Cost Data

Collection Methods provides supplemental information on the appropriateness, advantages, and limitations of rapid appraisal techniques.

Table 2-12. Evaluation Methods and Key Considerations EVALUATION METHOD

! Short workshops to reflect on whether the development hypothesis is valid ! Community interview or customer focus groups

! Large scale surveys

! Rapid appraisal or participatory techniques ! Traditional, formal impact evaluations

KEY CONSIDERATIONS

! Nature of the information, analysis, or feedback needed ! Trade-off between quality/validity and cost

! Cultural considerations

! Time-frame of the management need for information ! Time and resources available

! Level of accuracy required

CONSIDER THIS – Special Studies: In some cases, you may need to conduct special studies that go beyond the scope of program monitoring or evaluation.

One example is the Demographic and Health Surveys (DHS) that are conducted approximately every five years to inform health sector programming.

As discussed in ADS 203.3.6.1, an important role of special studies is to examine whether the needs of vulnerable groups are being met. Special studies need to be designed with the same care as evaluations. When collecting data for analysis, take particular care to design and implement the study in order to minimize error and ensure good data quality.

WORKSHEET 10 – Evaluations and Special Studies Planning: Use Worksheet 10 to help document the possible evaluations and special studies identified to complement performance monitoring. This worksheet will help you identify the subject, timing, and any special research considerations. Table 2-13 presents an example of how an evaluation or special study may be planned.

Table 2-13. Evaluations and Special Studies Planning

Evaluation/Study Subject When Key Research Question(s) Reproductive Health

Survey On-going

What is the status of reproductive health in the country?

What is the nature of citizen’s knowledge, attitudes, behaviors, and practices?

Community Assessment for the Social

Development Fund

April – May, 2000 Which communities should be targeted? How should funds be programmed?

World Vision Feeding

Program April – May, 2000

How effective and efficient has the program been? What are the opportunities for improvement?

Community Development

Approaches April – May, 2000

What are the merits of various approaches? What lessons can be learned from community development efforts in the country?

How can sustainability be ensured?

Mission’s Value-Added

in the Health Sector June – Sept, 2000 Should USAID continue to fund these programs?

Although the timing of the PMP development process may be such that the evaluation will not take place in the immediate future, you may want to consider the types of questions found in Worksheet 11 when the time comes to develop the evaluation scope of work.

WORKSHEET 11 – Evaluation Scope of Work Planning: Use Worksheet 11 to help develop the scope of work for planned evaluations. An evaluation scope of work (SOW) is a plan for conducting evaluations or special studies. It conveys clear directions to the evaluation team. A good SOW usually:

! Provides a brief background on implementation ! Identifies existing performance information sources ! States the purpose, audience and use of the evaluation ! Identifies the evaluation method to answer the questions ! Clarifies the evaluation questions

! Identifies the evaluation method(s) to answer the questions

! Discusses evaluation team composition and participation of customers, partners and stakeholders

! Covers procedures such as schedule and logistics ! Clarifies requirements for reporting and dissemination ! Includes a budget