Data Collection Method Types and Sources

The following summarizes key data collection methods and tools used in monitoring and evaluation (M&E) in alphabetical order. This list is not exhaustive, as tools and techniques are emerging and evolving in the M&E field. For each, some key sources are included for further information.

Method Overview and Methodology When to use Method

After-Action

Review A facilitated discussion that focuses on four questions: • What was planned?

• What actually happened? • What went well?

• What could have been better?

For internal reflection, review and lesson learning following a specific activity, an event or a project.

Case Study A detailed descriptive narrative of individuals, communities, events, programmes, time periods, or a story (discussed below). They are particularly useful in evaluating complex situations and exploring qualitative impact. A case study only helps to illustrate data and find commonalities; only when combined (triangulated) with other case studies or methods can one extrapolate key principles. To write a case study of a project, consider the following questions:

What type of project is it? What does it aim to achieve? How will it achieve this aim? What will the final output be?

How many people are being assisted, and what proportion of the total catchment area is this? Why was this community selected?

Useful through-out a project to document examples of project achievements. Useful to exemplify specific activities or effects on individual households, particularly for inclusion in internal or donor reports or for communications / media messaging.

Case Study

(contd) What is the impact on beneficiaries and how was it achieved? When profiling an individual

beneficiary consider: Personal details (e.g. Name, age, family size, who is head of HH, family circumstances, current income, current sources of income and coping strategies); Context of the person›s life: What major changes have happened in their life in the past 5 months? What assistance is the beneficiary receiving? Why? How does the beneficiary feel he/she is benefiting? What difference is this assistance making to the beneficiary? What hopes does he/she have for the future?

Useful through-out a project to document examples of project achievements. Useful to exemplify specific activities or effects on individual households, particularly for inclusion in internal or donor reports or for communications / media messaging.

Checklist A list of items used for validating or inspecting that procedures/steps have been followed, or the presence of examined behaviors. Checklists allow for systematic review that can be useful in setting benchmark standards and establishing periodic measures of improvement.

Useful reminders for project teams that have multiple priorities.

Community Book A community maintained document of a project

belonging to a community. It can include written records, pictures, drawings, songs or whatever community members feel is appropriate. Where communities have low literacy rates, a memory team is identified whose responsibility it is to relate the written record to the rest of the community in keeping with their oral traditions.

Useful where high levels of illiteracy to help communities monitor and document change.

Community Interviews / Meetings

A form of public meeting open to all community members. Interaction is between the

participants and the interviewer, who presides over the meeting and asks questions following a prepared interview guide.

Useful to provide information to

communities for project kick-off, monitoring and verbal reporting back.

Desk / Document

/ Literature review A review of documents (secondary data) can provide a cost effective and timely baseline or other information and a historical perspective of the project. This is a key first step in any data collection process. It includes written documentation, (i.e. project records and reports, administrative databases, training materials, correspondence, legislation, and policy documents), as well as videos, electronic data or photos. However, it can be difficult to assess the reliability and validity of some sources.

Research before a project starts and as part of an evaluation.

Ethnographic

interviewing In-depth interviewing of a limited number of individuals to provide a good picture of how a particular event has affected them. Helps to put human detail into a larger picture.

In depth research into the impact of an event.

Focus group

discussion Focused discussion with a small group (usually 8 to 12 people) of participants to record attitudes, perceptions, and beliefs pertinent to the issues being examined. A moderator introduces the topic and uses a prepared interview guide to lead the discussion and elicit discussions, opinions, and reactions. A low-cost and efficient means of collecting beneficiary views in a quicker but less rigorous way than a formal survey.

To explore issues in more detail as part of research on why certain things are happening or understand change.

Interviews An open-ended (semi-structured) interview is a technique for questioning that allows the interviewer to probe and pursue topics of interest in depth (rather than just “yes/ no” questions). A closed-ended (structured) interview systematically follows carefully organized questions (prepared in advance in an interviewer’s guide) that only allow a limited range of answers, such as “yes/no,” or expressed by a rating/number on a scale. Replies can easily be numerically coded for statistical analysis.

Tend to be used as part of a quantitative survey with individuals or households.

Key informant

interviews An interview with a person having special information about a particular topic. These interviews are generally conducted in an open- ended or semi-structured fashion.

When seeking specific information (e.g. from experts).

Laboratory

testing Precise measurement of specific objective phenomenon, for example, for iron content, seed, food or water quality testing.

Resource quality- checks or medical checks.

Mini-survey Data collected from interviews with 25 to 50 individuals, usually selected using non- probability sampling techniques. Structured questionnaires with a limited number of closed-ended questions are used to generate quantitative data that can be collected and analyzed quickly.

Seeks information on specific issues using a small sample.

Most Significant

Change (MSC) A participatory monitoring technique based on stories about important or significant changes, rather than indicators. They give a rich picture of the impact of development work and provide the basis for dialogue over key objectives and the value of development programs.

To get a detailed qualitative overview of change faced by individuals/households over time.

Observation A record of what observers see and hear at a specified site, using a detailed observation form. Observation may be of physical surroundings, activities, or processes. It is a good technique for collecting data on behavior patterns and physical conditions. It is a very useful method, especially for triangulating the information from other sources. It can also be used as a primary data collection method (e.g. observing food distributions) but needs to be combined with other data collection methods (such as interviews) to ensure the observations are not misinterpreted.

Useful for project monitoring by staff or donors.

On-line survey Limited to those with internet access. Allows quick and cheap surveys that can be used to identify issues for further analysis, e.g. through www.surveymonkey.com

Where target populations have internet access.

Participant

observation A technique first used by anthropologists; it requires the researcher to spend considerable time with the group being studied (days) and to interact with them as a participant in their community. This method gathers insights that might otherwise be overlooked, but is time- consuming.

For in-depth anthropological research.

Participatory

Project Review A form of participatory self evaluation which can be tailored to different timeframes and contexts according to need. It combines participatory methodologies, drawing from Empowerment Evaluation, and Most Significant Change. Source: Fetterman, D. M. (2001). Foundations of Empowerment Evaluation. Sage

Publications. Thousand Oaks, London, New Delhi. http://evaluation.blogspot.com/ and http://wwwstatic.kern.org/gems/region4/ DavidFettermanPresentation.pdf

Similar to AAR can be used for internal review, but includes beneficiaries.

Participatory rapid (or rural) appraisal (PRA)

This uses community engagement techniques to understand community views on a particular issue. They enable those from outside the community to capture knowledge that is held by the community. PRA tools can be thought of as helping communities to overtly analyse issues and to translate their analysis into a format that those outside the community can understand. They are usually done quickly and intensively – over a 2 to 3-week period. Examples of PRA techniques include (see Annex 33 - Establishing a Community-Based Monitoring System Guidance Note and Annexes 40-51): • Calendars (seasonal, 24 hour, multi-

annual) and other calendars

• Proportional piling

• Ranking (pair-wise, wealth, seeds, coping

strategies, etc)

• Transect walk

• Mapping (wealth, hazard, mobility,

social, resource, risk, network, influence, relationship etc)

• Venn diagrams • Time lines/histories • Stakeholder analysis

Useful throughout the project at assessment, planning, monitoring and evaluation phases to get more in-depth information about a community.

Questionnaire A data collection instrument containing a set of questions organized in a systematic way, as well as a set of instructions to the enumerator/ interviewer about how to ask the questions (typically used in a survey).

Useful at assessment, monitoring and evaluation phases.

Rapid appraisal

(or assessment) A quick cost-effective technique to gather data systematically for decision-making, using qualitative and quantitative methods, such as site visits, observations, and sample surveys. This technique shares many of the characteristics of participatory appraisal (such as triangulation and multi-disciplinary teams) and recognizes that indigenous knowledge is a critical consideration for decision-making.

For quick assessments.

Seasonal

calendar A graphical presentation of the months in which food and cash crop production and key food and income acquisition strategies take place, also showing key seasonal periods such as the rains, periods of peak illness and the lean season.

Source: The Practitioners’ Guide to the Household Economy Approach; Regional Hunger and Vulnerability Programme (RHVP), Save the Children UK (SC UK) and the Food Economy Group (F.E.G.). http://www.docstoc. com/docs/3466254/THE-PRACTITIONERS- GUIDE-TO-THE-HOUSEHOLD-ECONOMY- APPROACH-The-Food

At planning stage when planning activities for the year.

Statistical data

review A review of population censuses, research studies, and other sources of statistical data. At planning, monitoring and evaluation phases.

Story-telling/

collection Obtaining participants experiences of change by collating their observations of an event or a series of events. A success story illustrates a project’s impact by detailing an individual’s positive experiences in his or her own words. A learning story focuses on the lessons learned through an individual’s positive and negative experiences (if any) with a project. The Most Significant Change technique is an example of this method.

For monitoring and evaluation. Can also be helpful in setting qualitative baselines.

Survey Systematic collection of information from a defined population, usually by means of interviews or questionnaires administered to a sample of units in the population (e.g., person, beneficiaries, adults, etc.). An enumerated survey is administered by someone trained (an enumerator) to record responses from respondents. A self-administered survey is a written survey completed by the respondent, either in a group setting or in a separate location. Respondents must be literate.

Useful at assessment, and monitoring and evaluation phases.

Visual techniques These include maps (e.g. zoning maps),

diagrams, calendars, timelines, and other visual displays to examine the study topics. Participants can be prompted to construct visual responses to questions posed by the interviewers, for example, by constructing a map of their local area. This technique is especially effective as a participatory technique where verbal methods can be problematic due to low literacy or mixed language populations or in situations where the desired information is not easily expressed in either words or numbers.

At assessment, planning, monitoring and evaluation phases.

Zoning The mapping of differences in geography, agro-ecology and types of livelihoods present in the area to be surveyed, to facilitate analysis of FSL-related challenges that may occur in an at- risk area as well as consider response options.

At assessment and planning phases.

Adapted from: Buchanan-Smith, M. & Cosgrave, J. (2010) Evaluation of Humanitarian Action, ALNAP