Assessing Impact of
Development Programmes
on Food Security
Quantitative Methods:
Secondary Data
Learners’ Notes
© FAO, 2011Table of contents
Learning objectives ... 2
Introduction ... 2
When to use secondary data? ... 2
Selecting the right datasets ... 4
Using secondary data ... 6
Summary ... 9
If you want to know more ... 9
Annex I: Potential data sources ... 11
Annex II: Example of use of national statistics ... 13
Learning objectives
At the end of this lesson you will be able to:
• identify the possibilities and limitations of using secondary data for assessing impact;
• understand how to select appropriate datasets to use;
• identify the main ways of using secondary data.
Introduction
We can define secondary data as data gathered by a given party for one purpose that is then utilized by another party for a different purpose.
Almost all impact assessments will use secondary quantitative data as a source of information, even if primary data are also collected for the specific programme.
In this lesson you will see how to use data from secondary sources, ensuring the adequacy and quality of the information they provide.
When to use secondary data?
Primary data collection for impact assessment may provide the most direct and specific information to assessors, however it can be costly and time consuming.
In some cases, good quality data have already been collected, so it may not be necessary to duplicate the effort.
In other cases, limited financial resources or time are an issue for the programme. In which case, it may be less expensive to hire an experienced assessor or statistician to identify data sources and carry out an appropriate data analysis, rather than conduct a survey to collect primary data.
Note that like all other approaches and methods, the use of secondary data requires specialized skills and competencies, particularly in statistics.
This lesson will provide programme managers, M&E staff and external assessors with knowledge about the salient points of using secondary data in impact assessment, including potential data sources.
With this knowledge, you can feel more confident about planning to use secondary data in your impact assessment and you will have the necessary elements to help you employ appropriate persons with the necessary skills to successfully assist you.
The following table compares the strengths, weaknesses, opportunities and constraints of primary and secondary data for assessing programme impact.
Collecting primary data Using secondary data
Strengths • you can control timing of data collection and the location of data to be obtained
• you can use indicators not routinely collected in other surveys
• you have information on programme participation status so you can distinguish participants from non
participants
• less expensive than primary data collection
• no need to set up data collection system
• in many cases, quality of national surveys is high
• national surveys may also provide information at sub national levels
Weaknesses • primary data collection is very expensive
• much time is required for data collection, analysis and
reporting
• cannot control the timeliness of data collection and availability of datasets
• cannot regulate level of disaggregation of data, therefore coverage of project areas cannot be guaranteed
• existing indicators may not be specific enough for impact assessment of the programme
• might be difficult to identify programme participants
Opportunities • strengthens the capacity of programme M&E staff in carrying out primary data collection
• can add key indicators on food security to national surveys such as Household Budget Surveys (HBS), census, Nutrition or Agricultural surveys
• possibility of using small area statistics to disaggregate national surveys to lower levels than the survey foresaw (combined with recent census data) Constraints • may need to set up a new
system from scratch to collect data
• need capacity to design and pre-test questionnaires, train enumerators, supervise data collection, perform analysis and interpret data
• need to obtain a valid and recent sampling frame for sample selection
• lack of control over data collection leads to non specific impact assessment of programmes (e.g. timing of existing surveys may not coincide with baseline or end of programme)
• may require statistical expertise to manipulate data and carry out analyses
Selecting the right datasets
Once you have decided to use secondary data, you will need to know how to go about finding usable datasets and how to decide whether they are of the right quality for carrying out the impact assessment.
There are several steps to follow that can guide you in the process of selecting the right datasets: 1. Clearly define your information needs from secondary data sources
2. Locate the potential datasets
3. Assess whether your potential datasets are usable for assessing impact
1. Clearly define your information needs from secondary data sources
As a first step, you need to have a clear vision of the information you want from a secondary data source. You should consider the following questions:
Which impact indicators do you need to create from secondary data sources?
The data you hope to use should include responses to questions from which you can create food security impact indicators for the unit level (individuals, households, communities) that your programme works at.
Which background characteristic variables should be included in secondary data sources?
You will also need unit level information that describes the population, e.g. demographic characteristics, educational and labour market status, household assets and living
arrangements. This assists in identifying an appropriate control group.
Do the secondary data sources include information on programme participation status?
It will be necessary to distinguish within your potential datasets those individuals, households or communities that participated in the programme from those that did not.
2. Locate the potential datasets
The second step is to locate the potential datasets.
For national or sector wide surveys that have been carried out by government institutions, two logical places to look for data are: government departments and statistical agencies. National surveys carried out on either a regular or periodic basis are typically the preserve of your country’s national statistical organization (NSO)
(http://unstats.un.org/unsd/methods/inter-natlinks/sd_natstat.asp).
Small area surveys and censuses may also be conducted on a sub national basis, in which case it is important to review the archives of regional (states, provinces, local governments) statistical agencies.
Other appropriate datasets may be identified from a number of sources, such as NGOs, universities or research institutes.
Finally, international agencies (including development agencies, donors and United Nations organizations) often play a role in funding data collection efforts and may have access to datasets you could use.
Important note about accessing the data
Census and survey information, once located, can be requested through inter agency exchanges.
For external evaluators, official requests sometimes have to be cleared by senior departmental heads such as permanent secretaries or deputy ministers.
Other roadblocks to data acquisition may emerge. For instance, you may face lengthy delays as you await official clearance. Also, NSO staff may be unable to provide the data in a usable format or not have sufficient staff resources to answer requests. You need to be aware of potential constraints in acquiring data from NSOs and other organizations that have archives of usable data. Occasionally, national datasets are available via Internet from international organizations that partnered with government institutions for data collection, such as Demographic and Health Surveys or poverty assessments. Therefore an Internet search is justified when trying to locate datasets that can be used for the impact assessment.
Please note that often survey data used to create national indicators are also made available to researchers and other users for re-analysis.
3. Assess whether your potential datasets are usable for assessing impact
Once you have identified datasets that match your information needs, you will need to check several data features to judge whether these datasets can be used for your purpose. In particular, you should look at:
a. Reference period
The data used to assess programme impact correspond to time periods that are relevant to the programme. The data need to have been collected close in time to when programme
implementation began and after or near the completion of the programme in order to measure change.
b. Population coverage
The data were collected in the geographic area targeted by the programme c. Completeness
There is little missing data for the variables you need to use; i.e. non responses to questions on household characteristics or questions to be used to create food security indicators
the recorded responses are within plausible limits. The questions asked to respondents were carefully formulated to provide valid information.
A preliminary (but often overlooked) task is to examine the raw data for errors. These may include duplicated records, invalid responses, inconsistent or contradictory coding, and missing data. Because quantitative impact assessment is so heavily data-reliant, its final quality will vary directly with that of its data input.
Probing the data for errors and inconsistencies therefore represents a prudent use of early-stage resources and may lead to eliminating certain datasets from consideration if the problems are excessive.
e. Bias
The sample was chosen validly to represent the population it is meant to cover; certain sub groups or characteristics are not over- or under- represented in the dataset.
Using secondary data
When measuring impact with secondary data, you can either: 1. use information from published national survey reports, or 2. re-analyse data.
1. Use of national indicators from published reports
Large scale food security programmes are part of national initiatives and policies enacted by Governments to reduce poverty and improve the well-being of their citizens.
They contribute towards achievement of international targets such as those indicated in the Millennium Development Goals (MDG) and the World Food Summit (WFS) Plan of Action.
Because of this, one option for assessing your programme with secondary data is to use national-level indicators found in published reports. These can be used to measure the contribution of your programme towards meeting national and international objectives for reducing food insecurity and poverty.
For the purpose of tracking progress towards the achievement of the international targets, countries furnish data to international organizations on a series of indicators that are relevant to food security.
It is quite possible to directly use reported indicators, collected at different times at national level, to assess the impact of large scale food security programmes that are applied to the country as a whole.
This is particularly true when the timing of large scale national surveys corresponds roughly to the beginning and the end of your programme.
You can also use reported sub national level indicators when available, for example if your programme is only operating in rural communities.
The role of the counterfactual when using national statistics to assess impact
When the assumption is that all citizens are intended to benefit from the Programme – that it is truly at national scale – and that the available national statistics are pertinent to your programme, an assessment of progress can be carried out by tracking these national statistics over time to measure change. In this case, you will not be able to use a control group to create a
counterfactual for attributing change to the programme itself.
In reality, of course, not everyone would have an equal opportunity to benefit for a number of different reasons, but taking the example of a national agricultural policy governing trade and markets, it might not be possible to identify a valid control group that did not benefit from the policy in order to create the necessary counterfactual.
2. Re-analysis of secondary data
In the case that the published national indicators are not specific enough to the type of impact your programme wishes to achieve, it may be necessary to re-analyze secondary data in order to:
• create outcome and impact indicators,
• construct a control group,
• fill in information not collected first hand, or
• provide baseline levels when it is possible to identify participants from the data. Note that the same survey datasets that generate national statistics are often available to
researchers or assessors for re-analysis.
Programme-specific indicators can be created from secondary data, provided that the data contain sufficient information to create the indicators and that information is available at the level of observation you are interested in (for example household or community).
To do this, you will need to have access to the raw data (i.e. databases with all the data from which you will do the analysis).
Take the case of a programme that conducted a post programme survey with both a participant and a control group but did not carry out a baseline survey.
In this case you may want to recreate a baseline situation for both participants and non participants. However, data collected for another purpose may not allow you to identify your actual programme participants and distinguish them from non participants.
This is an important limiting factor of using secondary data when the programme is not at
national scale. Therefore, it is important to determine early on whether such a situation exists and if other techniques can be used to resolve the problem.
When there is no indication of programme participation status in the data you want to use, you must try to find a way to assign that status by triangulating other sources of information. This reinforces the usefulness of collecting administrative data and using key informants. In cases where programmes have targeted specific individuals or households for participation, household surveys carried out by others in the same geographical location may include a variable that identifies participation status, although this is not very common.
With census data, it may be possible to identify communities that you know have taken part in the programme.
Secondary data can be re-analyzed using specific statistical techniques to extract households that can be used as a control group.
For example, this holds when a programme collected primary data from participant households at baseline and at the end, but:
• included a control group only for the end-of-programme survey; or
• did not collect data on a control group at either time period.
Secondary data can also be used to fill in missing information that might not have been collected first hand by the programme, such as market or climate data at different times.
Summary
Use of secondary data to assess programme impact is a wise choice when appropriate data can be located or when the programme has limited financial or human resources to collect data directly.
National statistics can be used as reported to measure improvement towards meeting national and international goals, for example, the MDGs or the World Food Summit Goals, especially when the scope of the programme is to benefit all citizens.
Secondary data may be re-analyzed, using specific statistical techniques, to create a control group, provide baseline levels when it is possible to identify participants from the data, or fill in information not collected first hand.
If you want to know more
Online resources
FAO. 2008. Deriving Food Security Information from National Household Budget Surveys. Rome. ftp://ftp.fao.org/docrep/fao/011/i0430e/i0430e00.pdf
FAO. 2008. Global Donor Platform for Rural Development, World Bank, Tracking results in agriculture and rural development in less-than-ideal conditions. Rome.
http://www.fao.org/fileadmin/templates/ess/documents/Sourcebook-Web-Version.pdf FAO. 2009. Pathways to Success. Rome.
http://www.fao.org/fileadmin/user_upload/newsroom/docs/pathways.pdf
Ravallion, Martin. 1999. The mystery of the vanishing benefits: Ms. Speedy Analyst's introduction to evaluation. World Bank Policy Research Working Paper No. 2153. Washington, D.C.: World Bank. http://www.acoes.org.co/pdf/Documentos%20HFTF/33.pdf
Wassenich, Paul. 2007. Data for impact evaluation. Washington, D.C.: World Bank. http://www-wds.worldbank.org/external/default/WDSContentServer/WDSP/IB/2008/02/05/000333037_2008020 5043847/Rendered/PDF/423800NWP0Doin10Box321452B01PUBLIC1.pdf
Online links to national data sources and information on specific national surveys: Demographic and Health Surveys (DHS) website. http://www.measuredhs.com/
FEWSNET homepage http://www.fews.net/Pages/default.aspx
FAO/WFP Joint Guidelines for Crop and Food Security Assessment Missions.
http://www.wfp.org/content/faowfp-joint-guidelines-crop-and-food-security-assessment-missions-cfsams
Integrated Food Security Phase Classification homepage. http://www.ipcinfo.org/overview.php International Household Survey Network home page. http://surveynetwork.org/home/
Multiple Indicator Cluster Survey website. http://www.childinfo.org/mics.html National Household Survey basics
https://www.gtap.agecon.purdue.edu/events/Conferences/2006/documents/HHSurveyBasics_GTAP _POSTCW06.pdf
Smith, L. The Use of Household Expenditure Surveys for the Assessment of Food Security. FAO International Scientific Symposium on Measurement of Food Deprivation and Undernutrition. 2002. http://www.fao.org/docrep/005/Y4249E/Y4249E00.HTM
http://www.fao.org/docrep/005/Y4249E/y4249e08.htm#bm08 World Bank. Living Standard Measurement Surveys.
http://econ.worldbank.org/WBSITE/EXTERNAL/EXTDEC/EXTRESEARCH/EXTLSMS/0,,contentMDK:2 1610833~pagePK:64168427~piPK:64168435~theSitePK:3358997,00.html
World Bank, FAO. 2008. Tracking results in agriculture and rural development in less-than-ideal conditions. Rome.http://www.fao.org/fileadmin/templates/ess/documents/Sourcebook-Web-Version.pdf]
World Food Programme. Comprehensive Food Security and Vulnerability Analysis Guidelines. 2009.
http://www.wfp.org/content/comprehensive-food-security-and-vulnerability-analysis-cfsva-guidelines-first-edition
FAO 2010. World Programme for the Census of Agriculture. http://www.fao.org/economic/ess/ess-wca/en/
Additional Reading
Bamberger, M. 2006. Conducting quality impact evaluations under budget, time and data constraints. Washington, D.C.: World Bank.
Carleto, C,, and Morris, S.S. 1999. Designing methods for the monitoring and evaluation of
household food security rural development projects. Technical Guide #10. International Food Policy Research Institute. Washington, D.C.: IFPRI.
Habicht, J.P., Pelto G.H., and Lapp, J. 2008. Methodologies to evaluate the impact of large-scale nutrition projects. Washington, D.C.: World Bank.
Ravallion, Martin. 2009. Evaluation in the practice of development. World Bank Research Observer
Annex I: Potential data sources
The following table lists potential sources of data and the types of food security and nutrition information they can contribute towards assessment of large scale food security programmes:
Potential sources Indicators
Household Budget or Income and Expenditure Surveys, Living Standards
Measurement Surveys (World Bank)
At household level: income sources, expenditures (of particular interest for food, school, health), poverty indicators, food consumption and expenditures, undernourishment.
Demographic and Health surveys (USAID) / Multiple Indicator Cluster
Surveys (UNICEF)
Maternal and child diet, mortality, nutritional status, knowledge, water/sanitation, gender-specific literacy and school enrolment, access to health care.
National Nutritional surveys Individual nutritional status, household and individual diet, household food security indicators.
Agricultural surveys (e.g. WFP/FAO Crop assessments, Agricultural Census,
production surveys)
Crop and livestock data, production, productivity, input use, farmer organizations.
Food security and vulnerability surveys, early warning systems, monitoring and information systems (such as: WFP’s
Comprehensive Food Security and Vulnerability Analysis, Early Warning
Systems, Famine Early Warning Network (FEWSNET), Integrated Phase
Classification)
Individual nutritional status, household dietary diversity, coping strategies, livelihood data, food availability and access, market and trade, shocks and climate.
One appropriate source of data on relevant food security indicators is MDG Monitor which also tracks progress towards achieving MDG Goal Number 1: Eradicate Extreme Poverty and Hunger, through specific indicators which could be used to assess improvement in food security, such as prevalence of underweight children under-five years of age and proportion of population below minimum level of dietary energy requirement, or the prevalence of undernourishment
http://www.fao.org/economic/ess/ess-fs/en/ ).
Another source of data for tracking a country’s progress is through monitoring of its Poverty Reduction Strategies process
(http://siteresources.worldbank.org/INTPAME/Resources/Pov-Mon-Systems/PRSMonitoringDiagnosticTool_English.pdf).
Annex II: Example of use of national statistics
The following table demonstrates the use of national statistics at different time periods to track progress on improving food security.
Food Security Dimension
Indicator Time Period
Armenia Brazil Nigeria Vietnam
Distribution of Wealth
Poverty Gap Ratio (less than USD 1.25 per day, percent)
1996 5 3 32 24
2002 3 2 30 22
Food Availability
Dietary Energy Supply (kcal per capita) 1990-92 1960 2810 2540 2180 2004-06 2290 3090 2650 2680 Food Accessibility Road Network (km per 1000 rural persons) 1990-92 7,2 44,5 2,2 1,8 2004-06 6,9 58,1 2,5 3,7 Food Utilization (safe food) Share of population with access to clean water, percent 1990-92 - 83 49 65 2004-06 92 90 49 85 Outcome (active life) Level of undernourishment, percent of the population 1990-92 46 10 15 28 2004-06 23 6 8 13
Modified from FAO 2009, Pathways to success: Success stories in agricultural production and food security, table p. 7.
Annex III: Case study
A Case Study of Rural Road Rehabilitation in Trinidad and Tobago in the 1990s Ainsley Charles
Road infrastructure has traditionally been regarded as a critical input for the rural economy in general and for the agricultural sector in particular, since it serves to lower transportation costs, to relax constraints to market access, and consequently to boost productivity. Farmers in road-deprived areas cannot easily exploit available market opportunities and so are more likely to choose to produce less risky staple crops over market crops, and to do so using traditional, lower-yield methods. They also tend to under-invest in health and education for themselves and for their children which, in a dynamic setting, lowers their long-term prospects of escaping poverty and food insecurity. One objective of rural road rehabilitation programmes, therefore, is to reverse (or moderate) these adverse outcomes by promoting a higher degree of market integration, the adoption of modern agricultural techniques, and the formation of enhanced human capital.
1. Objective: The objective of this study was to assess whether rehabilitated roads, as compared to unimproved roads that restrict farmers’ access to agricultural plots, can lead to a) increased school
attendance and schooling tenure among school-age children and school leavers, respectively; b) increased use of modern agricultural techniques such as high-yield seed varieties; c) an increase in the variety of crops and in the proportion of marketed harvest; and d) a boost in overall agricultural production in low-income agricultural communities.
2. Beneficiaries: Beneficiaries were farmers and farming households in 50 rural communities throughout Trinidad and Tobago.
3. Programme intervention: Unimproved local roads and tracks that provided access to agricultural plots and a direct link to higher-level national roads were rehabilitated up to a standard of all-weather permanent access. These agricultural access roads, located in 50 programme communities, typically measured between two to three kilometres in length and were selected after having met a series of agriculture ministry-approved technical, economic and environmental criteria. To facilitate the impact assessment, agricultural communities in which access roads were not refurbished were chosen as potential comparison units.
4. Design: The programme was not randomised across communities but was instead targeted to certain communities but not to others. It was therefore necessary to construct a valid counterfactual group from the datasets and minimize differences between targeted farming communities and similar communities selected as controls. To select the best match for targeted communities among all other communities for whom
information was available in the datasets, a statistical technique called propensity-score matching was employed. 1
5. Data: Only secondary data were available for the impact assessment. Data came from multiple sources but were primarily composed of programme documentation and micro-unit census data. Programme
documents were retrieved from the Inter-American Development Bank, a multilateral development institution operating throughout the Americas and the programme’s principal funder. Analytical and feasibility studies and administrative and project monitoring reports came from project consultants and government ministries, respectively. Key informants provided additional institutional details and expert knowledge about the
programme during a series of in-country interviews. And the national statistical agency provided unit-level data in the form of two population censuses and an agricultural census.
6. Methods: Analysis was conducted at the level of the community, a level of aggregation that was both closest to the actual level of the intervention and identifiable in all three census datasets. First, a model of community programme participation was estimated using maximum-likelihood logit regression which related to a set of key household-, head of household-, labour force-, and community-related characteristics. Using the Propensity Score Matching technique, non programme communities were selected that were most similar to the programme communities. This allowed having two sets of communities (i.e. participating and control) from which it was possible to compare outcome and impact variables to estimate programme impact
7. Outcome measures: For education the two primary outcomes indicators were the school attendance rate and the number of years of schooling (for new labour market entrants). For agriculture, the study used a series of outcomes indicators relating to a mix of agricultural inputs and output.
Results: In respect of education, school-age children living in communities with rehabilitated roads attended school in higher proportion compared with children in non-programme comparison communities. Early-childhood attendance was up to 6.3 percentage points higher, primary school attendance up to 2.0 percentage points higher, and advanced secondary school attendance up to 4.4 percentage points higher. Nineteen- and twenty-four-year-olds from programme communities newly entering the labour market had up to one-half of a year more of completed education compared with similar workers from non-programme communities. In terms of agriculture, farming operators in programme communities were 11 percentage points less likely to retain an informal land tenure status compared with operators in non-programme communities. Also, a lower proportion (-10 percentage points) worked full-time at agriculture, perhaps suggesting a diversification towards other employment options. Within agriculture, programme community farmers used more modern high-yield seed varieties when cultivating pulses (corn) and leafy vegetables
1 The propensity score matching technique, used in this study to match potential control communities with
(cabbage; lettuce). In addition, they harvested a higher quantity of corn, achieved higher yields of corn and string beans, and marketed a larger share of the total harvest of chive, pigeon peas, and pumpkin.
9. Findings: The study showed that in road-deprived agricultural communities, upgrading roads to permanent-access standard was likely to increase school attendance, extend the number of years spent in school for young labour market entrants, and spur changes in the agricultural input and output mix available to farmers.
10. Lessons: Impact assessment of large-scale, nationally-run programmes is possible, even when primary data are unavailable. In this case, it was possible to use secondary data such as censuses and ongoing survey, providing such data had a minimum number of key features. These features included the ability to identify programme participation, socioeconomic and demographic data for the communities included in the study as well as availability of outcome and impact variables of interest.
