Limitations of the study
Chapter 3 Appendix: Significance testing using permutation tests
The classical exposition of permutation tests for hypothesis testing and, more generally, of permutation inference is presented in Fisher (1935).† Despite its popularity in the analysis of clinical trials, its use in economics is relatively new. Permutation tests are significance tests based on permutation samples drawn at random from the data in a way that is consistent with the study design. Permutation sampling is performed by assigning the full set of observed outcomes at random to both treatment and control groups under the null hypothesis that the treatment has no effect on the outcome of interest. To this end, permutation tests do not require a model of outcomes but rather a model of assignment. The permutation samples are then employed to build a permutation distribution, which is used to test for whether the size of the treatment effect of interest is occurring “just by chance”.
The general procedure for a permutation test is as follows. First, one computes the statistic that measures the size of the treatment effect of interest under the original data. Second, one pools the observations of the treatment and control groups and randomly draws permutation samples. Third, one constructs the permutation distribution of the statistic by using each of the randomly selected permutation sample. Lastly, one computes the p-value of the statistic by locating the original statistic on the permutation distribution. A value in the main body of the distribution could easily be occurring just by chance. A value in the tail, however, would rarely occur just by chance and presents evidence that something other than chance is underlying the data generation process. In this sense, permutation tests partially protect us from low power. Yet, when the true treatment effect is large, this technique has low power relative to more parametric approaches because it does not put even a minimal structure on the error term (Kremer, Bloom, Bhushan, Clingingsmith, Hung, King, Loevinsohn, and Schwartz, 2006).
To illustrate how I use permutation tests in my setting to compute the p-value of the t-statistic of the difference in means between Chaskinet and control groups, consider for instance the case of the p-values reported in column (3) of Table 3.5. The domain of both distributions is 8
4
= 8!/(4!)2 = 70 (i.e. there are 70 possible ways of dividing the 8 district councils into 2 treatment groups of 4 district councils each).‡ As the number of clusters is small, I perform exact permutation tests (i.e. I calculate the values of the t-test statistic under all these possible permutations of the treatment status). The set of these calculated statistics are used to estimate two separate permutation distributions for each of the two outcomes of interest
†Rosenberger and Lachin (2002) present an up-to-date comprehensive exposition of permutation inference in clinical trials.
Permutation tests are also called randomization tests or re-randomization tests.
‡When blocking is explicitly taken into consideration, the domain of the distribution is 4 Y s=1 2 1 s = 24 = 16 (i.e. there are 16 possible ways of dividing the 8 district councils into 4 pairs, each with one and only one treated unit). To this extent, therefore, the blocking into pairs makes the procedure a bit simpler, but also less powerful to detect treatment effects.
under the null that the Chaskinet does not have any effect on these outcomes. Given these lists of possible realizations, I now calculate the probability of the observed difference in means in the standard way (i.e. by locating it in the distribution and taking into consideration the fact that, under the null, each of the realizations has an equal chance of 1/70 of being selected).
References
Angrist, J., and J.-S. Pischke (2009): Mostly Harmless Econometrics: Am Empiricist’s Companion. Princeton University Press.
Angrist, J. D., G. W. Imbens, andD. B. Rubin (1993): “Identification of Causal Effects Using Instru- mental Variables,” Journal of the American Statistical Association, 91(434), 444–455.
Baland, J.-M., andJ. A. Robinson (2000): “Is Child Labor Inefficient?,” Journal of Political Economy, 108(4), 663–679.
Banerjee, A., A. Deaton, and E. Duflo (2004): “Wealth, Health, and Health Services in Rural Ra- jasthan,” American Economic Review, 94(2), 326–330.
Banerjee, A., and E. Duflo (2006): “Addressing Absence,” Journal of Economic Perspectives, 20(1), 117–132.
Banerjee, A. V., R. Banerji, E. Duflo, R. Glennerster, and S. Khemani (2010): “Pitfalls of Participatory Programs: Evidence from a Randomized Evaluation in Education in India,” American Economic Journal: Economic Policy, 2(1), 1–30.
Bardhan, P. (2002): “Decentralization of Governance and Development,” Journal of Economic Perspec- tives, 16(4), 185–205.
Bardhan, P.,and D. Mookherjee (2006): “Pro-poor targeting and accountability of local governments in West Bengal,” Journal of Development Economics, 79(2), 303–327.
Basu, K., S. Das, and B. Dutta (2010): “Child labor and household wealth: Theory and empirical evidence of an inverted-U,” Journal of Development Economics, 91(1), 8–14.
Beegle, K., R. H. Dehejia, and R. Gatti (2006): “Child labor and agricultural shocks,” Journal of Development Economics, 81(1), 80–96.
Bertrand, M., S. Djankov, R. Hanna, and S. Mullainathan (2007): “Obtaining a Driver’s License in India: An Experimental Approach to Studying Corruption,” The Quarterly Journal of Economics, 122(4), 1639–1676.
Bertrand, M., E. Duflo, and S. Mullainathan (2004): “How Much Should We Trust Differences-in- Differences Estimates?,” The Quarterly Journal of Economics, 119(1), 249–275.
Bezerra, M., A. Kassouf,andM. Arends-Kuenning (2009): “The Impact of Child Labor and School Quality on Academic Achievement in Brazil,” IZA Discussion Papers 4062, Institute for the Study of Labor (IZA).
Bhalotra, S.,andC. Heady (2003): “Child Farm Labor: The Wealth Paradox,” World Bank Economic Review, 17(2), 197–227.
Bresnahan, T., and R. Gordon (1997): The Economics of New Goods (National Bureau of Economic Research Studies in Income and Wealth), vol. 58. University of Chicago Press.
Colecchia, A., and P. Schreyer (2001): “ICT Investment and Economic Growth in the 1990s: Is the United States a Unique Case? A Comparative Study of Nine OECD Countries,” Oecd science, technology and industry working papers, OECD, Directorate for Science, Technology and Industry.
de Janvry, A., F. Finan, and E. Sadoulet (2010): “Local Electoral Incentives and Decentralized Program Performance,” NBER Working Papers 16635, National Bureau of Economic Research, Inc. Del Carpio, X. V. (2008): “Does child labor always decrease with income ? an evaluation in the context
of a development program in Nicaragua,” Policy Research Working Paper Series 4694, The World Bank. Donald, S. G., and K. Lang (2007): “Inference with Difference-in-Differences and Other Panel Data,”
The Review of Economics and Statistics, 89(2), 221–233.
Draca, M., R. Sadun, and J. V. Reenen (2006): “Productivity and ICT: A Review of the Evidence,” CEP Discussion Papers dp0749, Centre for Economic Performance, LSE.
Dumas, C. (2007): “Why do parents make their children work? A test of the poverty hypothesis in rural areas of Burkina Faso,” Oxford Economic Papers, 59(2), 301–329.
Duryea, S., D. Lam, andD. Levison (2007): “Effects of economic shocks on children’s employment and schooling in Brazil,” Journal of Development Economics, 84(1), 188–214.
Duryea, S., A. Olgiati, and L. Stone (2006): “The Under-Registration of Births in Latin America,” RES Working Papers 4443, Inter-American Development Bank, Research Department.
Edmonds, E. (2006): “Child Labor and Schooling Responses to Anticipated Income in South Africa,” Journal of Development Economics, 81(2), 386–414.
Faguet, J.-P. (2004): “Does decentralization increase government responsiveness to local needs?: Evidence from Bolivia,” Journal of Public Economics, 88(3-4), 867 – 893.
(2006): “Decentralizing Bolivia : Local government in the jungle,” in Decentralization and local governance in developing countries, ed. by P. Bardhan,andD. Mookherjee. MIT Press, London, UK. Faguet, J.-P., and F. Sanchez (2008): “Decentralization’s Effects on Educational Outcomes in Bolivia
and Colombia,” World Development, 36(7), 1294–1316.
Filmer, D., and L. Pritchett (1998): “Estimating wealth effects without expenditure data - or tears : with an application to educational enrollments in states of India,” Policy Research Working Paper Series 1994, The World Bank.
Fisher, R. A. (1935): The Design of Experiments. Oliver and Boyd, Edinburgh, 8th edn., Hafner, New York, 1966.
Fisman, R., andJ. Svensson (2007): “Are corruption and taxation really harmful to growth? Firm level evidence,” Journal of Development Economics, 83(1), 63–75.
Fitzsimmons, E. (2003): “The Effect of Risk on Education and Child Labor,” Working Papers 02/07, Institute for Fiscal Studies.
Foster, A. D.,andM. R. Rosenzweig (2001): “Democratization, Decentralization and the Distribution of Local Public Goods in a Poor Rural Economy,” Discussion paper, PIER Working Paper No. 01-056. Frey, B. S. (1971): “Why do High Income People Participate More in Politics?,” Public Choice, 11, pp.
101–105.
Galasso, E., andM. Ravallion (2005): “Decentralized targeting of an antipoverty program,” Journal of Public Economics, 89(4), 705–727.
Gamboa-Rivera, R. (2006): “Evaluaci´on de la Sostenibilidad del Pago del Bonosol,” An´alisis Econ´omico 21, Unidad de An´alisis de Pol´ıticas Sociales y Econ´omicas. La Paz- Bolivia.
Gong, T. (2002): “Dangerous collusion: Corruption as a Collective venture in Contemporary China,” Communist and Post-Communist Studies, 35(1), 85–103.
Griliches, Z. (1998): “Productivity, R&D, and the Data Constraint,” in R&D and Productivity: The Econometric Evidence, NBER Chapters, pp. 347–374. National Bureau of Economic Research, Inc. Grossman, S. J.,and O. D. Hart (1983): “An Analysis of the Principal-Agent Problem,” Econometrica,
51(1), 7–45.
Guarcello, L., F. Mealli, and F. Rosati (2010): “Household vulnerability and child labor: the effect of shocks, credit rationing, and insurance,” Journal of Population Economics, 23(1), 169–198.
Gugerty, M. K.,andM. Kremer (2000): “Outside Funding of Community Organizations: Benefiting or Displacing the Poor?,” NBER Working Papers 7896, National Bureau of Economic Research, Inc. Hahn, J., P. Todd, and W. Van der Klaauw (2001): “Identification and Estimation of Treatment
Effects with a Regression-Discontinuity Design,” Econometrica, 69(1), 201–09.
Hoddinott, J., M. Adato, T. Besley,and L. Haddad (2001): “Participation and Poverty Reduction: Issues, Theory, and New Evidence from South Africa,” FCND Discussion Paper 98, International Food Policy Research Institute.
Imbens, G. W., and J. D. Angrist (1994): “Identification and Estimation of Local Average Treatment Effects,” Econometrica, 62(2), 467–75.
Inter-American Development Bank (2011): “Development Connections,” Flagship publication, Washington DC.
Jacoby, H. G. (1994): “Borrowing Constraints and Progress through School: Evidence from Peru,” The Review of Economics and Statistics, 76(1), 151–60.
Jacoby, H. G., andE. Skoufias (1997): “Risk, Financial Markets, and Human Capital in a Developing Country,” Review of Economic Studies, 64(3), 311–35.
Jimenez, E., and V. Paqueo (1996): “Do local contributions affect the efficiency of public primary schools?,” Economics of Education Review, 15(4), 377–386.
Jimenez, E., and Y. Sawada (1999): “Do Community-Managed Schools Work? An Evaluation of El Salvador’s EDUCO Program,” World Bank Economic Review, 13(3), 415–41.
Kernan, W. N., C. M. Viscoli, R. W. Makuch, L. M. Brass,andR. I. Horwitz (1999): “Stratified Randomization for Clinical Trials,” Journal of Clinical Epidemiology, 52(1), 19 – 26.
Kezdi, G. (2005): “Robust Standard Error Estimation in Fixed-Effects Panel Models,” Econometrics, EconWPA.
Khwaja, A. I. (2001): “Can Good Projects Succeed in Bad Communities? Collective Action in the Himalayas,” Discussion paper, Harvard University, Department of Economics, Cambridge, Mass.
King, E.,and B. Ozler (2000): “What‘s Decentralization Got to Do with Learning? Endogenous School Quality and Student Performance in Nicaragua,” Discussion paper, World Bank, Development Research Group, Washington, D.C.
Kohl, B. (2003): “Democratizing Decentralization in Bolivia: The Law of Popular Participation,” Journal of Planning Education and Research, 23, 153–164.
Kremer, M., E. Bloom, I. Bhushan, D. Clingingsmith, R. Hung, E. King, B. Loevinsohn, and
Lazar, S. (2004): “Personalist Politics, Clientelism and Citizenship: Local Elections in El Alto, Bolivia,” Bulletin of Latin American Research, 23(2), 228–243.
(2005): “Citizens despite the state: Everyday corruption and local politics in El Alto-Bolivia,” in Corruption: Anthropological Perspectives, ed. by D. Haller,andC. Shore. London: Pluto Press.
(2006): “El Alto, Ciudad Rebelde: Organisational Bases for Revolt,” Bulletin of Latin American Research, 25(2), 183–199.
(2007): El Alto, rebel city: Self and citizenship in Andean Bolivia. Duke University Press.
Lundberg, S., R. Startza, and S. Stillman (2003): “The retirement-consumption puzzle: a marital bargaining approach,” Journal of Public Economics, 87(5-6), 1199–1218.
Machin, S., S. McNally, andO. Silva (2006): “New Technology in Schools: Is There a Payoff?,” CEE Discussion Papers 0055, Centre for the Economics of Education, LSE.
Manser, M.,andM. Brown (1980): “Marriage and Household Decision-Making: A Bargaining Analysis,” International Economic Review, 21(1), 31–44.
Mansilla, H. C. F. (2003): La Polic´ıa Boliviana: Entre los c´odigos informales y los intentos de modern- izaci´on. Plural Editores, La Paz-Bolivia.
Mansuri, G.,andV. Rao (2004): “Community-based (and driven) development : A critical review,” The World Bank Research Observer, 19(1), 1–39.
Mart´ınez, S. (2004): “Pensions, Poverty and Household Investments in Bolivia,” Mimeograph, University of Berkeley, Department of Economics.
Menon, N. (2010): “Investment credit and child labour,” Applied Economics, 42(12), 1461–1479.
Mirrlees, J. A. (1999): “The Theory of Moral Hazard and Unobservable Behaviour: Part I,” Review of Economic Studies, 66(1), 3–21.
Muller, K. (2009): “Contested universalism: from Bonosol to Renta Dignidad in Bolivia,” International Journal of Social Welfare, 18(2), 163–172.
Nagin, D. S., J. B. Rebitzer, S. Sanders, and L. J. Taylor (2002): “Monitoring, Motivation, and Management: The Determinants of Opportunistic Behavior in a Field Experiment,” American Economic Review, 92(4), 850–873.
Nalbantian, H. R., andA. Schotter (1997): “Productivity under Group Incentives: An Experimental Study,” American Economic Review, 87(3), 314–41.
Newman, J., M. Pradhan, L. B. Rawlings, G. Ridder, R. Coa, andJ. L. Evia (2002): “An Impact Evaluation of Education, Health, and Water Supply Investments by the Bolivian Social Investment Fund,” World Bank Economic Review, 16(2), 241–274.
Olken, B. A. (2007): “Monitoring Corruption: Evidence from a Field Experiment in Indonesia,” Journal of Political Economy, 115(2), 200–249.
Olken, B. A.,and P. Barron (2009): “The Simple Economics of Extortion: Evidence from Trucking in Aceh,” Journal of Political Economy, 117(3), 417–452.
Paxson, C.,andN. R. Schady (2002): “The Allocation and Impact of Social Funds: Spending on School Infrastructure in Peru,” World Bank Economic Review, 16(2), 297–319.
Philip, D., J. V. Domelen, E. Ostrom, A. Ryan, W. Wakeman, A. Bebbington, S. Alkire, T. Esmail,andM. Polski (2001): “Community Driven Development,” In world bank, poverty reduction strategy paper sourcebook, Washington, D.C.
PNUD (2006): Polic´ıa Nacional y seguridad ciudadana. Programa de las Naciones Unidas para el Desarrollo, La Paz-Bolivia.
Pollak, R. A. (2005): “Bargaining Power in Marriage: Earnings, Wage Rates and Household Production,” NBER Working Papers 11239, National Bureau of Economic Research, Inc.
Quintana, J. R. (2003): Polic´ıa y Democracia en Bolivia: Una pol´ıtica institucional pendiente. Programa de Investigaci´on Estrat´egica en Bolivia, La Paz-Bolivia.
Quisumbing, A. R.,andJ. A. Maluccio (2003): “Resources at Marriage and Intrahousehold Allocation: Evidence from Bangladesh, Ethiopia, Indonesia, and South Africa,” Oxford Bulletin of Economics and Statistics, 65(3), 283–327.
Ranjan, P. (2001): “Credit constraints and the phenomenon of child labor,” Journal of Development Economics, 64(1), 81–102.
Reinikka, R.,andJ. Svensson (2004a): “Local Capture: Evidence From a Central Government Transfer Program in Uganda,” The Quarterly Journal of Economics, 119(2), 678–704.
(2004b): “The power of information : Evidence from a newspaper campaign to reduce capture,” Policy Research Working Paper Series 3239, The World Bank.
Robinson, J. A. (2008): “The Political Economy of Decentralization in Bolivia,” Discussion paper, Paper written for the World Bank Group.
Romer, N. (2008): “Bolivia: Latin America’s Experiment in Grassroots Democracy,” New Politics, XI(4), 23–39.
Rose-Ackerman, S. (2005): “The Challenge of Poor Governance and Corruption,” Revista Direitogv, Especial 1, 207–266.
Rosenbaum, P. R. (2002): “Covariance Adjustment in Randomized Experiments and Observational Stud- ies,” Statistical Science, 17(3), 286–327.
Rosenberger, W. F.,andJ. M. Lachin (2002): Randomization in Clinical Trials: Theory and Practice. John Wiley and Sons, New York.
Schreyer, P. (2000): “The Contribution of Information and Communication Technology to Output Growth: A Study of the G7 Countries,” OECD Science, Technology and Industry Working Papers 2000/2, OECD, Directorate for Science, Technology and Industry.
Sequeira, S.,andS. Djankov (2008): “On the Waterfront: An Empirical Study of Corruption in Ports,” Mimeograph, Harvard University.
Shaw, W. D. (1992): “Searching for the Opportunity Cost of an Individual’s Time,” Land Economics, 68(1), 107–115.
SPVS (2002): “Bolet´ın Estad´ıstico de Pensiones (Octubre-Diciembre) No. 23,” Superintendencia de Pen- siones, Valores y Seguros. Intendencia de Pensiones. La Paz- Bolivia.
Stiglitz, J. E. (2002): “Participation and Development: Perspectives from the Comprehensive Develop- ment Paradigm,” Review of Development Economics, 6(2), 163–82.
Stiroh, K. J. (2001): “Information Technology and the U.S. Productivity Revival: What Do the Industry Data Say?,” Staff Reports 115, Federal Reserve Bank of New York.
Summers, L. (2011): “Speech at World Bank Country Directors Retreat,” Washington D.C.
Sunden, S.,and G. Wicander (2006): Information and communication technology applied for developing countries in a rural context. Faculty of Economic Sciences, Communication and IT, Information Systems, Karlstad University.
Svensson, J. (2003): “Who Must Pay Bribes And How Much? Evidence From A Cross Section Of Firms,” The Quarterly Journal of Economics, 118(1), 207–230.
UDAPE (2009): “Dossier de Estad´ısticas, Unidad de An´alisis de Pol´ıticas Sociales y Econ´omicas, vol 19,” La Paz, Bolivia.
Van Cott, D. (2008): Radical Democracy in the Andes. Cambridge University Press.
Van Reenen, J.,andL. Chennells (2002): “The effects of technical change on skills, wages and employ- ment: A survey of the micro-econometric evidence,” in Productivity, inequality, and the digital economy: A transatlantic perspective, ed. by N. Greenan, Y. L’Horty,andJ. Mairesse. MIT Press, Cambridge, Mass. von Gersdorff, H. (1997): “The Bolivian Pension Reform: Innovative Solutions to Common Problems,”
World Bank Policy Research Working Paper 1832, The World Bank, Washington D.C.
Wanderley, F. (2007): Ejercer Ciudadan´ıa En Bolivia: Sociolog´ıa Del Estado. Informe de Desarrollo Humano (IDH) - PNUD.
Wooldridge, J. M. (2003): “Cluster-Sample Methods in Applied Econometrics,” American Economic Review, 93(2), 133–138.
World Bank (2000): “From Patronage to a Professional State: Bolivia Institutional and Governance Review,” .
World Bank, (2004): “Making Services Work for Poor People,” World Development Report, Washington, DC: World Bank.
World Bank (2011): “Capturing Technology for Development: An Evaluation of World Bank Group Activ- ities in Information and Communication Technologies,” The Independent Evaluation Group, Washington DC.