Survey Experiments in Political Science Matthew S. Winters
Assistant Professor
University of Illinois at Urbana-Champaign [email protected]
Objectives
The objective of this class is to expose students to survey experimental research in political science and international relations through concrete examples complemented by methodological readings. The course will explain the general methodological logic behind experiments, discuss the strengths and weaknesses of survey experiments and discuss some of the frontiers in analyzing experimental data. Students will develop an improved capacity to read research that uses experimental methodologies and to think about designing their own experimental and non-experimental research. At the end of the course, students will have developed, in consultation with the instructor, their own research design using survey experimental methods for a substantive question of interest to them.
Justification
Experimental analysis has long been recognized as the best methodology for causal inference.
Embedding text- and image-based experiments into surveys is a straightforward way of collecting data that can be used for causal inference with regard to a population of interest. This course will provide students with examples from leading journals and scholars in political science, providing both a broad view of the field and specific examples of a particular, widely applicable methodology.
Structure
The two-week course will include two seminar meetings and one laboratory session each week. During the seminar meetings, we will discuss as a class relevant readings from political science and
international relations. During the laboratory session, students will discuss their individual research designs with the instructor and each other.
Evaluation
Participation in the seminar (30 percent); attendance at laboratory sessions (20 percent); final research design (50 percent).
Program
I. Experimental Logic and the Growth of Survey Experiments II. Survey Experiments across Different Fields of Political Science III. Lab I
IV. Survey Experiments, Field Experiments and Observational Studies V. The Analysis of Survey Experimental Data
Course Outline with Readings
1. Experimental Logic and the Growth of Survey Experiments
In this first session, we will review experimental logic in general with comparisons to other research methods commonly used in political science and international relations. We will consider the survey experiment as one type of experimental methodology. The first reading provides basic notation from the potential outcomes model that underlies the design and analysis of experiments. The second and third readings review the use of survey experiments and some issues in interpreting the results from survey experiments. The final reading provides an applied example with which we will anchor our conversation about methodological issues.
Required
Gerber, Alan S. and Donald P. Green. 2012. Field Experiments: Design, Analysis and Interpretation. New York: W.W. Norton, ch. 2 (“Causal Inference and Experimentation”).
Mutz, Diana. 2011. Population-Based Survey Experiments. Princeton: Princeton University Press, ch. 1 (“Population-Based Survey Experiments: A Hybrid Methodology for the Social Sciences”).
Gaines, Brian J., James H. Kuklinski and Paul J. Quirk. 2007. “The Logic of the Survey Experiment Reexamined,” Political Analysis 15: 1-20.
Kuklinski, James H., Paul M. Sniderman, Kathleen Knight, Thomas Piazza, Philip E. Tetlock, Gordon R. Lawrence, and Barbara Mellers. 1997. “Racial Prejudice and Attitudes toward Affirmative Action,” American Journal of Political Science 41(2): 402-19.
For Additional Reference
Rebecca B. Morton and Kenneth Williams. 2010. Experimental Political Science and the Study of Causality: From Nature to the Lab. New York: Cambridge University Press.
Druckman, James N., Donald P. Green, James H. Kuklinski, and Arthur Lupia. 2006. “The Growth and Development of Experimental Research in Political Science,” American Political Science Review 100: 627-635.
2. Survey Experiments Across Different Fields of Political Science
In this class, we will discuss five different survey experiments. We will discuss the way in which each experiment was designed and analyzed, and we will think about potential new experiments that could build on each of these. The examples are drawn from political science and international relations.
Required
Anduiza, Eva, Aina Gallego and Jordi Muñoz. 2014. “Turning a Blind Eye: Experimental Evidence of Partisan Bias in Attitudes toward Corruption,” Comparative Political Studies.
Lupu, Noam. 2013. “Party Brands and Partisanship: Theory with Evidence from a Survey Experiment in Argentina,” American Journal of Political Science 57(1): 49-64.
Press, Daryl G., Scott D. Sagan and Benhamin A. Valentino. 2013. “Atomic Aversion: Experimental Evidence on Taboos, Traditions and the Non-Use of Nuclear Weapons,” American Political Science Review 107(1): 188-206.
Pepinsky, Thomas B. , R. William Liddle and Saiful Mujani. 2012. “Testing Islam’s Political Advantage: Evidence from Indonesia,” American Journal of Political Science 56(3): 584-600.
For Additional Reference
Hainmueller, Jens and Michael J. Hiscox. 2010. “Attitudes toward Highly Skilled and Low-skilled Immigration: Evidence from a Survey Experiment,” American Political Science Review 104(1): 61-84.
Tomz, Michael and Robert P. Van Houweling. 2009. “The Electoral Implications of Candidate Ambiguity,” American Political Science Review 103(1): 83-98.
Prior, Markus and Arthur Lupia. 2008. “Money, Time, and Political Knowledge: Distinguishing Quick Recall and Political Learning Skills,” American Journal of Political Science 52(1): 169-83.
Druckman, James N. 2004. “Political Preference Formation: Competition, Deliberation, and the (Ir)relevance of Framing Effects,” American Political Science Review 98(4): 671-86.
Canache, Damarys, Jeffrey J. Mondak and Ernesto Cabrera. 2000. “Voters and the Personal Vote: A Counterfactual Simulation,” Political Research Quarterly 53(3): 663-76.
3. Survey Experiments, Field Experiments and Observational Studies
In this class, we discuss several potential critiques of survey experimental research with an eye to comparing survey experiments to field experiments and observational studies. The first reading is concerned exactly with the conclusions about the “real world” that can be drawn from survey
experiments. The second reading defends the typical population of laboratory experiments (university students), something that is relevant for limited-sample survey experiments. The third reading is a field experiment that is very similar to a survey experiment but with an observable behavioral outcome. The fourth reading is an example of a piece of research that uses a survey experiment to explore a
hypothesis that originates in observational research.
Required
Druckman, James N. and Cindy D. Kam. 2011. “Students as Experimental Participants: A Defense of the ‘Narrow Data Base,’” in James N. Druckman, Donald P. Green, James H. Kuklinski and Arthur Lupia, eds. Cambridge Handbook of Experimental Political Science, pp. 41-57.
Bertrand, Marianne and Sendhil Mullanathan. 2004. “Are Emily and Greg More Employable than Lakisha and Jamal? A Field Experiment on Labor Market Discrimination,” American Economic Review 94(4): 991-1013.
Baldwin, Kate. Forthcoming. “Why Vote with the Chief? Political Connections and Public Goods Provision in Zambia,” American Journal of Political Science.
For Additional Reference
List, John A. 2011. “Why Economists Should Conduct Field Experiments and 14 Tips for Pulling One Off,” Journal of Economic Perspectives 25.
4. The Analysis of Survey Experimental Data
In our final seminar session, we discuss the statistical analysis of (survey) experimental data. The first reading describes the importance of pre-registering experimental (and non-experimental) studies; the list of core fields in the registry is useful for thinking about the design of any research project. The second reading describes design and analysis issues in survey experiments in particular. The third reading talks about improving our understanding of mediating effects and causal mechanisms. The fourth reading describes using blocking to get more precise estimates of treatment effects.
Required
EGAP VIII Steering Group. 2012. “Draft Proposal for a Pilot Registry for Political Science (PREPS),” available at http://e-gap.org/wp/wp-content/uploads/20121025-EGAP-Proposal.pdf.
Horiuchi, Yusaku, Kosuke Imai and Naoko Taniguchi. 2007. “Designing and Analyzing Randomized Experiments: Application to a Japanese Election Survey Experiment,” American Journal of Political Science 51(3): 669-87.
Imai, Kosuke, Luke Keele, Dustin Tingley and Teppei Yamamoto. 2011. “Unpacking the Black Box of Causality: Learning about Causal Mechanisms from Experimental and Observational Studies,” American Political Science Review 105(4): 765-89.
Moore, Ryan T. 2012. “Multivariate Continuous Blocking to Improve Political Science Experiments,” Political Analysis 20: 460-79.
For Additional Reference
Holbrook, Allyson L. and Jon A. Krosnick. 2010. “Social Desirability Bias in Voter Turnout Reports: Tests Using the Item Count Technique,” Public Opinion Quarterly 74(1): 37-67.
Malhotra, Neil and Jon Krosnick. 2007. “The Effect of Survey Mode and Sampling on Inferences about Political Attitudes and Behavior: Comparing the 2000 and 2004 ANES to Internet Surveys with
