Hostname: page-component-cc8bf7c57-hbs24 Total loading time: 0 Render date: 2024-12-10T21:07:04.127Z Has data issue: false hasContentIssue false
  • English
  • Français

Unexpected Event during Survey Design: Promise and Pitfalls for Causal Inference

Published online by Cambridge University Press:  15 July 2019

Jordi Muñoz Open the ORCID record for Jordi Muñoz [Opens in a new window] ,
Albert Falcó-Gimeno Open the ORCID record for Albert Falcó-Gimeno [Opens in a new window]  and
Enrique Hernández Open the ORCID record for Enrique Hernández [Opens in a new window]
Jordi Muñoz*
Affiliation:
Department of Political Science, University of Barcelona, 08034, Barcelona, Spain. Email: jordi.munoz@ub.edu
Albert Falcó-Gimeno
Affiliation:
Department of Political Science, University of Barcelona, 08034, Barcelona, Spain. Email: jordi.munoz@ub.edu
Enrique Hernández
Affiliation:
Department of Political Science, Universitat Autònoma de Barcelona, 08193, Bellaterra, Spain
*
*Email: jordi.munoz@ub.edu
Get access Rights & Permissions [Opens in a new window]

Abstract

An increasing number of studies exploit the occurrence of unexpected events during the fieldwork of public opinion surveys to estimate causal effects. In this paper, we discuss the use of this identification strategy based on unforeseen and salient events that split the sample of respondents into treatment and control groups: the Unexpected Event during Survey Design. In particular, we focus on the assumptions under which unexpected events can be exploited to estimate causal effects and we discuss potential threats to identification, paying especial attention to the observable and testable implications of these assumptions. We propose a series of best practices in the form of various estimation strategies and robustness checks that can be used to lend credibility to the causal estimates. Drawing on data from the European Social Survey, we illustrate the discussion of this method with an original study of the impact of the Charlie Hebdo terrorist attacks (Paris, 01/07/2015) on French citizens’ satisfaction with their national government.

Keywords

natural experimentscausal inferencesurvey design
Type
Articles
Information
Political Analysis , Volume 28 , Issue 2 , April 2020 , pp. 186 - 206
Copyright
Copyright © The Author(s) 2019. Published by Cambridge University Press on behalf of the Society for Political Methodology.

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

Authors’ note: This research has received financial support from the Spanish Ministry of Science, Innovation, and Universities through the research grant CSO2017-89085-P. We are grateful to Lucas Leeman, Silja Häusermann, Macarena Ares, Guillem Rico, Salomo Hirvonen, the editor, and anonymous reviewers of Political Analysis and participants at the University of Zurich IPZ publication seminar, the Autonomous University of Barcelona DEC Seminar, and the 2018 EPSA conference for helpful comments and suggestions. We are especially grateful to Erik Gahner Larsen for his detailed and insightful comments and for generously sharing with us his collection of UESD references. Enrique Hernández also thanks the University of Zurich Political Science Department for hospitality during the spring of 2018. Replication materials are available at the Political Analysis Dataverse: doi.org/10.7910/DVN/RDIIVL (Muñoz, Falcó-Gimeno, and Hernández 2019).

Contributing Editor: Jeff Gill

References

Ares, M., and Hernández, E.. 2017. “The Corrosive Effect of Corruption on Trust in Politicians: Evidence From a Natural Experiment.” Research and Politics 4(2):1–8.Google Scholar
Balcells, L., and Torrats-Espinosa, G.. 2018. “Using a Natural Experiment to Estimate the Electoral Consequences of Terrorist Attacks.” Proceedings of the National Academy of Sciences 115(42):10624–10629.Google Scholar
Barabas, J., and Jerit, J.. 2010. “Are Survey Experiments Externally Valid?” American Political Science Review 104(2):226–242.Google Scholar
Berinsky, A. J. 2017. “Measuring Public Opinion with Surveys.” Annual Review of Political Science 20:309–329.Google Scholar
Boomgaarden, H. G., and de Vreese, C. H.. 2007. “Dramatic Real-world Events and Public Opinion Dynamics: Media Coverage and its Impact on Public Reactions to an Assassination.” International Journal of Public Opinion Research 19(3):354–366.Google Scholar
Boydstun, A. E., Feezell, J. T., and Glazier, R. A.. 2018. “In the Wake of a Terrorist Attack, do Americans’ Attitudes Toward Muslims Decline?” Research and Politics 5(4):1–7.Google Scholar
Branton, R., Martinez-Ebers, V., Carey, T. E., and Matsubayashi, T.. 2015. “Social Protest and Policy Attitudes: The Case of the 2006 Immigrant Rallies.” American Journal of Political Science 59(2):390–402.Google Scholar
Brehm, J. 1993. The Phantom Respondents: Opinion Surveys and Political Representation. University of Michigan Press.Google Scholar
Coupe, T. 2017. “The Impact of Terrorism on Expectations, Trust and Happiness – the Case of the November 13 Attacks in Paris, France.” Applied Economics Letters 24(15):1084–1087.Google Scholar
Csikszentmihalyi, M., and Hunter, J.. 2003. “Happiness in Everyday Life: The Uses of Experience Sampling.” Journal of Happiness Studies 4(2):185–199.Google Scholar
De Vries, C. E. 2018. Euroscepticism and the Future of European Integration. Oxford University Press.Google Scholar
Dinesen, P. T., and Jæger, M. M.. 2013. “The Effect of Terror on Institutional Trust: New Evidence From the 3/11 Madrid Terrorist Attack.” Political Psychology 34(6):917–926.Google Scholar
Eggers, A. C., Freier, R., Grembi, V., and Nannicini, T.. 2018. “Regression Discontinuity Designs Based on Population Thresholds: Pitfalls and Solutions.” American Journal of Political Science 62(1):210–229.Google Scholar
ESS Round 7: European Social Survey Round 7 Data. Data file edition 2.1. NSD - Norwegian Centre for Research Data, Norway – Data Archive and distributor of ESS data for ESS ERIC.N.d.Google Scholar
Finseraas, H., and Listhaug, O.. 2013. “It Can Happen Here: the Impact of the Mumbai Terror Attacks on Public Opinion in Western Europe.” Public Choice 156(1–2):213–228.Google Scholar
Flores, R. D. 2018. “Can Elites Shape Public Attitudes Toward Immigrants? Evidence from the 2016 US Presidential Election.” Social Forces 96(4):1649–1690.Google Scholar
Gaines, B. J., Kuklinski, J. H., and Quirk, P. J.. 2006. “The Logic of the Survey Experiment Reexamined.” Political Analysis 15(1):1–20.Google Scholar
Geys, B., and Qari, S.. 2017. “Will You Still Trust me Tomorrow? The Causal Effect of Terrorism on Social Trust.” Public Choice 96(3–4):289–305.Google Scholar
Hainmueller, J. 2012. “Entropy Balancing for Causal Effects: A Multivariate Reweighting Method to Produce Balanced Samples in Observational Studies.” Political Analysis 20(1):25–46.Google Scholar
Hetherington, M. J., and Nelson, M.. 2003. “Anatomy of a Rally Effect: George W. Bush and the War on Terrorism.” PS: Political Science and Politics 36(1):37–42.Google Scholar
Hofstetter, C. R. 1969. “Political Disengagement and the Death of Martin Luther King.” The Public Opinion Quarterly 33(2):174–179.Google Scholar
Imbens, G. W., and Lemieux, T.. 2008. “Regression Discontinuity Designs: A Guide to Practice.” Journal of Econometrics 142(2):615–635.Google Scholar
Jakobsson, N., and Blom, S.. 2014. “Did the 2011 Terror Attacks in Norway Change Citizens’ Attitudes Toward Immigrants?” International Journal of Public Opinion Research 26(4):475–486.Google Scholar
Jensen, C., and Naumann, E.. 2016. “Increasing Pressures and Support for Public Healthcare in Europe.” Health Policy 120(6):698–705.Google Scholar
Johnston, R., and Brady, H. E.. 2002. “The Rolling Cross-Section Design.” Electoral Studies 21(2):283–295.Google Scholar
Keele, L., and Titiunik, R.. 2016. “Natural Experiments Based on Geography.” Political Science Research and Methods 4(1):65–95.Google Scholar
Kim, J. W., and Kim, E.. 2019. “Identifying the Effect of Political Rumor Diffusion Using Variations in Survey Timing.” Quarterly Journal of Political Science, forthcoming.Google Scholar
Kocher, M. A., and Monteiro, N. P.. 2016. “Lines of Demarcation: Causation, Design-Based Inference, and Historical Research.” Perspectives on Politics 14(4):952–975.Google Scholar
Larsen, E. G. 2018. “Welfare Retrenchments and Government Support: Evidence from a Natural Experiment.” European Sociological Review 34(1):40–51.Google Scholar
Legewie, J. 2013. “Terrorist Events and Attitudes toward Immigrants: A Natural Experiment.” American Journal of Sociology 118(5):1199–1245.Google Scholar
Lever, H. 1969. “The Johannesburg Station Explosion and Ethnic Attitudes.” The Public Opinion Quarterly 33(2):180–189.Google Scholar
MacKinlay, A. C. 1997. “Event Studies in Economics and Finance.” Journal of Economic Literature 35(1):13–39.Google Scholar
Metcalfe, R., Powdthavee, N., and Dolan, P.. 2011. “Destruction and Distress: Using a Quasi-Experiment to Show the Effects of the September 11 Attacks on Mental Well-Being in the United Kingdom.” The Economic Journal 121(550):F81–F103.Google Scholar
Minkus, L., Deutschmann, E., and Delhey, J.. 2018. “A Trump Effect on the EU’s Popularity? The US Presidential Election as a Natural Experiment.” Perspectives on Politics 1–18, forthcoming.Google Scholar
Montgomery, J. M., Nyhan, B., and Torres, M.. 2018. “How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It.” American Journal of Political Science 62(3):760–775.Google Scholar
Motta, M. 2018. “The Polarizing Effect of the March for Science on Attitudes toward Scientists.” PS: Political Science and Politics 51(4):782–788.Google Scholar
Muñoz, J., Falcó-Gimeno, A., and Hernández, E.. 2019. “Replication Data for: Unexpected Event during Survey Design: Promise and Pitfalls for Causal Inference.” doi.org/10.7910/DVN/RDIIVL, Harvard Dataverse, V1.Google Scholar
Perrin, A. J., and Smolek, S. J.. 2009. “Who Trusts? Race, Gender, and the September 11 Rally Effect Among Young Adults.” Social Science Research 38(1):134–145.Google Scholar
Pierce, L., Rogers, T., and Snyder, J. A.. 2016. “Losing Hurts: The Happiness Impact of Partisan Electoral Loss.” Journal of Experimental Political Science 3(1):44–59.Google Scholar
Pollock, W., Barabas, J., Jerit, J., Schoonvelde, M., Banducci, S., and Stevens, D.. 2015. “Studying Media Events in the European Social Surveys Across Research Designs, Countries, Time, Issues, and Outcomes.” European Political Science 14(4):394–421.Google Scholar
Silber Mohamed, H. 2013. “Can Protests make Latinos “American” Identity, Immigration Politics, and the 2006 Marches.” American Politics Research 41(2):298–327.Google Scholar
Silva, B. C. 2018. “The Non-Impact of the 2015 Paris Terrorist Attacks on Political Attitudes.” Personality and Social Psychology Bulletin 44(6):838–850.Google Scholar
Solaz, H., De Vries, C. E., and de Geus, R. A.. 2018. “In-Group Loyalty and the Punishment of Corruption.” Comparative Political Studies OnlineFirst:1–31.Google Scholar
St. Clair, T., Cook, T. D., and Hallberg, K.. 2014. “Examining the Internal Validity and Statistical Precision of the Comparative Interrupted Time Series Design by Comparison With a Randomized Experiment.” American Journal of Evaluation 35(3):311–327.Google Scholar
Stokes, S. C. 2014. “A Defense of Observational Research.” In Field Experiments and Their Critics: Essays on the Uses and Abuses of experimentation in the Social Sciences, 33–57. Yale University Press: Dawn Teele, New Haven.Google Scholar
Stoop, I. A. L. 2004. “Surveying Nonrespondents.” Field Methods 16(1):23–54.Google Scholar
Tiokhin, L., and Hruschka, D.. 2017. “No Evidence That an Ebola Outbreak Influenced Voting Preferences in the 2014 Elections After Controlling for Time-Series Autocorrelation: A Commentary on Beall, Hofer, and Schaller (2016).” Psychological Science 28(9):1358–1360.Google Scholar
Zepeda-Millán, C., and Wallace, S. J.. 2013. “Racialization in Times of Contention: How Social Movements Influence Latino Racial Identity.” Politics, Groups, and Identities 1(4):510–527.Google Scholar
Supplementary material: Link

Muñoz et al. Dataset

Dataset

https://doi.org/10.7910/DVN/RDIIVL
Link
Supplementary material: File

Muñoz et al. supplementary material

Muñoz et al. supplementary material 1

Download Muñoz et al. supplementary material(File)
File 570.1 KB