Hostname: page-component-7bb8b95d7b-2h6rp Total loading time: 0 Render date: 2024-09-13T22:25:35.915Z Has data issue: false hasContentIssue false
  • English
  • Français

Relaxing the No Liars Assumption in List Experiment Analyses

Published online by Cambridge University Press:  10 May 2019

Yimeng Li Open the ORCID record for Yimeng Li [Opens in a new window]
Yimeng Li*
Affiliation:
Graduate Student, Division of the Humanities and Social Sciences, California Institute of Technology, Pasadena, CA 91106, USA. Email: yimeng.li@caltech.edu
*
*Email: yimeng.li@caltech.edu
Get access Rights & Permissions [Opens in a new window]

Abstract

The analysis of list experiments depends on two assumptions, known as “no design effect” and “no liars”. The no liars assumption is strong and may fail in many list experiments. I relax the no liars assumption in this paper, and develop a method to provide bounds for the prevalence of sensitive behaviors or attitudes under a weaker behavioral assumption about respondents’ truthfulness toward the sensitive item. I apply the method to a list experiment on the anti-immigration attitudes of California residents and on a broad set of existing list experiment datasets. The prevalence of different items and the correlation structure among items on the list jointly determine the width of the bound estimates. In particular, the bounds tend to be narrower when the list consists of items of the same category, such as multiple groups or organizations, different corporate activities, and various considerations for politician decision-making. My paper illustrates when the full power of the no liars assumption is most needed to pin down the prevalence of the sensitive behavior or attitude, and facilitates estimation of the prevalence robust to violations of the no liars assumption for many list experiment applications.

Keywords

list experimentspartial identificationMonte Carlo simulation
Type
Articles
Information
Political Analysis , Volume 27 , Issue 4 , October 2019 , pp. 540 - 555
Copyright
Copyright © The Author(s) 2019. Published by Cambridge University Press on behalf of the Society for Political Methodology. 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

Author’s note: I thank Ines Levin for collecting and sharing with me the data I use in Section 3.1. Previous versions of this research were presented as a poster at the 34th Annual Meeting of the Society for Political Methodology (Polmeth 2017), and in a paper session at the 2018 Annual Meeting of the Midwest Political Science Association (MPSA 2018). I thank R. Michael Alvarez, Jonathan N. Katz, Seo-young Silvia Kim, Ines Levin, Lucas Núñez, Alejandro Robinson-Cortés, Robert Sherman, Matthew Shum, and participants at my poster and paper presentations at Polmeth 2017 and MPSA 2018 for discussions and comments. All errors are my own. Replication data are available in Li (2018) and an R function to implement the proposed bounds is available at https://github.com/lymolympic/list_relaxed_liars.

Contributing Editor: Jeff Gill

References

Ahlquist, J. S. 2018. “List Experiment Design, Non-Strategic Respondent Error, and Item Count Technique Estimators.” Political Analysis 26(1):34–53.10.1017/pan.2017.31Google Scholar
Ahlquist, J. S., Mayer, K. R., and Jackman, S.. 2014. “Alien Abduction and Voter Impersonation in the 2012 U.S. General Election: Evidence from a Survey List Experiment.” Election Law Journal: Rules, Politics, and Policy 13(4):460–475.10.1089/elj.2013.0231Google Scholar
Alvarez, R. M., Atkeson, L. R., Levin, I., and Li, Y.. 2019. “Paying Attention to Inattentive Survey Respondents.” Political Analysis 27(2):145–162.10.1017/pan.2018.57Google Scholar
Aronow, P. M., Coppock, A., Crawford, F. W., and Green, D. P.. 2015. “Combining List Experiment and Direct Question Estimates of Sensitive Behavior Prevalence.” Journal of Survey Statistics and Methodology 3(1):43–66.10.1093/jssam/smu023Google Scholar
Berinsky, A. J., Margolis, M. F., and Sances, M. W.. 2014. “Separating the Shirkers from the Workers? Making Sure Respondents Pay Attention on Self-Administered Surveys.” American Journal of Political Science 58(3):739–753.10.1111/ajps.12081Google Scholar
Blair, G., and Imai, K.. 2012. “Statistical Analysis of List Experiments.” Political Analysis 20(1):47–77.Google Scholar
Blair, G., Imai, K., and Lyall, J.. 2014. “Comparing and Combining List and Endorsement Experiments: Evidence from Afghanistan.” American Journal of Political Science 58(4):1043–1063.10.1111/ajps.12086Google Scholar
Coffman, K. B., Coffman, L. C., and Marzilli Ericson, K. M.. 2017. “The Size of the LGBT Population and the Magnitude of Antigay Sentiment Are Substantially Underestimated.” Management Science 63(10):3168–3186.Google Scholar
Coppock, A. 2017. “Did Shy Trump Supporters Bias the 2016 Polls? Evidence from a Nationally-Representative List Experiment.” Statistics, Politics and Policy 8(1):29–40.Google Scholar
Corstange, D. 2009. “Sensitive Questions, Truthful Answers? Modeling the List Experiment with LISTIT.” Political Analysis 17(1):45–63.Google Scholar
Eady, G. 2017. “The Statistical Analysis of Misreporting on Sensitive Survey Questions.” Political Analysis 25(2):241–259.10.1017/pan.2017.8Google Scholar
Frye, T., Gehlbach, S., Marquardt, K. L., and Reuter, O. J.. 2016. “Is Putin’s Popularity Real? Post-Soviet Affairs 33(1):1–15.Google Scholar
Glynn, A. N.2010. “What Can We Learn with Statistical Truth Serum? Design and Analysis of the List Experiment.” Unpublished manuscript.Google Scholar
Glynn, A. N. 2013. “What Can We Learn with Statistical Truth Serum? Design and Analysis of the List Experiment.” Public Opinion Quarterly 77(S1):159–172.Google Scholar
González-Ocantos, E. et al. . 2012. “Vote Buying and Social Desirability Bias: Experimental Evidence from Nicaragua.” American Journal of Political Science 56(1):202–217.Google Scholar
González-Ocantos, E., Kiewiet de Jonge, C., and Nickerson, D. W.. 2015. “Legitimacy Buying: The Dynamics of Clientelism in the Face of Legitimacy Challenges.” Comparative Political Studies 48(9):1127–1158.10.1177/0010414015574882Google Scholar
Heerwig, J. A., and McCabe, B. J.. 2009. “Education and Social Desirability Bias: The Case of a Black Presidential Candidate.” Social Science Quarterly 90(3):674–686.Google Scholar
Holbrook, A. L., and Krosnick, J. A.. 2010. “Social Desirability Bias in Voter Turnout Reports: Tests Using the Item Count Technique.” Public Opinion Quarterly 74(1):37–67.Google Scholar
Imai, K. 2011. “Multivariate Regression Analysis for the Item Count Technique.” Journal of the American Statistical Association 106(494):407–416.Google Scholar
Imai, K., Park, B., and Greene, K. F.. 2015. “Using the Predicted Responses from List Experiments as Explanatory Variables in Regression Models.” Political Analysis 23(2):180–196.10.1093/pan/mpu017Google Scholar
Imbens, G. W., and Manski, C. F.. 2004. “Confidence Intervals for Partially Identified Parameters.” Econometrica 72(6):1845–1857.10.1111/j.1468-0262.2004.00555.xGoogle Scholar
Kane, J. G., Craig, S. C., and Wald, K. D.. 2004. “Religion and Presidential Politics in Florida: A List Experiment.” Social Science Quarterly 85(2):281–293.10.1111/j.0038-4941.2004.08502004.xGoogle Scholar
Kiewiet de Jonge, C. P. 2015. “Who Lies About Electoral Gifts? Experimental Evidence from Latin America.” Public Opinion Quarterly 79(3):710–739.Google Scholar
Kiewiet de Jonge, C. P., and Nickerson, D. W.. 2014. “Artificial Inflation or Deflation? Assessing the Item Count Technique in Comparative Surveys.” Political Behavior 36(3):659–682.10.1007/s11109-013-9249-xGoogle Scholar
Köszegi, B. 2006. “Ego Utility, Overconfidence, and Task Choice.” Journal of the European Economic Association 4(4):673–707.Google Scholar
Kramon, E., and Weghorst, K. R.. 2012. “Measuring Sensitive Attitudes in Developing Countries: Lessons from Implementing the List Experiment.” Newsletter of the APSA Experimental Section 3(2):14–24.Google Scholar
Kuklinski, J. H., Cobb, M. D., and Gilens, M.. 1997. “Racial Attitudes and the ‘New South’.” The Journal of Politics 59(2):323–349.Google Scholar
Lax, J. R., Phillips, J. H., and Stollwerk, A. F.. 2016. “Are Survey Respondents Lying about Their Support for Same-Sex Marriage? Lessons from a List Experiment.” Public Opinion Quarterly 80(2):510–533.Google Scholar
Li, Y.2018. “Replication Data for: Relaxing the No Liars Assumption in List Experiment Analyses.” https://doi.org/10.7910/DVN/DFRSKZ, Harvard Dataverse, V1.Google Scholar
Malesky, E. J., Gueorguiev, D. D., and Jensen, N. M.. 2015. “Monopoly Money: Foreign Investment and Bribery in Vietnam, a Survey Experiment.” American Journal of Political Science 59(2):419–439.10.1111/ajps.12126Google Scholar
Meng, T., Pan, J., and Yang, P.. 2014. “Conditional Receptivity to Citizen Participation: Evidence From a Survey Experiment in China.” Comparative Political Studies 50(4):399–433.Google Scholar
Oppenheimer, D. M., Meyvis, T., and Davidenko, N.. 2009. “Instructional Manipulation Checks: Detecting Satisficing to Increase Statistical Power.” Journal of Experimental Social Psychology 45(4):867–872.Google Scholar
Redlawsk, D. P., Tolbert, C. J., and Franko, W.. 2010. “Voters, Emotions, and Race in 2008: Obama as the First Black President.” Political Research Quarterly 63(4):875–889.10.1177/1065912910373554Google Scholar
Rosenfeld, B., Imai, K., and Shapiro, J. N.. 2016. “An Empirical Validation Study of Popular Survey Methodologies for Sensitive Questions.” American Journal of Political Science 60(3):783–802.Google Scholar
Stoye, J. 2009. “More on Confidence Intervals for Partially Identified Parameters.” Econometrica 77(4):1299–1315.Google Scholar
Streb, M. J., Burrell, B., Frederick, B., and Genovese, M. A.. 2008. “Social Desirability Effects and Support for a Female American President.” Public Opinion Quarterly 72(1):76–89.Google Scholar
Supplementary material: File

Li supplementary material

Li supplementary material 1

Download Li supplementary material(File)
File 888.4 KB