Hostname: page-component-66644f4456-swwnq Total loading time: 0 Render date: 2025-02-13T09:24:49.378Z Has data issue: true hasContentIssue false
  • English
  • Français

A randomized experiment evaluating survey mode effects for video interviewing

Published online by Cambridge University Press:  01 August 2022

Kyle Endres Open the ORCID record for Kyle Endres [Opens in a new window] ,
D. Sunshine Hillygus ,
Matthew DeBell  and
Shanto Iyengar
Kyle Endres*
Affiliation:
University of Northern Iowa, Cedar Falls, IA, USA
D. Sunshine Hillygus
Affiliation:
Duke University, Durham, NC, USA
Matthew DeBell
Affiliation:
Stanford University, Stanford, CA, USA
Shanto Iyengar
Affiliation:
Stanford University, Stanford, CA, USA
*
*Corresponding author. Email: kyle.endres@gmail.com
Get access Rights & Permissions [Opens in a new window]

Abstract

Rising costs and challenges of in-person interviewing have prompted major surveys to consider moving online and conducting live web-based video interviews. In this paper, we evaluate video mode effects using a two-wave experimental design in which respondents were randomized to either an interviewer-administered video or interviewer-administered in-person survey wave after completing a self-administered online survey wave. This design permits testing of both within- and between-subject differences across survey modes. Our findings suggest that video interviewing is more comparable to in-person interviewing than online interviewing across multiple measures of satisficing, social desirability, and respondent satisfaction.

Keywords

Survey methodology
Type
Original Article
Information
Political Science Research and Methods , Volume 11 , Issue 1 , January 2023 , pp. 144 - 159
Copyright
Copyright © The Author(s), 2022. Published by Cambridge University Press on behalf of the European Political Science Association

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Abrajano, M and Alvarez, RM (2018) Answering questions about race: How racial and ethnic identities influence survey response. American Politics Research 47, 250–274.CrossRefGoogle Scholar
Anderson, AH (2008) Video-mediated interactions and surveys. In Conrad, FG and Schober, MF (eds), Envisioning the Survey Interview of the Future. Hoboken, NJ: Wiley, pp. 95–118.Google Scholar
Ansolabehere, S and Schaffner, B (2018) Taking the study of political behavior online. In Atkeson, LR and Alvarez, RM (eds). The Oxford Handbook of Polling and Survey Methods. New York, NY: Oxford University Press, pp. 76–96.Google Scholar
Atkeson, LR and Adams, AN (2018) Mixing survey modes and its implications. In Atkeson, LR and Alvarez, RM (eds). The Oxford Handbook of Polling and Survey Methods. New York, NY: Oxford University Press, pp. 53–75.Google Scholar
Atkeson, LR, Adams, AN and Alvarez, RM (2014) Nonresponse and mode effects in self-and interviewer-administered surveys. Political Analysis 22, 304–320.CrossRefGoogle Scholar
Baker, R, Blumberg, S, Brick, JM, Couper, MP, Courtright, M, Dennis, M, Dillman, D, Frankel, MR, Garland, P, Groves, R, Kennedy, C, Krosnick, J and Lavrakas, PJ (2010) AAPOR report on online panels. The American Association for Public Opinion Research.Google Scholar
Ballejos, MP, Oglesbee, S, Hettema, J and Sapien, R (2018) An equivalence study of interview platform: Does videoconference technology impact medical school acceptance rates of different groups?. Advances in Health Sciences Education 23, 601–610.CrossRefGoogle ScholarPubMed
Carmines, E and Nassar, R (2021) How social desirability bias affects immigration attitudes in a hyperpolarized political environment. Social Science Quarterly 102, 1803–1811.Google Scholar
Chang, L and Krosnick, JA (2009) National surveys via RDD telephone interviewing versus the Internet: comparing sample representativeness and response quality. Public Opinion Quarterly 73, 641–678.CrossRefGoogle Scholar
Chang, L and Krosnick, JA (2010) Comparing oral interviewing with self-administered computerized questionnaires. Public Opinion Quarterly 74, 154–167.Google Scholar
Clifford, S, Sheagley, G and Piston, S (2021) Increasing precision without altering treatment effects: Repeated measures designs in survey experiments. American Political Science Review 115, 1048–1065.CrossRefGoogle Scholar
Conrad, FG, Schober, MF, Hupp, AL, West, BT, Larsen, K, Ong, AR, Wang, T and Interviewers, Video, and Survey Data Collection (2020) Annual Meeting of the American Association for Public Opinion Research.Google Scholar
Couper, MP (2011) The future of modes of data collection. Public Opinion Quarterly 75, 889–908.CrossRefGoogle Scholar
Davis, DW (1997) Nonrandom measurement error and race of interviewer effects among African Americans. The Public Opinion Quarterly 61, 183–207.Google Scholar
Forrestal, SG, D'Angelo, AV and Vogel, LK (2015) Considerations for and lessons learned from online, synchronous focus groups. Survey Practice 8, 1–8.CrossRefGoogle Scholar
Frankel, LL and Hillygus, DS (2014) Looking beyond demographics: Panel attrition in the ANES and GSS. Political Analysis 22, 336–353.Google Scholar
Gooch, A and Vavreck, L (2019) How face-to-face interviews and cognitive skill affect item non-response: A randomized experiment assigning mode of interview. Political Science Research and Methods 7, 143–162.CrossRefGoogle Scholar
Groves, RM and Couper, MP (2012) Nonresponse in Household Interview Surveys. Hoboken, NJ: John Wiley & Sons.Google Scholar
Guay, B, Hillygus, DS and Valentino, N (2019) Opportunities and challenges of conducting large-scale political surveys online. Unpublished manuscript. Duke University.Google Scholar
Guggenheim, L, Maisel, N, Howell, D, Amsbary, M, Brader, T, DeBell, M, Good, C and Hillygus, S (2021) Live video interviewing in the ANES 2020 time series study. Presented at the 2021 Virtual Meeting of the American Association of Public Opinion Research, May 11, 2021.Google Scholar
Haan, M, Ongena, YP, Vannieuwenhuyze, JT and De Glopper, K (2017) Response behavior in a video-web survey: A mode comparison study. Journal of Survey Statistics and Methodology 5, 48–69.Google Scholar
Hanson, T (2021) The European Social Survey during COVID-19: Using video interviews and other innovations. Presented at the 2021 Virtual Meeting of the American Association of Public Opinion Research, May 11, 2021.Google Scholar
Heerwegh, D and Loosveldt, G (2008) Face-to-face versus web surveying in a high-Internet coverage population: Differences in response quality. Public Opinion Quarterly 72, 836.Google Scholar
Hillygus, DS and Snell, SA (2018) Longitudinal surveys: Issues and opportunities. In Atkeson, LR and Alvarez, RM (eds). The Oxford Handbook on Polling and Survey Methods. New York, NY: Oxford University Press, pp. 28–52.Google Scholar
Hillygus, DS, Valentino, N, Vavreck, L, Barreto, M and Layman, G (2017) Assessing the implications of a mode change. Unpublished ANES report.Google Scholar
Holbrook, A, Green, M and Krosnick, JA (2003) Telephone versus face-to-face interviewing of national probability samples with long questionnaires—Comparison of respondent satisficing and social desirability response bias. Public Opinion Quarterly 67, 79–125.Google Scholar
Homola, J, Jackson, N and Gill, J (2016) A measure of survey mode differences. Electoral Studies 44, 255–274.CrossRefGoogle Scholar
Janghorban, R, Roudsari, RL and Taghipour, A (2014) Skype interviewing: The new generation of online synchronous interview in qualitative research. International Journal of Qualitative Studies on Health and Well-Being 9, 24152.Google ScholarPubMed
Jeannis, M, Terry, T, Heman-Ackah, R and Price, M (2013) Video interviewing: An exploration of the feasibility as a mode of survey application. Survey Practice 6, 1–5.CrossRefGoogle Scholar
Klausch, T, Hox, JJ and Schouten, B (2013) Measurement effects of survey mode on the equivalence of attitudinal rating scale questions. Sociological Methods and Research 42, 227–263.CrossRefGoogle Scholar
Krosnick, JA (1991) Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology 5, 213–236.Google Scholar
Larsen, K, Hupp, A, Conrad, F, Schober, M, Ong, A, West, B and Wang, T (2021) Recruitment & participation in video interviews. Presented at the 2021 Virtual Meeting of the American Association of Public Opinion Research, May 11, 2021.Google Scholar
Lesser, V, Newton, L and Yang, D (2012) Comparing item nonresponse across different delivery modes in general population surveys. Survey Practice 5, 1–5.CrossRefGoogle Scholar
Liu, M (2018) Data collection mode. Effect on abortion questions: a comparison of face-to-face and web surveys. Gender and Women's Studies 1, 1–10.Google Scholar
Liu, M and Wang, Y (2015) Data collection mode effect on feeling thermometer questions: a comparison of face-to-face and Web surveys. Computers in Human Behavior 48, 212–218.CrossRefGoogle Scholar
Liu, M and Wang, Y (2016) Comparison of face-to-face and web surveys on the topic of homosexual rights. Journal of homosexuality 63, 838–854.CrossRefGoogle ScholarPubMed
Malhotra, N (2008) Completion time and response order effects in web surveys. Public Opinion Quarterly 72, 914–934.CrossRefGoogle Scholar
Okon, S, Schober, M, Conrad, F, Hupp, A, Ong, AR and Larsen, K (2021) Predictors of willingness to participate in live video survey interviews: A pilot study. Presented at the 2021 Virtual Meeting of the American Association of Public Opinion Research, May 11, 2021.Google Scholar
Pasadhika, S, Altenbernd, T, Ober, RR, Harvey, EM and Miller, JM (2012) Residency interview video conferencing. Ophthalmology 119, 426–426.CrossRefGoogle ScholarPubMed
Reuning, K and Plutzer, E (2020) Valid vs. invalid straightlining: The complex relationship between straightlining and data quality. Survey Research Methods 14, 439–459.Google Scholar
Revilla, MA and Saris, WE (2013) A comparison of the quality of questions in a face-to-face and a web survey. International Journal of Public Opinion Research 25, 242–253.CrossRefGoogle Scholar
Roberts, CEG, Allum, N and Eisner, L (2019) Research synthesis: Satisficing in surveys: A systematic review of the literature. Public Opinion Quarterly 83, 598–626.CrossRefGoogle Scholar
Schaeffer, NC, Dykema, J and Maynard, DW (2010) Interviewers and interviewing. In Marsden, PV and Wright, JD (eds). Handbook of Survey Research, 2nd edn. Bingley, UK: Emerald Group Publishing, pp. 437–471.Google Scholar
Schober, MF (2018) The future of face-to-face interviewing. Quality Assurance in Education 26, 290–302.CrossRefGoogle Scholar
Schober, MF, Conrad, FG, Hupp, AL, Larsen, KM, Ong, AR and West, BT (2020) Design considerations for live video survey interviews. Survey Practice 13, 1–11.Google Scholar
Sullivan, JR (2012) Skype: An appropriate method of data collection for qualitative interviews? The Hilltop Review 6, 10.Google Scholar
Sun, H (2014) Rapport and Its Impact on the Disclosure of Sensitive Information in Standardized Interviews (Doctoral dissertation). University of Maryland, College Park. UMI No. 3682810.Google Scholar
Sun, H, Conrad, FG and Kreuter, F (2020) The relationship between interviewer-respondent rapport and data quality. Journal of Survey Statistics and Methodology 9, 429–448.Google Scholar
Valentino, NA, Zhirkov, K, Hillygus, DS and Guay, B (2020) The consequences of personality biases in online panels for measuring public opinion. Public Opinion Quarterly 84, 446–468.Google Scholar
Vavreck, L (2014) The consequences of face-to-face interviews for respondents with low cognitive skills: A randomized experiment assigning in-person and self-complete survey modes. Unpublished manuscript, University of California, Los Angeles.Google Scholar
Villar, A and Fitzgerald, R (2017) Using mixed modes in survey data research: Results from six experiments. In Breen, M (ed). Values and Identities in Europe. Evidence From the European Social Survey. London: Routledge, pp. 259–292.Google Scholar
Wenz, A (2021) Do distractions during web survey completion affect data quality? Findings from a laboratory experiment. Social Science Computer Review 39, 148–161.CrossRefGoogle Scholar
West, BT, Ong, AR, Conrad, FG, Schober, MF, Larsen, KM and Hupp, AL (2021) Interviewer effects in live video and prerecorded video interviewing. Journal of Survey Statistics and Methodology 10, 317–336.CrossRefGoogle Scholar
Yeager, DS, Krosnick, JA, Chang, L, Javitz, HS, Levendusky, MS, Simpser, A and Wang, R (2011) Comparing the accuracy of RDD telephone surveys and internet surveys conducted with probability and non-probability samples. Public Opinion Quarterly 75, 709–747.Google Scholar
Supplementary material: Link

Endres et al. Dataset

Dataset

https://doi.org/10.7910/DVN/5T7ET3
Link
Supplementary material: PDF

Endres et al. supplementary material

Appendix

Download Endres et al. supplementary material(PDF)
PDF 290 KB