Hostname: page-component-78c5997874-fbnjt Total loading time: 0 Render date: 2024-11-17T17:43:59.752Z Has data issue: false hasContentIssue false
  • English
  • Français

Measuring Emotional Response: Comparing Alternative Approaches to Measurement*

Published online by Cambridge University Press:  25 November 2015

George E. Marcus ,
W. Russell Neuman  and
Michael B. MacKuen
Get access Rights & Permissions [Opens in a new window]

Abstract

Scholarly interest in the role of emotion in accounting for how people react to political figures, events, and messages has escalated over the past two plus decades in political science and psychology. However, research on the validity of the measurement of subjective self-report of emotional responses is rather limited. We introduce here a new measurement approach, a “slider” format and compare it with the long used “radio button” item format. We assess the reliability and validity of these two approaches to the measurement of affect. The study examines self-report measures of emotion to three generated news stories about terrorist threats. We report that both measurement formats are able to extract the expected threefold affect structure from a ten affect word battery. The slider format is, however, modestly more reliable, and more efficient in time to complete, has the ability to limit missing data, and generates continuous data that is less truncated than data derived from the radio button format. Finally, we report on three tests of construct validity. Both approaches exhibit equivalent results on two of those tests. However, the radio button format does poorly on one test of construct validity, that on the anticipated relationship between anxiety and interest in novel information. We present an assessment of two methods for measuring emotional reactions to stimuli such as political issues, political figures, or events. Both methods are suitable for use in online surveys or computer-driven experiments. The traditional method utilizes labeled “radio buttons” that enable a participant in a study to select by clicking on one of an array of typically five response options, ranging from lower to higher of some identified affect term (e.g., how angry one might feel). Second, the slider method offers a participant the ability to move an “arrow” up or down to indicate how much (up) or little (down) they feel. The goal of both measures is to ascertain the level of a targeted emotion, i.e., how little or how much, say anger. The slider method has been specifically developed to be used with participants using a computer. The slider approach falls within the category of visual analog scales. This method for measuring affective responses to stimuli of whatever sort has not hitherto been examined to determine its reliability and validity. The literature on the reliability and validity of these measurement strategies is thin and we found no studies including an explicit comparison.

Type
Original Articles
Information
Political Science Research and Methods , Volume 5 , Issue 4 , October 2017 , pp. 733 - 754
Copyright
© The European Political Science Association 2015 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

*

George E. Marcus, Professor of Political Science, Department of Political Science, Williams College, Williamstown, MA 01267 (gmarcus@williams.edu). W. Russell Neuman, Professor of Communications, New York University, MAGNET, 2 MetroTech Center, Suite 864, Brooklyn NY, 11201. Michael B. MacKuen, Professor of Political Science, Department of Political Science, University of North Carolina Chapel Hill, Chapel Hill NC 27599-3265. To view supplementary material for this article, please visit http://dx.doi.org/10.1017/psrm.2015.65

References

Abelson, Robert P., Kinder, Donald R., Peters, Mark D., and Fiske, Susan T.. 1982. ‘Affective and Semantic Components in Political Personal Perception’. Journal of Personality and Social Psychology 42:619–630.CrossRefGoogle Scholar
Allport, Gordon W. 1954. The Nature of Prejudice. Cambridge, MA: Addison-Wesley.Google Scholar
Alwin, Duane F. 2010. ‘How Good is Survey Measurement? Assessing the Reliability and Validity of Survey Measures’. In Peter V. Wright Marsden and James D. Bingley (eds), Handbook of Survey Research, 405–426. UK: Emerald Group.Google Scholar
Amodio, David M., Jost, John T., Master, Sarah L., and Yee, Cindy M. 2007. ‘Neurocognitive Correlates of Liberalism and Conservatism’. Nature Neuroscience 4:1246–1247.CrossRefGoogle Scholar
Ansolabehere, Stephen D., and Iyengar, Shanto. 1995. Going Negative: How Political Advertisements Shrink and Polarize the Electorate. New York: The Free Press.Google Scholar
Bollen, Kenneth A., and Barb, Kenney. 1981. ‘Pearson’s R and Coarsely Categorized Measures’. American Sociological Review 46:232–239.CrossRefGoogle Scholar
Borsboom, Denny, Mellenbergh, Gideon J., and van Heerden, Jaap. 2003. ‘The Theoretical Status of Latent Variables’. Psychological Review 110:203–219.CrossRefGoogle ScholarPubMed
Brader, Ted. 2006. Campaigning for Hearts and Minds: How Emotional Appeals in Political Ads Work. Chicago, IL: University of Chicago Press.Google Scholar
Brader, Ted, Groenendyk, Eric W., and Valentino, Nicholas. 2010. ‘Fight or Flight? When Political Threats Arouse Public Anger and Fear’. Annual Meeting of the Midwest Political Science Association, April 22–25, 2010, Chicago, IL.Google Scholar
Brader, Ted, and Marcus, George E.. 2013. ‘Emotion and Political Psychology’. In Leonie Huddy, Jack S. Levy and David O. Sears (eds), Oxford Handbook of Political Psychology, 165–204. New York: Oxford University Press.Google Scholar
Cacioppo, John T., and Petty, Richard W. 1982. ‘The Need for Cognition’. Journal of Personality and Social Psychology 42:116–131.CrossRefGoogle Scholar
Cacioppo, John T., Gardner, Wendi L., and Berntson, Gary G.. 1999. ‘The Affect System has Parallel and Integrative Processing Components: Form Follows Function’. Journal of Personality and Social Psychology 76:839–855.CrossRefGoogle Scholar
Campbell, Angus, Converse, Philip E., Miller, Warren E., and Stokes, Donald E.. 1960. The American Voter. New York: John Wiley and Sons.Google Scholar
Campbell, Angus, Converse, Philip E., Miller, Warren E., and Stokes, Donald E.. 1966. Elections and the Political Order. New York: John Wiley and Sons, Inc. Google Scholar
Campbell, Donald T., and Fiske, Donald W.. 1959. ‘Convergent and Discriminant Validation by the Multitrait-Multimethod Matrix’. Psychological Bulletin 56:81–105.CrossRefGoogle ScholarPubMed
Chaiken, Shelly, and Trope, Yaacov (eds). 1999. Dual Process Models in Social Psychology. New York: Guilford Press.Google Scholar
Cornelius, Randolf R. 1996. The Science of Emotion: Research and Tradition in the Psychology of Emotions. Upper Saddle River, NJ: Prentice Hall.Google Scholar
Couper, Mick P., Tourangeau, Roger, Conrad, Frederick G., and Singer, Eleanor. 2006. ‘Evaluating the Effectiveness of Visual Analog Scales: A Web Experiment’. Social Science Computer Review 24:227–245.CrossRefGoogle Scholar
Feldt, Leonard S. 1969. ‘A Test of the Hypothesis That Cronbach’s Alpha or Kuder-Richardson Coefficient Twenty is the Same for Two Tests’. Psychometrika 34:363–373.CrossRefGoogle Scholar
Feldt, Leonard S., Woodruff, David J., and Salih, Fathi A.. 1987. ‘Statistical Inference for Coefficient Alpha’. Applied Psychological Measurement 11:93–101.CrossRefGoogle Scholar
Fiorina, Morris P., and Abrams, Samuel J.. 2008. ‘Political Polarization in the American Public’. Annual Review of Political Science 11:563–588.CrossRefGoogle Scholar
Fiske, Susan T., and Taylor, Shelley E.. 1991. Social Cognition. New York: McGraw-Hill.Google Scholar
Gadarian, Shana K., and Albertson, Bethany. 2014. ‘Anxiety, Immigration, and the Search for Information’. Political Psychology 35:133–164.CrossRefGoogle Scholar
Haidt, Jonathan. 2001. ‘The Emotional Dog and its Rational Tail: A Social Intuitionist Approach to Moral Judgment’. Psychological Review 108:814–834.CrossRefGoogle ScholarPubMed
Hayduk, Leslie A., and Littvay, Levente. 2012. ‘Should Researchers Use Single Indicators, Best Indicators, or Multiple Indicators in Structural Equation Models?’. BMC Medical Research Methodology 12:1–17.CrossRefGoogle ScholarPubMed
Izard, Carroll E. 1971. Patterns of Emotions: A New Analysis of Anxiety and Depression. New York: Academic Press.Google Scholar
Izard, Carroll. E. 1977. Human Emotions. New York: Plenum Press.CrossRefGoogle Scholar
Jost, John T., and Amodio, David M.. 2012. ‘Political Ideology as Motivated Social Cognition: Behavioral and Neuroscientific Evidence’. Motivation and Emotion 36:55–64.CrossRefGoogle Scholar
Jost, John T., Glaser, Jack, Kruglanski, Arie W., and Sulloway, Frank J.. 2003. ‘Political Conservatism as Motivated Social Cognition’. Psychological Bulletin 129:339–375.CrossRefGoogle ScholarPubMed
Just, Marion R., Crigler, Ann N., and Belt, Todd L.. 2007. ‘Don’t Give Up Hope: Emotions, Candidate Appraisals, and Votes’. In W. Russell Neuman, George E. Marcus, Ann N. Crigler and Michael B. MacKuen (eds), The Affect Effect, 231–259. Chicago, IL: University of Chicago Press.CrossRefGoogle Scholar
Kahneman, Daniel. 2011. Thinking, Fast and Slow. New York: Farrar, Straus and Giroux.Google Scholar
Krosnick, Jon A., and Presser, Stanley. 2010. ‘Question and Questionnaire Design’. In Peter V. Wright Marsden and James D. Bingley (eds), Handbook of Survey Research, 263–314. UK: Emerald Group.Google Scholar
Lerner, Jennifer S., and Keltner, Dacher. 2000. ‘Beyond Valence: Toward a Model of Emotion-Specific Influences on Judgement and Choice’. Cognition and Emotion 14:473–493.CrossRefGoogle Scholar
Lerner, Jennifer S., and Keltner, Dacher. 2001. ‘Fear, Anger, and Risk’. Journal of Personality and Social Psychology 81:146–159.CrossRefGoogle ScholarPubMed
Lerner, Jennifer S., Gonzalez, Roxana M., Small, Deborah A., and Fischhoff, Baruch. 2003. ‘Effects of Fear and Anger on Perceived Risks of Terrorism: A National Field Experiment’. Psychological Science 14:144–150.CrossRefGoogle ScholarPubMed
Lodge, G. Milton, and Tursky, Bernard. 1979. ‘Comparisons Between Category and Magnitude Scaling of Political Opinion Employing SRC/CPS Items’. American Political Science Review 73:50–66.CrossRefGoogle Scholar
Lodge, G. Milton, and Tursky, Bernard. 1981. ‘On the Magnitude Scaling of Public Opinion in Survey Research’. American Journal of Political Science 25:376–419.CrossRefGoogle Scholar
Lord, Charles G., Ross, Lee, and Lepper, Mark R.. 1979. ‘Biased Assimilation and Attitude Polarization: The Effects of Prior Theories on Subsequently Considered Evidence’. Journal of Personality and Social Psychology 37:2098–2109.CrossRefGoogle Scholar
MacKuen, Michael B., Wolak, Jennifer, Keele, Luke, and Marcus, George E.. 2010. ‘Civic Engagements: Resolute Partisanship or Reflective Deliberation’. American Journal of Political Science 54:440–458.CrossRefGoogle Scholar
MacKuen, Michael B., Miller, Patrick R., Marcus, George E., and Neuman, W. Russell. 2013. ‘Affective Signatures and Attention: The Persistent Impact of Emotional Responses to the News’. Midwest Political Science Association Annual Meeting, April 11–13, 2013, Chicago, IL.Google Scholar
Marcus, George E. 2002. The Sentimental Citizen: Emotion in Democratic Politics. University Park, PA: Pennsylvania State University Press.Google Scholar
Marcus, George E. 2003. ‘The Psychology of Emotion and Politics’. In David O. Sears, Leonie Huddy and Robert Jervis (eds), Oxford Handbook of Political Psychology, 182–221. Oxford: Oxford University Press.Google Scholar
Marcus, George E. 2008. ‘Different Situations, Different Responses: Threat, Partisanship, Risk, and Deliberation’. Critical Review 20:75–89.CrossRefGoogle Scholar
Marcus, George E., and MacKuen, Michael B.. 1993. ‘Anxiety, Enthusiasm and the Vote: The Emotional Underpinnings of Learning and Involvement During Presidential Campaigns’. American Political Science Review 87:688–701.CrossRefGoogle Scholar
Marcus, George E., MacKuen, Michael B., Wolak, Jennifer, and Keele, Luke. 2006. ‘The Measure and Mismeasure of Emotion’. In David Redlawsk (ed.), Feeling Politics: Emotion in Political Information Processing, 31–45. New York: Palgrave Macmillan.CrossRefGoogle Scholar
Marcus, George E., Neuman, W. Russell, and MacKuen, Michael B. 2000. Affective Intelligence and Political Judgment. Chicago, IL: University of Chicago Press.Google Scholar
Marcus, George E., Neuman, W. Russell, and MacKuen, Michael B. 2009a. ‘Assessing Subjective Emotional Responses: Two Approaches to Measurement’. Midwest Political Science Association Annual Meetings. Chicago, IL.Google Scholar
Marcus, George E., Neuman, W. Russell, and MacKuen, Michael B. 2009b. ‘Measuring Subjective Emotional Response: New Evidence on an Alternative Method’. Annual Scientific Meeting of the International Society of Political Psychology, July 14–17, 2009. Dublin, Ireland.Google Scholar
Masters, Roger D. 2001. ‘Biology and Politics: Linking Nature and Nurture’. Annual Review of Political Science 4:345–369.CrossRefGoogle Scholar
Masters, Roger D., and Sullivan, Denis. 1993. ‘Nonverbal Behavior and Leadership: Emotion and Cognition in Political Attitudes’. In Shanto Iyengar and William McGuire (eds), Explorations in Political Psychology, 150–182. Durham, NC: Duke University Press.Google Scholar
Masters, Roger D., and Sullivan, Denis G. 1989. ‘Facial Displays and Political Leadership in France’. Behavioral Processes 19:1–30.CrossRefGoogle ScholarPubMed
Mauss, Iris B., and Robinson, Michael D.. 2009. ‘Measures of Emotion: A Review’. Cognition and Emotion 23:209–237.CrossRefGoogle ScholarPubMed
Mayer, John D., and Gaschke, Yvonne N.. 1988. ‘The Experience and Meta-Experience of Mood’. Journal of Personality and Social Psychology 55:102–111.CrossRefGoogle ScholarPubMed
McGuire, William J. 1969. ‘The Nature of Attitudes and Attitude Change’. In Gardner Lindzey and Elliot Aronson (eds), The Handbook of Social Psychology, 136–314. Reading, MA: Addison-Wesley.Google Scholar
McHugo, Gregory J., Lanzetta, John T., Sullivan, Denis G., Masters, Roger D., and Englis, Basil. 1985. ‘Emotional Reactions to Expressive Displays of a Political Leader’. Journal of Personality and Social Psychology 49:1512–1529.CrossRefGoogle Scholar
Mutz, Diana C., and Reeves, Byron. 2005. ‘The New Videomalaise: Effects of Televised Incivility on Political Trust’. American Political Science Review 99:1–15.CrossRefGoogle Scholar
Neuman, W. Russell, Marcus, George E., and MacKuen, Michael B.. 2013. ‘Hardwired for News: Affective Intelligence and Political Attention’. Paper presented at the American Political Science Association Annual Meetings, Chicago, IL.Google Scholar
Nunnally, Jum C., and Bernstein, Ira H.. 1994. Psychometric Theory. New York: McGraw-Hill.Google Scholar
Ortony, Andrew, Clore, Gerald L., and Collins, Allan. 1989. The Cognitive Structure of Emotions. New York: Cambridge University Press.Google Scholar
Plutchik, Robert, and Conte, Hope R.. 1997. Circumplex Models of Personality and Emotions. Washington, DC: American Psychological Association.CrossRefGoogle Scholar
Rammstedt, Beatrice, and John, Oliver P.. 2007. ‘Measuring Personality in One Minute or Less: A 10-Item Short Version of the Big Five Inventory in English and German’. Journal of Research in Personality 41:203–212.CrossRefGoogle Scholar
Redlawsk, David P. 2002. ‘Hot Cognition or Cool Consideration? Testing the Effects of Motivated Reasoning on Political Decision Making’. Journal of Politics 64:1021–1044.CrossRefGoogle Scholar
Roseman, Ira. 1979. ‘Cognitive Aspects of Emotion and Emotional Behavior’. Paper presented at the 87th Annual Convention of the American Psychological Association, New York, September 1–5, 1979.Google Scholar
Roseman, Ira. 1984. ‘Cognitive Determinants of Emotions: A Structural Theory’. In Phillip R. Shaver (ed.), Review of Personality and Social Psychology, 11–36. Beverly Hills, CA: Sage.Google Scholar
Ruef, Anna M., and Levenson, Robert W. 2007. ‘Continuous Measurement of Emotion: The Affect Rating Dial’. In James A. Coan and John J. B. Allen (eds), The Handbook of Emotion Elicitation and Assessment, 286–297. New York: Oxford University Press.CrossRefGoogle Scholar
Russell, James A., and Carroll, James M. 1999. ‘On the Bipolarity of Positive and Negative Affect’. Psychological Bulletin 125:3–30.CrossRefGoogle ScholarPubMed
Sullivan, Dennis, and Masters, Roger. 1988. ‘Happy Warriors: Leaders’ Facial Displays, Viewers Emotions, and Political Support’. American Journal of Political Science 32:345–368.CrossRefGoogle Scholar
Valentino, Nicholas A., Brader, Ted, Groenendyk, Eric W., Gregorowicz, Krysha, and Hutchings, Vincent L.. 2011. ‘Election Night’s Alright for Fighting: The Role of Emotions in Political Participation’. Journal of Politics 73:156–170.CrossRefGoogle Scholar
Valentino, Nicholas A., Hutchings, Vincent L., Banks, Antoine J., and Davis, Anne K.. 2008. ‘Is a Worried Citizen a Good Citizen? Emotions, Political Information Seeking, and Learning Via the Internet’. Political Psychology 29:247–273.CrossRefGoogle Scholar
Watson, David. 1988aIntraindividual and Interindividual Analyses of Positive and Negative Affect: Their Relation to Health Complaints, Perceived Stress, and Daily Activities’. Journal of Personality and Social Psychology 54:1020–1030.CrossRefGoogle ScholarPubMed
Watson, David. 1988bThe Vicissitudes of Mood Measurement: Effects of Varying Descriptors, Time Frames, and Response Formats on Measures of Positive and Negative Affect’. Journal of Personality and Social Psychology 55:128–141.CrossRefGoogle ScholarPubMed
Watson, David, and Tellegen, Auke. 1985. ‘Toward a Consensual Structure of Mood’. Psychological Bulletin 98:219–235.CrossRefGoogle Scholar
Watson, David, Wiese, David, Vaidya, Jatin, and Tellegen, Auke. 1999. ‘The Two General Activation Systems of Affect: Structural Findings, Evolutionary Considerations, and Psychobiological Evidence’. Journal of Personality and Social Psychology 76:820–838.CrossRefGoogle Scholar
Watson, David, and Clark, Lee Anna. 1994. The PANAS-X: Manual for the Positive and Negative Affect Schedule-Expanded Form, 35.Google Scholar
Watson, David, and Clark, Lee Anna. 1997. ‘Measurement and Mismeasurement of Mood: Recurrent and Emergent Issues’. Journal of Personality Assessment 68:267–296.CrossRefGoogle ScholarPubMed
Watson, David, Clark, Lee Anna, and Tellegen, Auke. 1988. ‘Development and Validation of Brief Measures of Positive and Negative Affect: The PANAS Scales’. Journal of Personality and Social Psychology 54:1063–1070.CrossRefGoogle ScholarPubMed
Way, Baldwin M., and Masters, Roger D.. 1996. ‘Political Attitudes: Interactions of Cognition and Affect’. Motivation and Cognition 20:205–236.Google Scholar
Zajonc, Robert B. 1980. ‘Feeling and Thinking: Preferences Need No Inferences’. American Psychologist 35:151–175.CrossRefGoogle Scholar
Zuckerman, Marvin. 1991. Psychobiology of Personality. Cambridge, England: Cambridge University Press.Google Scholar
Supplementary material: PDF

Marcus supplementary material

Appendix

Download Marcus supplementary material(PDF)
PDF 798.5 KB