Hostname: page-component-848d4c4894-v5vhk Total loading time: 0 Render date: 2024-07-01T12:18:11.017Z Has data issue: false hasContentIssue false
  • English
  • Français

What you see is what you do: on the relationship between gaze and gesture in multimodal alignment

Published online by Cambridge University Press:  02 November 2015

BERT OBEN  and
GEERT BRÔNE
BERT OBEN
Affiliation:
University of Leuven, Department of Linguistics
GEERT BRÔNE
Affiliation:
University of Leuven, Department of Linguistics
Get access Rights & Permissions [Opens in a new window]

Abstract

Interactive language use inherently involves a process of coordination, which often leads to matching behaviour between interlocutors in different semiotic channels. We study this process of interactive alignment from a multimodal perspective: using data from head-mounted eye-trackers in a corpus of face-to-face conversations, we measure which effect gaze fixations by speakers (on their own gestures, condition 1) and fixations by interlocutors (on the gestures by those speakers, condition 2) have on subsequent gesture production by those interlocutors. The results show there is a significant effect of interlocutor gaze (condition 2), but not of speaker gaze (condition 1) on the amount of gestural alignment, with an interaction between the conditions.

Keywords

interactive alignmenteye-gazegesturemultimodalface-to-face conversation
Type
Research Article
Information
Language and Cognition , Volume 7 , Issue 4: Special issue on Cognitive Linguistics and Interactional Discourse , December 2015 , pp. 546 - 562
Copyright
Copyright © UK Cognitive Linguistics Association 2015 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

references

Argyle, M., & Cook, M. (1976). Gaze and mutual gaze. London: Cambridge University Press.Google Scholar
Bavelas, J., Coates, L., & Johnson, T. (2002). Listener responses as a collaborative process: the role of gaze. Journal of Communication, 52, 566–580.Google Scholar
Bolinger, D. L. (1961). Syntactic blends and other matters. Language, 37(3), 366–381.Google Scholar
Branigan, H., Pickering, M., & Cleland, A. (2000). Syntactic co-ordination in dialogue. Cognition, 75, 13–25.Google Scholar
Branigan, H., Pickering, M., McLean, J., & Cleland, A. (2007). Participant role and syntactic alignment in dialogue. Cognition, 104, 163–197.Google Scholar
Brennan, S., Chen, X., Dickinson, C., Neider, M., & Zelinsky, G. (2008). Coordinating cognition: the costs and benefits of shared gaze during collaborative search. Cognition, 106, 1465–1477.CrossRefGoogle ScholarPubMed
Brennan, S. E., & Clark, H. H. (1996). Conceptual pacts and lexical choice in conversation. Journal of Experimental Psychology: Learning, Memory, and Cognition, 22, 1482–93.Google ScholarPubMed
Bressem, J. (2008). Notating gestures – proposal for a form based notation system of coverbal gestures. Unpublished manuscript, European University Viadrina.Google Scholar
Brône, G., & Oben, B. (2015). InSight Interaction: a multimodal and multifocal dialogue corpus. Language Resources and Evaluation, 49, 195–214.Google Scholar
Carter, R. (1999). Common language: corpus, creativity and cognition. Language and Literature, 8(3), 195–216.Google Scholar
Cassell, J., Torres, O. E., & Prevost, S. (1999). Turn taking vs. discourse structure: how best to model multimodal conversation. The Hague: Kluwer.Google Scholar
Chartrand, T., & Bargh, J. (1999). The chameleon effect: the perception-behavior link and social interaction. Journal of Personality and Social Psychology, 76, 893–910.Google Scholar
Cienki, A., & Müller, C. (2008). Metaphor and gesture. Amsterdam/Philadelphia: John Benjamins.Google Scholar
Clark, H. H. (1996). Using language. Cambridge: Cambridge University Press.Google Scholar
Clark, H. H., & Brennan, S. E. (1991). Grounding in communication. In Resnick, L. B., Levine, J. M., & Teasley, S. D. (Eds.), Perspectives on socially shared cognition (pp. 127–149). Washington, DC: American Psychological Association.Google Scholar
Condon, W., & Ogston, W. (1971). Speech and body motion synchrony of the speaker–hearer. In Horton, D. & Jenkins, J. J. (Eds.), The perception of language (pp. 150–184). Columbus, OH: Charles E. Merrill.Google Scholar
Dale, R., Kirkham, N., & Richardson, D. (2011). How two people become a tangram recognition system. Proceedings of the European Conference on Computer-Supported Cooperative Work. Berlin: Springer Verlag. Online: <http://www.researchgate.net/publication/228506291_How_two_people_become_a_tangram_recognition_system>.Google Scholar
Du Bois, J. (2010). Towards a dialogic syntax. Unpublished manuscript, University of California, Santa Barbara.Google Scholar
Emery, N. J. (2000). The eyes have it: the neuroethology, function and evolution of social gaze. Neuroscience and Biobehavioral Reviews, 24(6), 581–604.Google Scholar
Fried, M., & Östman, J.-O. (2005). Construction grammar and spoken language: the case of pragmatic particles. Journal of Pragmatics, 37(11), 1752–1778.Google Scholar
Frischen, A., Bayliss, A. P., & Tipper, S. P. (2007). Gaze cueing of attention: visual attention, social cognition, and individual differences. Psychological Bulletin, 133(4), 694–724.Google Scholar
Garrod, S., & Pickering, M. J. (2004). Why is conversation so easy? Trends in Cognitive Sciences, 8, 8–11.CrossRefGoogle ScholarPubMed
Garrod, S., & Pickering, M. J. (2009). Joint action, interactive alignment, and dialog. Topics in Cognitive Science, 1, 292–304.Google Scholar
Giles, H., Coupland, N., & Coupland, J. (1992). Accommodation theory: communication, context, and consequence. In Giles, H., Coupland, J., & Coupland, N. (Eds.), Contexts of accommodation: developments in applied sociolinguistics (pp. 1–68). Cambridge: Cambridge University Press.Google Scholar
Goldberg, A. (2006). Constructions at work. Oxford: Oxford University Press.Google Scholar
Gullberg, M., & Holmqvist, K. (2006). What speakers do and what addressees look at: visual attention to gestures in human interaction live and on video. Pragmatics & Cognition, 14(1), 53–82.Google Scholar
Gullberg, M., & Kita, S. (2009). Attention to speech-accompanying gestures: eye movements and information uptake. Journal of Nonverbal Behaviour, 33, 251–277.Google Scholar
Halliday, M. A. K., & Hasan, R. (1976). Cohesion in English. London: Longman.Google Scholar
Jokinen, K. (2010). Non-verbal signals for turn-taking and feedback. Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC), online: <http://www.lrec-conf.org/proceedings/lrec2010/pdf/173_Paper.pdf>.Google Scholar
Kendon, A. (1967). Some functions of gaze-direction in social interaction. Acta Psychologica, 26, 22–63.CrossRefGoogle ScholarPubMed
Kendon, A. (2004). Gesture: visible action as utterance. Cambridge: Cambridge University Press.Google Scholar
Kimbara, I. (2006). On gestural mimicry. Gesture, 6, 39–61.Google Scholar
Lachat, F., Conty, L., Hugueville, L., & George, N. (2012). Gaze cueing effect in a face-to-face situation. Journal of Nonverbal Behavior, 36, 177–190.Google Scholar
Langacker, R. W. (2001). Discourse in Cognitive Grammar. Cognitive Linguistics, 12, 143–188.Google Scholar
Lewandowski, N. (2012). Talent in nonnative phonetic convergence. Unpublished doctoral dissertation, Universität Stuttgart.Google Scholar
Lewandowski, N., & Schweitzer, A. (2010). Prosodic and segmental convergence in spontaneous German conversations. Journal of the Acoustical Society of America, 128(4), 2458.Google Scholar
McNeill, D. (2006). Gesture, gaze, and ground. In Renals, S. & Bengio, S. (Eds.), Proceedings of Machine Learning For Multimodal Interaction: second international workshop 2005 (pp. 1–14). Berlin/Heidelberg: Springer Verlag.Google Scholar
Mittelberg, I. (2007). Methodology for multimodality: one way of working with speech and gesture data. In M. Gonzalez-Marquez et al. (Eds.), Methods in cognitive linguistics (pp. 225–248). Amsterdam/Philadelphia: John Benjamins.Google Scholar
Muir, L., & Richardson, I. (2005). Perceptions of sign language and its application to visual communications for deaf people. Journal of Deaf Studies and Deaf Education, 10(4), 390–401.Google Scholar
Munn, S., Stefano, L., & Pelz, J. (2008). Fixation-identification in dynamic scenes: comparing an automated algorithm to manual coding. Proceedings of the 5th Symposium on Applied Perception in Graphics and Visualization (APGV 08) (pp. 33–42), online: <http://dl.acm.org/citation.cfm?id=1394281&picked=prox>.Google Scholar
Neider, M., Chen, X., Dickinson, C., Brennan, S., & Zelinsky, G. (2010). Coordinating spatial referencing using shared gaze. Psychonomic Bulletin & Review, 17(5), 718–724.CrossRefGoogle ScholarPubMed
Novick, D. G., Hansen, B., & Ward, K. (1996). In Bunnell, T. & Idsardi, W. (Eds.), Proceedings of the International Conference on Spoken Language Processing (pp. 1888–1891), Philadelphia, 3–6 October.Google Scholar
Pickering, M., & Garrod, S. (2004). Toward a mechanistic psychology of dialogue. Behavioral and Brain Sciences, 27, 169–226.Google Scholar
Pickering, M., & Garrod, S. (2006). Alignment as the basis for successful communication. Research on Language and Computation, 4, 203–228.Google Scholar
Posner, M. I., Snyder, C. R., & Davidson, B. J. (1980). Attention and the detection of signals. Journal of Experimental Psychology, 109(2), 160–174.Google Scholar
Raidt, S. (2008). Gaze and face-to-face communication between a human speaker and an animated conversational agent – mutual attention and multimodal deixis. Unpublished doctoral dissertation, University of Grenoble.Google Scholar
Rayner, K. (1998). Eye movements in reading and information processing: 20 years of research. Psychological Bulletin, 85, 618–660.Google Scholar
Richardson, D., & Dale, R. (2005). Looking to understand: the coupling between speakers’ and listeners’ eye movements and its relationship to discourse comprehension. Cognitive Science, 29, 1045–1060.Google Scholar
Richardson, D., Dale, R., & Kirkham, N. (2007). The art of conversation is coordination: common ground and the coupling of eye movements during dialogue. Psychological Science, 18, 407–413.Google Scholar
Richardson, D., Dale, R., & Tomlinson, J. (2009). Conversation, gaze coordination, and beliefs about visual context. Cognitive Science, 33, 1468–1482.Google Scholar
Roche, J. M., Dale, R., & Caucci, G. M. (2012). Doubling up on double meanings: pragmatic alignment. Language and Cognitive Processes, 27(1), 1–24.Google Scholar
Sacks, H., Schegloff, E. A. & Jefferson, G. (1974). A simplest systematics for the organisation of turn-taking for conversation. Language, 50, 696–735.Google Scholar
Schober, M. F. (1993). Spatial perspective-taking in conversation. Cognition, 47(1), 1–24.Google Scholar
Shockley, K., Richardson, D. C., & Dale, R. (2009). Conversation and coordinative structures. Topics in Cognitive Science, 1(2), 305–319.CrossRefGoogle ScholarPubMed
Shockley, K., Santana, M. V., & Fowler, C. A. (2003). Mutual interpersonal postural constraints are involved in cooperative conversation. Journal of Experimental Psychology: Human Perception and Performance, 29, 326–332.Google Scholar
Streeck, J. (1993). Gesture as communication I: its coordination with gaze and speech. Communication Monographs, 60, 275–299.Google Scholar
Streeck, J. (2008). Depicting by gesture. Gesture, 8, 285–301.Google Scholar
Sweetser, E. (2007). Looking at space to study mental spaces: co-speech gestures as a crucial data source in cognitive linguistics. In Gonzalez-Marquez, M.et al, . (Eds.), Methods in cognitive linguistics (pp. 203–226). Amsterdam/Philadelphia: John Benjamins.Google Scholar
Szczepek Reed, B. (2010). Prosody and alignment: a sequential perspective. Cultural Studies of Science Education, 5(4), 859–867.Google Scholar
Tannen, D. (1987). Repetition in conversation: toward a poetics of talk. Language, 63(3), 574–605.Google Scholar
Tannen, D. (1989). Talking voices: repetition, dialogue, and imagery in conversational discourse. Cambridge: Cambridge University Press.Google Scholar
Verhagen, A. (2005). Constructions of intersubjectivity. Oxford: Oxford University Press.Google Scholar
Wang, Y., & Hamilton, A. (2014). Why does gaze enhance mimicry? Placing gaze-mimicry effects in relation to other gaze phenomena. Quarterly Journal of Experimental Psychology, 67, 747–762.Google Scholar
Wang, Y., Newport, R., & Hamilton, A. (2011). Eye contact enhances mimicry of intransitive hand movements. Biology Letters, 7, 7–10.Google Scholar