Hostname: page-component-848d4c4894-p2v8j Total loading time: 0.001 Render date: 2024-06-01T15:15:05.827Z Has data issue: false hasContentIssue false
  • English
  • Français

Semantic integration of multidimensional perceptual information in L1 sentence comprehension

Published online by Cambridge University Press:  20 December 2021

Bing Bai Open the ORCID record for Bing Bai [Opens in a new window] ,
Caimei Yang Open the ORCID record for Caimei Yang [Opens in a new window]  and
Jiabao Fan
Bing Bai
Affiliation:
Department of English, Soochow University, China
Caimei Yang*
Affiliation:
Department of English, Soochow University, China
Jiabao Fan
Affiliation:
Department of English, Soochow University, China
*
*Corresponding author. Email: cmyang@suda.edu.cn
Get access Rights & Permissions [Opens in a new window]

Abstract

Many studies have substantiated the perceptual symbol system, which assumes a routine generation of perceptual information during language comprehension, but little is known about the processing format in which the perceptual information of different dimensions is conveyed simultaneously during sentence comprehension. The current study provides the first experimental evidence of how multidimensional perceptual information (color and shape) was processed during online sentence comprehension in Mandarin. We designed three consecutive sentence–picture verification tasks that only differed in the delay of the display of pictures preceded by declarative sentences. The processing was analyzed in three stages based on time intervals (i.e., 0ms, +750ms, +1500ms). The response accuracy and response time data were reported. The initial stage (i.e., ISI=0ms) attested the match effect of color and shape, but the simulated representation of color and shape did not interact. In the intermediate stage (i.e., ISI=750ms), the routinely simulated color and shape interacted, but the match facilitation was found only in cases where one perceptual information was in mismatch while the other was not. In the final stage (i.e., ISI=1500ms), the match facilitation of one particular perceptual property was influenced by a mismatch with the other perceptual property. These results suggested that multiple perceptual information presented simultaneously was processed in an additive manner to a large extent before entering into the final stage, where the simulated perceptual information was integrated in a multiplicative manner. The results also suggested that color and shape were comparable to object recognition when conjointly conveyed. In relation to other evidence from behavioral and event-related potential studies on sentence reading in the discussion, we subscribed to the idea that the full semantic integration became available over time.

Keywords

semantic integrationL1colorshapesentence comprehensionembodied cognition
Type
Article
Information
Language and Cognition , Volume 14 , Issue 1 , March 2022 , pp. 109 - 130
Check for updates
Copyright
© The Author(s), 2021. Published by Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Aginsky, V. & Tarr, M. J. (2000). How are different properties of a scene encoded in visual memory? Visual Cognition 7(1), 147–162.CrossRefGoogle Scholar
Anllo-Vento, L. & Hillyard, S. A. (1996). Selective attention to the color and direction of moving stimuli: electrophysiological correlates of hierarchical feature selection. Perception & Psychophysics 58(2), 191–206.CrossRefGoogle Scholar
Baayen, R. H., Davidson, D. J. & Bates, D. M., (2008). Mixed-effects modeling with crossed random effects for subjects and items. Journal of Memory and Language 59(4), 390–412.10.1016/j.jml.2007.12.005CrossRefGoogle Scholar
Barr, D. J., Levy, R., Scheepers, C. & Tily, H. J. (2013). Random effects structure for confirmatory hypothesis testing: keep it maximal. Journal of Memory and Language 68(3), 255–278.CrossRefGoogle ScholarPubMed
Barsalou, L. W. (1999). Perceptions of perceptual symbols. Behavioral and Brain Sciences 22(4), 637–660.CrossRefGoogle Scholar
Bates, D., Kliegl, R., Vasishth, S. & Baayen, H. (2015). Parsimonious mixed models. Online <http://arxiv.org/abs/1506.04967>..>Google Scholar
Berndt, E., Dudschig, C. & Kaup, B. (2020). Green as a cbemcuru: modal as well as amodal color cues can help to solve anagrams. Psychological Research 84(2), 491–501.CrossRefGoogle Scholar
Biederman, I. & Cooper, E. E. (1991). Priming contour-deleted images: evidence for intermediate representations in visual object recognition. Cognitive Psychology 23(3), 393–419.CrossRefGoogle ScholarPubMed
Bransford, J. D. & Franks, J. J. (1971). The abstraction of linguistic ideas. Cognitive Psychology 2(4), 331–350.CrossRefGoogle Scholar
Briner, S. W., Virtue, S. M. & Schutzenhofer, M. C. (2014). Hemispheric processing of mental representations during text comprehension: evidence for inhibition of inconsistent shape information. Neuropsychologia 61, 96–104.CrossRefGoogle ScholarPubMed
Collins, A. M. & Quillian, M. R. (1969). Retrieval time from semantic memory. Journal of Verbal Learning and Verbal Behavior 8(2), 240–247.CrossRefGoogle Scholar
Connell, L. (2007). Representing object colour in language comprehension. Cognition 102(3), 476–485.CrossRefGoogle ScholarPubMed
Connell, L. & Lynott, D. (2009). Is a bear white in the woods? Parallel representation of implied object color during language comprehension. Psychonomic Bulletin and Review 16(3), 573–577.CrossRefGoogle Scholar
de Koning, B. B., Wassenburg, S. I., Bos, L. T. & van der Schoot, M. (2016). Size does matter: implied object size is mentally simulated during language comprehension. Discourse Processes 54(7), 493–503.10.1080/0163853X.2015.1119604CrossRefGoogle Scholar
de Koning, B. B., Wassenburg, S. I., Bos, L. T. & van der Schoot, M. (2017). Mental simulation of four visual object properties: similarities and differences as assessed by the sentence–picture verification task. Journal of Cognitive Psychology 29(4), 420–432.CrossRefGoogle Scholar
Dirix, N., Cop, U., Drieghe, D. & Duyck, W. (2017). Cross-lingual neighborhood effects in generalized lexical decision and natural reading. Journal of Experimental Psychology: Learning, Memory, and Cognition 43(6), 887–915.Google ScholarPubMed
Duncan, J. (1984). Selective attention and the organization of visual information. Journal of Experimental Psychology: General 113(4), 501–517.CrossRefGoogle ScholarPubMed
Eimer, M. (1996). The N2pc component as an indicator of attentional selectivity. Electroencephalography and Clinical Neurophysiology 99(3), 225–234.CrossRefGoogle ScholarPubMed
Garnham, A. & Oakhill, J. (1992). Discourse processing and text representation from a ‘Mental Models’ perspective. Language and Cognitive Processes 7(3/4), 193–204.CrossRefGoogle Scholar
Gerlach, C., Aaside, C. T., Humphreys, G. W., Gade, A., Paulson, O. B. & Law, I. (2002). Brain activity related to integrative processes in visual object recognition: bottom-up integration and the modulatory influence of stored knowledge. Neuropsychologia 40(8), 1254–1267.CrossRefGoogle ScholarPubMed
Gershkoff-Stowe, L. & Smith, L. B. (2004). Shape and the first hundred words. Child Development 75(4), 1–17.CrossRefGoogle Scholar
Giora, R., Fein, O., Ganzi, J., Levi, N. A. & Sabah, H. (2005). On negation as mitigation: the case of negative irony. Discourse Processes 39(1), 81–100.CrossRefGoogle Scholar
Glass, A. L. & Holyoak, K. J. (1974). Alternative conceptions of semantic theory. Cognition 3(4), 313–339.CrossRefGoogle Scholar
Glenberg, A. M., Meyer, M. & Lindem, K. (1987). Mental models contribute to foregrounding during text comprehension. Journal of Memory and Language 26(1), 69–83.CrossRefGoogle Scholar
Graham, S. A. & Diesendruck, G. (2010). Fifteen-month-old infants attend to shape over other perceptual properties in an induction task. Cognitive Development 25(2), 111–123.CrossRefGoogle Scholar
Harris, I. M. & Dux, P. E. (2005). Orientation-invariant object recognition: evidence from repetition blindness. Cognition 95(1), 73–93.CrossRefGoogle ScholarPubMed
Hasson, U. & Glucksberg, S. (2006). Does understanding negation entail affirmation? Journal of Pragmatics 38(7), 1015–1032.CrossRefGoogle Scholar
Holman, A. C. & Gîrbă, A. E. (2019). The match in orientation between verbal context and object accelerates change detection. Psihologija 52(1), 93–105.CrossRefGoogle Scholar
Huettig, F., Guerra, E. & Helo, A. (2020). Towards understanding the task dependency of embodied language processing: the influence of colour during language-vision interactions. Journal of Cognition 3(1), 41.CrossRefGoogle ScholarPubMed
Hupp, J. M., Jungers, M. K., Porter, B. L. & Plunkett, B. A. (2020). The implied shape of an object in adults’ and children’s visual representations. Journal of Cognition and Development 21(3), 368–382.CrossRefGoogle Scholar
Jackson, F. (1977). Perception: a representative theory. Cambridge: Cambridge University Press.Google Scholar
Kang, X., Joergensen, G. H. & Altmann, G. T. (2020). The activation of object-state representations during online language comprehension. Acta Psychologica 210, 103–162.CrossRefGoogle ScholarPubMed
Karayanidis, F. & Michie, P. T. (1997). Evidence of visual processing negativity with attention to orientation and color in central space. Electroencephalography and Clinical Neurophysiology 103(2), 282–297.CrossRefGoogle ScholarPubMed
Kaup, B. (2001). Negation and its impact on the accessibility of text information. Memory and Cognition 29(7), 960–967.CrossRefGoogle ScholarPubMed
Kaup, B., Lüdtke, J. & Zwaan, R. A. (2006). Processing negated sentences with contradictory predicates: Is a door that is not open mentally closed? Journal of Pragmatics 38(7), 1033–1050.CrossRefGoogle Scholar
Kaup, B. & Zwaan, R. A. (2003). Effects of negation and situational presence on the accessibility of text information. Journal of Experimental Psychology: Learning, Memory and Cognition 29(3), 439–446.Google ScholarPubMed
Landau, B., Smith, L. & Jones, S. (1998). Object shape, object function, and object name. Journal of Memory and Language 38(1), 1–27.CrossRefGoogle Scholar
Lincoln, A. E., Long, D. L. & Baynes, K. (2007). Hemispheric differences in the activation of perceptual information during sentence comprehension. Neuropsychologia 45(2), 397–405.CrossRefGoogle ScholarPubMed
Loftus, E. F & Scheff, R. W. (1971). Categorization norms for fifty representative instances. Journal of Experimental Psychology 91(2), 355–364.CrossRefGoogle Scholar
Lüdtke, J., Friedrich, C. K., De Filippis, M. & Kaup, B. (2008). Event-related potential correlates of negation in a sentence–picture verification paradigm. Journal of Cognitive Neuroscience 20(8), 1355–1370.10.1162/jocn.2008.20093CrossRefGoogle Scholar
Mannaert, L. N. H., Dijkstra, K. & Zwaan, R. A. (2017). Is color an integral part of a rich mental simulation? Memory and Cognition 45(6), 974–982.CrossRefGoogle Scholar
Mannaert, L. N. H., Dijkstra, K. & Zwaan, R. A. (2019). How are mental simulations updated across sentences? Memory and Cognition 47(6), 1201–1214.CrossRefGoogle Scholar
Mannaert, L. N. H., Dijkstra, K. & Zwaan, R. A. (2021). Is color continuously activated in mental simulations across a broader discourse context? Memory and Cognition 49(1), 127–147.CrossRefGoogle Scholar
Michie, P. T., Karayanidis, F., Smith, G. L., Barrett, N. A., Large, M. M., O’Sullivan, B. T. & Kavanagh, D. J. (1999). An exploration of varieties of visual attention: ERP findings. Cognitive Brain Research 7(4), 419–450.CrossRefGoogle ScholarPubMed
Nigg, J. T. (2017). Annual research review: on the relations among self-regulation, self-control, executive functioning, effortful control, cognitive control, impulsivity, risk-taking, and inhibition for developmental psychopathology. Journal of Child Psychology and Psychiatry 58(4), 361–383.CrossRefGoogle ScholarPubMed
O’Craven, K. M., Downing, P. E. & Kanwisher, N. (1999). fMRI evidence for objects as the units of attentional selection. Nature 401(6753), 584–587.CrossRefGoogle ScholarPubMed
Pecher, D., van Dantzig, S., Zwaan, R. A. & Zeelenberg, R. (2009). Language comprehenders retain implied shape and orientation of objects. Quarterly Journal of Experimental Psychology 62(6), 1108–1114.CrossRefGoogle Scholar
Pickering, M., Crocker, M. W. & Clifton, C. (1999). Architectures and mechanisms in sentence comprehension. In Pickering, M., Crocker, M. W. & Clifton, C. (eds), Architectures and mechanisms for language processing (pp. 1–28). Cambridge: Cambridge University Press.Google Scholar
Posner, M. I. (1970). Abstraction and the process of recognition. In Bower, G. H. & Spence, J. T. (eds), Psychology of Learning and Motivation (Vol. 3, pp. 43–100). New York: Academic Press.Google Scholar
Proverbio, A. M., Burco, F., del Zotto, M. & Zani, A. (2004). Blue piglets? Electrophysiological evidence for the primacy of shape over color in object recognition. Cognitive Brain Research 18(3), 288–300.CrossRefGoogle ScholarPubMed
Redmann, A., FitzPatrick, I. & Indefrey, P. (2019). The time course of colour congruency effects in picture naming. Acta Psychologica 196, 96–108.CrossRefGoogle ScholarPubMed
Richter, T. & Zwaan, R. A. (2009). Processing of color words activates color representations. Cognition 111(3), 383–389.CrossRefGoogle ScholarPubMed
Richter, T. & Zwaan, R. A. (2010). Integration of perceptual information in word access. Quarterly Journal of Experimental Psychology 63(1), 81–107.CrossRefGoogle ScholarPubMed
Rommers, J., Meyer, A. S. & Huettig, F. (2013). Object shape and orientation do not routinely influence performance during language processing. Psychological Science 24(11), 2218–2225.CrossRefGoogle Scholar
Rosch, E. H. (1973). Natural categories. Cognitive Psychology 4(3), 328–350.CrossRefGoogle Scholar
R Core Team (2013). R: a Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna. Online <http://www.R-project.org/>..>Google Scholar
Samuelson, L. K. & Smith, L. B. (2005). They call it like they see it: spontaneous naming and attention to shape. Developmental Science 8(2), 182–198.CrossRefGoogle Scholar
Sato, M., Schafer, A. J. & Bergen, B. K. (2013). One word at a time: mental representations of object shape change incrementally during sentence processing. Language and Cognition 5(4), 345–373.CrossRefGoogle Scholar
Sera, M. D. & Millett, K. G. (2011). Developmental differences in shape processing. Cognitive Development 26(1), 40–56.CrossRefGoogle Scholar
Shipiro, S. I. & Palermo, D. S. (1970). Conceptual organization and class membership: normative data for representatives of 100 categories. Psychonomic Monograph Supplements 3(11), 107–127.Google Scholar
Smith, E. E, Shoben, E. J. & Rips, L. J. (1974). Structure and process in semantic memory: a featural model for semantic decisions. Psychological Review 81(3). 214–241.CrossRefGoogle Scholar
Stahl, C., Voss, A., Schmitz, F., Nuszbaum, M., Tüscher, O., Lieb, K. & Klauer, K. C. (2014). Behavioral components of impulsivity. Journal of Experimental Psychology 143(2), 850–886.CrossRefGoogle ScholarPubMed
Stanfield, R. A. & Zwaan, R. A. (2001). The effect of implied orientation derived from verbal context on picture recognition. Psychological Science 12(2), 153–156.CrossRefGoogle ScholarPubMed
Tanaka, J., Weiskopf, D. & Williams, P. (2001). The role of color in high-level vision. Trends in Cognitive Sciences 5(5), 211–215.CrossRefGoogle ScholarPubMed
Vandenbeld, L. A. & Rensink, R. A. (2003). The decay characteristics of size, color, and shape information in visual short-term memory. Journal of Vision 3(9), 682.10.1167/3.9.682CrossRefGoogle Scholar
van Weelden, L., Schilperoord, J. & Maes, A. (2014). Evidence for the role of shape in mental representations of similes. Cognitive Science 38(2), 303–321.CrossRefGoogle ScholarPubMed
Zani, A. & Proverbio, A. M. (1995). ERP signs of early selective attention effects to check size. Electroencephalography and Clinical Neurophysiology 95(4), 277–292.10.1016/0013-4694(95)00078-DCrossRefGoogle ScholarPubMed
Zeng, T., Zheng, L. & Mo, L. (2016). Shape representation of word was automatically activated in the encoding phase. PloS One 11(10), e0165534.CrossRefGoogle ScholarPubMed
Zwaan, R. A. & Pecher, D. (2012). Revisiting mental simulation in language comprehension: six replication attempts. Plos One 7(12), e51382.CrossRefGoogle ScholarPubMed
Zwaan, R. A. & Radvansky, G. A. (1998). Situation models in language comprehension and memory. Psychological Bulletin 123(2), 162–185.CrossRefGoogle ScholarPubMed
Zwaan, R. A., Stanfield, R. A. & Yaxley, R. H. (2002). Language comprehenders mentally represent the shapes of objects. Psychological Science 13(2), 168–171.CrossRefGoogle Scholar
Zwaan, R. A. & Yaxley, R. H. (2004). Lateralization of object-shape information in semantic processing. Cognition 94(2), B35–B43.CrossRefGoogle ScholarPubMed