Skip to main content Accessibility help

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings.

Close cookie message
×
Hostname: page-component-7479d7b7d-qlrfm Total loading time: 0 Render date: 2024-07-12T14:06:58.308Z Has data issue: false hasContentIssue false

12 - Multimodality

from Part III - Discourse Materialities and Embodiment

Published online by Cambridge University Press:  28 September 2020

By
Sabine Tan ,
Kay O’Halloran  and
Peter Wignell
Edited by
Anna De Fina  and
Alexandra Georgakopoulou
Anna De Fina
Affiliation:
Georgetown University, Washington DC
Alexandra Georgakopoulou
Affiliation:
King's College London
Get access

Summary

This chapter provides an overview of the recent theoretical, methodological and analytical trends in multimodal research, specifically focusing on how different approaches (e.g. critical discourse analysis, conversation analysis, social semiotics, systemic functional linguistics and interaction analysis) have addressed the complex problems arising from studying the integration of language with other resources, such as images, gesture, movement, space and so forth. In doing so, the chapter discusses how traditional divisions in discourse studies have become somewhat blurred, given the evident need to account for resources other than language and the meaning that arises as choices combine in texts, interactions and events. The chapter also explores how various digital approaches have been developed to handle the multidimensional complexity of multimodal analysis, in particular for the analysis of dynamic media such as videos. This discussion includes the development of mixed methods approaches, purpose-built software, automated techniques and the latest trends in big data approaches to multimodal analysis.

Keywords

Multimodalitycritical discourse analysissocial semioticssystemic functional linguisticsconversation analysisinteraction analysismixed methods approachespurpose-built softwareautomated techniquesbig data approaches
Type
Chapter
Information
The Cambridge Handbook of Discourse Studies , pp. 263 - 281
Publisher: Cambridge University Press
Print publication year: 2020

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Further Reading

This offers an introduction to multimodality from a practice-based perspective and approaches multimodal phenomena from different theoretical perspectives and disciplinary angles.

This provides a comprehensive introduction to various theoretical approaches to multimodality from different disciplines (e.g. visual studies, anthropology, conversation analysis, sociocultural theory, etc.).

This presents an accessible introduction to multimodality that illuminates the potential of multimodal research for understanding the ways in which people communicate. Offering a wide range of examples, clear practical support and a glossary of terms, the book is an ideal reference guide for beginners in multimodal analysis.

This is an essential resource for researchers interested in multimodal communication. Drawing on an enormous range of examples including children’s drawings, textbook illustrations, photojournalism, advertising images, fine art, websites, as well as three-dimensional material artefacts such as sculpture, the book presents a comprehensive account of the ways in which images communicate meaning.

This presents a systematic toolkit of theories, concepts and techniques for carrying out critical discourse analysis of language and images. Based on a variety of case studies and examples drawn from a range of traditional and new media genres, the book is an essential resource for beginners in critical discourse analysis.

Bateman, J., Wildfeuer, J. and Hiippala, T. (2017). Multimodality Foundations, Research and Analysis: A Problem-Oriented Introduction. Berlin: Mouton de Gruyter.Google Scholar
Jewitt, C. (2014). The Routledge Handbook of Multimodal Analysis, 2nd ed. London: Routledge.Google Scholar
Jewitt, C., Bezemer, J. and O’Halloran, K. L. (2016). Introducing Multimodality. London/New York: Routledge.Google Scholar
Kress, G. and van Leeuwen, T. ([1996]2006). Reading Images: The Grammar of Visual Design, 2nd ed. London: Routledge.Google Scholar
Machin, D. and Mayr, A. (2012). How to Do Critical Discourse Analysis: A Multimodal Introduction. London: Sage.Google Scholar

References

Barthes, R. ([1957]1987). Mythologies. New York: Hill & Wang.Google Scholar
Barthes, R. (1977). Image, Music, Text. London: Fontana Press.Google Scholar
Bateman, J. (2014). Looking for What Counts in Film Analysis: A Program of Empirical Research. In Machin, D. (ed.) Visual Communication. Berlin: Mouton de Gruyter. 301–30.Google Scholar
Bateman, J. and Schmidt, K.-H. (2012). Multimodal Film Analysis: How Films Mean. London/New York: Routledge.Google Scholar
Bateman, J., McDonald, D., Hiippala, T., Couto-Vale, D. and Costetchi, E. (2019). Systemic Functional Linguistics and Computation: New Directions, New Challenges. In Thompson, G., Bowcher, W. L., Fontaine, L. and Schönthal, D. (eds.) The Cambridge Handbook of Systemic Functional Linguistics. Cambridge: Cambridge University Press. 561–86.Google Scholar
Bateman, J., Tseng, C. I., Seizov, O., J acobs, A., Lüdtke, A., Müller, M. G. and Herzog, O. (2016). Towards Next-Generation Visual Archives: Image, Film and Discourse. Visual Studies 31(2): 131–54.Google Scholar
Bateman, J., Wildfeuer, J. and Hiippala, T. (2017). Multimodality Foundations, Research and Analysis: A Problem-Oriented Introduction. Berlin: De Gruyter Mouton.Google Scholar
Bernstein, B. (1990). Class, Codes and Control, Vol. IV: The Structuring of Pedagogic Discourse. London: Routledge.Google Scholar
Bezemer, J. and Jewitt, C. (2009). Social Semiotics. In Östman, J.-O., Verschueren, J. and Versluys, E. (eds.) Handbook of Pragmatics: 2009 Installment. Amsterdam: John Benjamins.Google Scholar
Brown, P. and Levinson, S. D. (1987). Politeness: Some Universals in Language Usage. Cambridge: Cambridge University Press.Google Scholar
Cao, Y. and O’Halloran, K. L. (2014). Learning Human Photo Shooting Patterns from Large-Scale Community Photo Collections. Multimedia Tools and Applications 24: 11499–516.Google Scholar
Deppermann, A. (2013). Multimodal Interaction from a Conversation Analytic Perspective. Journal of Pragmatics 46(1): 1–7.Google Scholar
Djonov, E. and Zhao, S. (2014). Critical Multimodal Studies of Popular Discourse. New York: Routledge.Google Scholar
Djonov, E. and Zhao, S. (2017). Social Semiotics: A Theorist and a Theory in Retrospect and Prospect. In Zhao, S., Djonov, E., Björkvall, A. and Boeriis, M. (eds.) Advancing Multimodal and Critical Discourse Studies: Interdisciplinary Research Inspired by Theo Van Leeuwen’s Social Semiotics. London/New York: Routledge. 1–18.Google Scholar
Dreyfus, S., Hood, S. and Stenglin, M. (2011). Semiotic Margins: Meaning in Multimodalities. London/New York: Continuum.Google Scholar
Flowerdew, J. and Richardson, J. (2017). Routledge Handbook of Critical Discourse Studies. London: Routledge.CrossRefGoogle Scholar
Forceville, C. (1996). Pictorial Metaphor in Advertising. London: Routledge.Google Scholar
Forceville, C. and Urios-Aparisi, E. (2009). Multimodal Metaphor, Vol. 11. Walter de Gruyter.Google Scholar
Goffman, E. (1967). Interactional Ritual: Essays on Face to Face Behaviour. New York: Garden City.Google Scholar
Goffman, E. (1981). Forms of Talk. Oxford: Blackwell.Google Scholar
Goodwin, C. (2000). Action and Embodiment within Situated Human Interaction. Journal of Pragmatics 32(10): 1489–522.Google Scholar
Hall, E. T. (1968). Proxemics. Current Anthropology 9(2/3): 83–108.Google Scholar
Halliday, M. A. K. (1978). Language as Social Semiotic. London: Edward Arnold.Google Scholar
Halliday, M. A. K. (1994). An Introduction to Functional Grammar, 2nd ed. London: Edward Arnold.Google Scholar
Halliday, M. A. K. and Matthiessen, C. M. I. M. (2014). Halliday’s Introduction to Functional Grammar. London/New York: Routledge.CrossRefGoogle Scholar
Halliday, M. A. K. (2009). Collected Works of M. A. K. Halliday, ed. by Webster, J. J.. Vol. 10. London: Bloomsbury Academic.Google Scholar
Halliday, M. A. K. (2018). Halliday in the 21st Century: Collected Works of M. A. K. Halliday, ed. by Webster, J. J.. Vol. 11. London: Bloomsbury Academic.Google Scholar
Hart, C. (2016). The Visual Basis of Linguistic Meaning and Its Implications for Critical Discourse Studies: Integrating Cognitive Linguiic and Multimodal Methods. Discourse & Society 27(3): 335–50.Google Scholar
Hodge, R. I. V. and Kress, G. (1988). Social Semiotics. Cambridge: Polity Press.Google Scholar
Jewitt, C. (2014). The Routledge Handbook of Multimodal Analysis, 2nd ed. London: Routledge.Google Scholar
Jewitt, C., Bezemer, J. and O’Halloran, K. L. (2016). Introducing Multimodality. London/New York: Routledge.Google Scholar
Jones, C. and Ventola, E. (2008). From Language to Multimodality: New Developments in the Study of Ideational Meaning. London: Equinox.Google Scholar
Kendon, A. (2004). Gesture: Visible Action as Utterance. Cambridge: Cambridge University Press.Google Scholar
Kress, G. (2010) Multimodality: A Social Semiotic Approach to Contemporary Communication. London: Routledge.Google Scholar
Kress, G. and van Leeuwen, T. ([1996]2006). Reading Images: The Grammar of Visual Design. London: Routledge.Google Scholar
Kress, G. and van Leeuwen, T. (2001). Multimodal Discourse: The Modes and Media of Contemporary Communication. London: Arnold.Google Scholar
Kress, G. and van Leeuwen, T. (2002). Colour as a Semiotic Mode: Notes for a Grammar of Colour. Visual Communication 1(3): 343–68.Google Scholar
Kress, G., Jewitt, C. L., Ogborn, J. and Tsatsarelis, C. (2014). Multimodal Teaching and Learning: The Rhetorics of the Science Classroom, 2nd ed. New York: Bloomsbury Academic.Google Scholar
Machin, D. (2007). Introduction to Multimodal Analysis. London: Hodder Arnold.Google Scholar
Machin, D. (2013). What Is Multimodal Critical Discourse Studies? Critical Discourse Studies 10(4): 347–55.Google Scholar
Machin, D. (2016). The Need for a Social and Affordance-Driven Multimodal Critical Discourse Studies. Discourse & Society 27(3): 322–34.Google Scholar
Machin, D. and Mayr, A. (2012). How to Do Critical Discourse Analysis: A Multimodal Introduction. London: Sage.Google Scholar
Machin, D. and Richardson, J. E. (2012). Discourses of Unity and Purpose in the Sounds of Fascist Music: A Multimodal Approach. Critical Discourse Studies 9(4): 329–45.CrossRefGoogle Scholar
Machin, D. and van Leeuwen, T. (2007). Global Media Discourse: A Critical Introduction. New York: Taylor & Francis.Google Scholar
Machin, D. and van Leeuwen, T. (2009). Toy As Discourse: Children’s War Toys and the War on Terror. Critical Discourse Studies 6(1): 51–64.Google Scholar
Martin, J. R. (1992). English Text: System and Structure. Amsterdam: Benjamins.Google Scholar
Martin, J. R. (2002). Meaning beyond the Clause: SFL Perspectives. Annual Review of Applied Linguistics 22: 52–74.Google Scholar
Martin, J. R. and White, P. R. R. (2005). The Language of Evaluation: Appraisal in English. London: Palgrave Macmillan.Google Scholar
Mondada, L. (2014). The Local Constitution of Multimodal Resources for Social Interaction. Journal of Pragmatics 65: 137–56.Google Scholar
Mondada, L. (2016). Challenges of Multimodality: Language and the Body in Social Interaction. Journal of Sociolinguistics 20(3): 336–66.Google Scholar
Norris, S. (2004). Analyzing Multimodal Interaction: A Methodological Framework. London/New York: Routledge.Google Scholar
O’Halloran, K. L. (1999). Towards a Systemic Functional Analysis of Multisemiotic Mathematics Texts. Semiotica 124(1/2): 1–29.Google Scholar
O’Halloran, K. L. (2000). Classroom Discourse in Mathematics: A Multisemiotic Analysis. Linguistics and Education 10(3): 359–88.Google Scholar
O’Halloran, K. L. (2004). Multimodal Discourse Analysis. London/New York: Continuum.Google Scholar
O’Halloran, K. L. (2008). Systemic Functional-Multimodal Discourse Analysis (SF-MDA): Constructing Ideational Meaning Using Language and Visual Imagery. Visual Communication 7: 443–75.Google Scholar
O’Halloran, K. L. (2011). Multimodal Discourse Analysis. In Hyland, K. and Paltridge, B. (eds.) Bloomsbury Companion to Discourse Analysis. London: Bloomsbury. 120–37.Google Scholar
O’Halloran, K. L. (2015). The Language of Learning Mathematics: A Multimodal Perspective. Journal of Mathematical Behaviour 40(Part A): 63–74.Google Scholar
O’Halloran, K. L. and Smith, B. A. (2011). Multimodal Studies: Exploring Issues and Domains. New York/London: Routledge.Google Scholar
O’Halloran, K. L., Chua, A. and Podlasov, A. (2014). The Role of Images in Social Media Analytics: A Multimodal Digital Humanities Approach. In Machin, D. (ed.), Visual Communication. Berlin: Gruyter. 565–88.Google Scholar
O’Halloran, Kay L., Tan, Sabine, and E, Marissa K. L. (2014). Multimodal Pragmatics. In Schneider, K. and Barron, A. (eds.) Pragmatics of Discourse. Berlin: De Gruyter Mouton. 239–68.Google Scholar
O’Halloran, K. L., Tan, S., Pham, D.-S., Bateman, J. and Vande Moere, A. (2018). A Digital Mixed Methods Research Design: Integrating Multimodal Analysis with Data Mining and Information Visualization for Big Data Analytics. Journal of Mixed Methods Research 12(1): 11–30. DOI: 10.1177/1558689816651015.Google Scholar
O’Halloran, K. L., Tan, S. and Wignell, P. (2016). Inter-Semiotic Translation as Resemiotization: A Multimodal Perspective. Signata [Special Issue on Translating: Signs, Texts, Practices] 7(1): 199–229.Google Scholar
O’Halloran, K. L., Tan, S. and Wignell, P. (2019). SFL and Multimodal Discourse Analysis. In Thompson, G., Bowcher, W. L., Fontaine, L. and Schönthal, D. (eds.) The Cambridge Handbook of Systemic Functional Linguistics. Cambridge: Cambridge University Press. 433–61.Google Scholar
O’Halloran, K. L., Tan, S., Wignell, P. and Lange, R. (2017). Multimodal Recontextualisations of Images in Violent Extremist Discourse. In Zhao, S., Djonov, E., Björkvall, A. and Boeriis, M. (eds.) Advancing Multimodal and Critical Discourse Studies: Interdisciplinary Research Inspired by Theo Van Leeuwen’s Social Semiotics. London/New York: Routledge. 181–202.Google Scholar
O’Toole, M. ([1994] 2011). The Language of Displayed Art, 2nd ed. London/New York: Routledge.Google Scholar
Peirce, C. S. (1984). Writings of Charles S. Peirce: A Chronological Edition, Vol. 2: 1867–1871. Bloomington: Indiana University Press.Google Scholar
Pennock-Speck, B. and Del Saz-Rubio, M. M. (2013). A Multimodal Analysis of Facework Strategies in a Corpus of Charity Ads on British Television. Journal of Pragmatics 49(1): 38–56.Google Scholar
Podlasov, A. and O’Halloran, K. L. (2014). Japanese Street Fashion for Young People: A Multimodal Digital Humanities Approach for Identifying Socio-Cultural Patterns and Trends. In Djonov, E. and Zhao, S. (eds.) Critical Multimodal Studies of Popular Culture. New York: Routledge. 71–90.Google Scholar
Saussure, F. de ([1916]/1983). Course in General Linguistics, trans. by Harris, R.. London: Duckworth.Google Scholar
Sacks, H., Schegloff, E. A. and Jefferson, G. (1974). A Simplest Systematics for the Organization of Turn-Taking for Conversation. Language 50(4): 696–735.Google Scholar
Scollon, R. (2001). Mediated Discourse: The Nexus of Practice. London: Routledge.Google Scholar
Scollon, R. and Scollon, S. W. (2003). Discourses in Place: Language in the Material World. London: Routledge.Google Scholar
Scollon, R. and Scollon, S. W. (2004). Nexus Analysis. London: Routledge.Google Scholar
Sindoni, M. G., Wildfeuer, J. and O’Halloran, K. L. (2017). Mapping Multimodal Performance Studies. London/New York: Routledge.Google Scholar
Streeck, J. (2009). Gesturecraft. Amsterdam: Benjamins.CrossRefGoogle Scholar
Tan, S., O’Halloran, K. L. and Wignell, P. (2016). Multimodal Research: Addressing the Complexity of Multimodal Environments and the Challenges for CALL. ReCALL [Special Issue on Multimodal Environments in CALL] 28(3): 253–73.Google Scholar
Tan, S., O’Halloran, K. L., Wignell, P., Chai, K. and Lange, R. (2018). A Multimodal Mixed Methods Approach for Examining Recontextualisation Patterns of Violent Extremist Images in Online Media Discourse, Context & Media 21: 18–35.Google Scholar
Tan, S., Wignell, P. and O’Halloran, K. L. (2016). From Book to Stage to Screen: Semiotic Transformations of Gothic Horror Genre Conventions. Social Semiotics [Special Issue: The Languages of Performing Arts: Semiosis, Communication and Meaning-Making] 26(4): 404–23.Google Scholar
Unsworth, L. (2008). Multimodal Semiotics: Functional Analysis in Contexts of Education. New York: Continuum.Google Scholar
van Leeuwen, T. (1995). Representing Social Action. Discourse and Society 6(1): 81–106.Google Scholar
van Leeuwen, T. (1996). The Representation of Social Actors. In Caldas-Coulthard, C. R. and Coulthard, M. (eds.) Texts and Practices: Readings in Critical Discourse Analysis. London: Routledge. 32–70.Google Scholar
van Leeuwen, T. (1999). Speech, Music, Sound. London: MacMillan.Google Scholar
van Leeuwen, T. (2006). Towards a Semiotics of Typography. Information Design Journal 14(20): 139–55.Google Scholar
van Leeuwen, T. (2008). Discourse and Practice: New Tools for Critical Discourse Analysis. London: Oxford University Press.Google Scholar
van Leeuwen, T. (2012). The Critical Analysis of Musical Discourse. Critical Discourse Studies 9(4): 319–28.Google Scholar
van Leeuwen, T. and Boeriis, M. (2017). Towards a Semiotics of Film Lighting. In Wildfeuer, J. and Bateman, J. A. (eds.) Film Text Analysis: New Perspectives on the Analysis of Filmic Meaning. New York/Abingdon: Routledge. 24–45.Google Scholar
van Leeuwen, T. and Djonov, E. (2015). Notes towards a Semiotics of Kinetic Typography. Social Semiotics 25(2): 244–53.Google Scholar
Wignell, P., Tan, S., O’Halloran, K. L. and Lange, R. (2017). A Mixed Methods Empirical Examination of Changes in Emphasis and Style in the Extremist Magazines Dabiq and Rumiyah. Perspectives on Terrorism 11(2): 2–20.Google Scholar
Wildfeuer, J. (2014). Film Discourse Interpretation: Towards a New Paradigm for Multimodal Film Analysis, Vol. 9. London/New York: Routledge.Google Scholar
Wildfeuer, J. and Bateman, J. A. (2016). Film Text Analysis: New Perspectives on the Analysis of Filmic Meaning, Vol. 50. New York: Taylor & Francis.Google Scholar
Wodak, R. and Meyer, M. (2015). Methods of Critical Discourse Analysis, 3rd ed. London: Sage.Google Scholar

Save book to Kindle

To save this book to your Kindle, first ensure coreplatform@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Available formats
×

Save book to Dropbox

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Dropbox.

Available formats
×

Save book to Google Drive

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Google Drive.

Available formats
×