Hostname: page-component-6d856f89d9-jhxnr Total loading time: 0 Render date: 2024-07-16T08:43:46.210Z Has data issue: false hasContentIssue false
  • English
  • Français

Navigating a multimodal ensemble: Learners mediating verbal and non-verbal turns in online interaction tasks

Published online by Cambridge University Press:  02 October 2019

Janine Knight Open the ORCID record for Janine Knight [Opens in a new window] ,
Melinda Dooly Open the ORCID record for Melinda Dooly [Opens in a new window]  and
Elena Barberà Open the ORCID record for Elena Barberà [Opens in a new window]
Janine Knight*
Affiliation:
Universitat Internacional de Catalunya, Spain Universitat Oberta de Catalunya, Spain (janine@uic.com)
Melinda Dooly
Affiliation:
Universitat Autònoma de Barcelona, Spain (melindaann.dooly@uab.cat)
Elena Barberà
Affiliation:
Universitat Oberta de Catalunya, Spain (ebarbera@uoc.edu)
Get access Rights & Permissions [Opens in a new window]

Abstract

Research into the multimodal aspects of language is increasingly important as communication through a screen plays a greater role in modern society than ever before (Liou, 2011). Multimodality has been explored from a number of angles relating to computer-mediated communication (CMC), such as its affordances and impact on language learners, highlighting its relevance and importance in the field of second language acquisition (SLA). Because CMC scenarios require attending to both peers and the screen, learners can be seen as positioned as “semiotic initiators and responders” (Coffin & Donohue, 2014). Increasingly, researchers are highlighting a need for a methodological “turn” to analyse this scenario from a “language” focus to a more holistic understanding of the interactions (Flewitt, 2008; Hampel & Hauck, 2006; Kress & van Leeuwen, 2001; Lamy, 2006). Along these lines, this case study explores how the action of task completion is mediated between six dyads (and individuals within the dyads) during an online peer-to-peer audioconferencing event. Drawing on notions from multimodal (inter)actional analysis (Norris, 2004, 2006) and the notion of “semiotic initiators and responders”, it investigates semiotic mediation with screen-based resources through analysis of audio recordings, screenshots, log files, task simulation and reconstruction. Results highlight oral and screen-based initiations and responses that take place during task completion, which is presented as a framework.

Keywords

task-based synchronous communication (TB-SCMC)spoken interactionsemiotic mediationmultimodal turn-takingscreen-based resourcesagency
Type
Regular papers
Information
ReCALL , Volume 32 , Issue 1 , January 2020 , pp. 25 - 46
Copyright
© European Association for Computer Assisted Language Learning 2019 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Balaman, U. & Sert, O. (2017) Local contingencies in L2 tasks: A comparison of context-sensitive interactional achievements across two different task types. Bellaterra Journal of Teaching & Learning Language & Literature, 10(3): 9–27. https://doi.org/10.5565/rev/jtl3.746 CrossRefGoogle Scholar
Benson, P. (2015) YouTube as text: Spoken interaction analysis and digital discourse. In Jones, R. H., Chik, A. & Hafner, C. A. (eds.), Discourse and digital practices: Doing discourse analysis in the digital age. Oxford: Routledge, 81–96.Google Scholar
Bezemer, J. & Kress, G. (2014) Touch: A resource for making meaning. Australian Journal of Language and Literacy, 37(2): 77–85.Google Scholar
Bezemer, J. & Kress, G. (2016) Multimodality, learning and communication: A social semiotic frame. London: Routledge. https://doi.org/10.4324/9781315687537 Google Scholar
Canto, S., de Graaff, R. & Jauregi, K. (2014) Collaborative tasks for negotiation of intercultural meaning in virtual worlds and video-web communication. In González-Lloret, M. & Ortega, L. (eds.), Technology-mediated TBLT: Researching technology and tasks. Washington: Georgetown University Press, 183–212. https://doi.org/10.1075/tblt.6.07can CrossRefGoogle Scholar
Clark, H. H. & Brennan, S. E. (1991) Grounding in communication. In Resnick, L. B., Levine, J. M. & Teasley, S. D. (eds.), Perspectives on socially shared cognition. Washington: American Psychological Association, 127–149. https://doi.org/10.1037/10096-006 CrossRefGoogle Scholar
Coffin, C. & Donohue, J. (2014) A language as social semiotic-based approach to teaching and learning in higher education (Language Learning Monograph Series). Hoboken: Wiley-Blackwell. https://doi.org/10.1111/lang.2014.64.issue-s1 Google Scholar
Collentine, K. (2009) Learner use of holistic language units in multimodal, task-based synchronous computer-mediated communication. Language Learning & Technology, 13(2): 68–87.Google Scholar
Dooly, M. (2018) “I do which the question”: Students’ innovative use of technology resources in the language classroom. Language Learning & Technology, 22(1): 184–217.Google Scholar
Dourish, P., Bellotti, V., Mackay, W. & Ma, C.-Y. (1993) Information and context: Lessons from the study of two shared information systems. In Proceedings of the Conference on Organizational Computing Systems (pp. 42–51). Milpitas, CA, 1–4 November. https://doi.org/10.1145/168555.168560 CrossRefGoogle Scholar
Flewitt, R. (2008) Multimodal literacies. In Marsh, J. & Hallet, E. (eds.), Desirable literacies: Approaches to language and literacy in the early years (2nd ed.). London: SAGE, 122–139. https://doi.org/10.4135/9781446279519.n7 CrossRefGoogle Scholar
Flewitt, R., Hampel, R., Hauck, M. & Lancaster, L. (2013) What are multimodal data and transcription? In Jewitt, C. (ed.), The Routledge Handbook of Multimodal Analysis (2nd ed.). London: Routledge, 44–59.Google Scholar
Gilabert, R., Manchón, R. & Vasylets, O. (2016) Mode in theoretical and empirical TBLT research: Advancing research agendas. Annual Review of Applied Linguistics, 36: 117–135. https://doi.org/10.1017/S0267190515000112 CrossRefGoogle Scholar
Goodwin, C. (2013) The co-operative, transformative organization of human action and knowledge. Journal of Pragmatics, 46(1): 8–23. https://doi.org/10.1016/j.pragma.2012.09.003 CrossRefGoogle Scholar
Hampel, R. & Hauck, M. (2006) Computer-mediated language learning: Making meaning in multimodal virtual learning spaces. The JALT CALL Journal, 2(2): 3–18.Google Scholar
Hampel, R. (2010) Task design for a virtual learning environment in a distant language course. In Thomas, M. & Reinders, H. (eds.), Task-based language learning and teaching with technology. London: Continuum, 131–153.Google Scholar
Hampel, R. & Stickler, U. (2012) The use of videoconferencing to support multimodal interaction in an online language classroom. ReCALL, 24(2): 116–137. https://doi.org/10.1017/S095834401200002X CrossRefGoogle Scholar
Hauck, M. (2010) The enactment of task design in telecollaboration 2.0. In Thomas, M. & Reinders, H. (eds.), Task-based language learning and teaching with technology. London: Continuum, 197–217.Google Scholar
Helm, F. & Dooly, M. (2017) Challenges in transcribing multimodal data: A case study. Language Learning & Technology, 21(1): 166–185. https://dx.doi.org/10125/44600 Google Scholar
Herring, S. C. (2015) New frontiers in interactive multimodal communication. In Georgakopoulou, A. & Spilioti, T. (eds.), The Routledge handbook of language and digital communication. New York: Routledge, 398–402.Google Scholar
Jenks, C. J. (2014) Social interaction in second language chat rooms. Edinburgh University Press.Google Scholar
Jewitt, C., Bezemer, J. & O’Halloran, K. (2016) Introducing multimodality. London: Routledge.CrossRefGoogle Scholar
Knight, J. & Barberà, E. (2016) The negotiation of shared and personal meaning making in spoken interaction tasks. In Pixel, (ed.), Proceedings of the 9th International Conference on ICT for Language Learning (pp. 248–252). Florence, Italy, 17–18 November. https://conference.pixel-online.net/ICT4LL/acceptedabstracts_scheda.php?id_abs=1994 Google Scholar
Kraut, R. E., Fussell, S. R. & Siegel, J. (2003) Visual information as a conversational resource in collaborative physical tasks. Human-Computer Interaction, 18(1-2): 13–49. https://doi.org/10.1207/S15327051HCI1812_2 CrossRefGoogle Scholar
Kress, G. (2003) Literacy in the new media age. London: Routledge. https://doi.org/10.4324/9780203299234 Google Scholar
Kress, G. & van Leeuwen, T. (2001) Multimodal discourse: The modes and media of contemporary communication. London: Edward Arnold.Google Scholar
Lantolf, J. P. (ed.) (2000) Sociocultural theory and second language learning. Oxford: Oxford University Press.Google Scholar
Lamy, M.-N. (2006) Multimodality in second language conversations online: Looking for a methodology. In Proceedings of the Third International Conference on Multimodality. Pavia, Italy, 385–403.Google Scholar
Liddicoat, A. J. (2010) Enacting participation: Hybrid modalities in on-line video conversation. In Develotte, C., Kern, R. & Lamy, M.-N. (eds.), Décrire la conversation en ligne: Le face à face distanciel. Lyon: ENS Éditions, 37–50. http://catalogue-editions.ens-lyon.fr/fr/livre/?GCOI=29021100952500 Google Scholar
Liou, H.-C. (2011) Blogging, collaborative writing, and multimodal literacy in an EFL context. WorldCALL: International perspectives on computer-assisted language learning. New York: Routledge, 3–18.Google Scholar
Mondada, L. (2007) Multimodal resources for turn-taking: Pointing and the emergence of possible next speakers. Discourse Studies, 9(2): 194–225. https://doi.org/10.1177/1461445607075346 CrossRefGoogle Scholar
Mondada, L. (2013) Embodied and spatial resources for turn-taking in institutional multi-party interactions: Participatory democracy debates. Journal of Pragmatics, 46(1): 39–68.CrossRefGoogle Scholar
Norris, S. (2004) Analyzing multimodal interaction: A methodological framework. London: Routledge. https://doi.org/10.4324/9780203379493 CrossRefGoogle Scholar
Norris, S. (2006) Multiparty interaction: A multimodal perspective on relevance. Discourse Studies, 8(3): 401–421. https://doi.org/10.1177/1461445606061878 CrossRefGoogle Scholar
Norris, S. (2016) Concepts in multimodal discourse analysis with examples from video conferencing. Yearbook of the Poznan Linguistic Meeting, 2(1): 141–165. https://doi.org/10.1515/yplm-2016-0007 CrossRefGoogle Scholar
Oben, B. & Brône, G. (2015) What you see is what you do: On the relationship between gaze and gesture in multimodal alignment. Language and Cognition, 7(4): 546–562. https://doi.org/10.1017/langcog.2015.22 CrossRefGoogle Scholar
Raudaskoski, P. (1999) The use of communicative resources in language technology environments: A conversation analytic approach to semiosis at computer media. University of Oulu, unpublished PhD. http://vbn.aau.dk/ws/files/72280035/pirkkosphd.pdf Google Scholar
Rossolatos, G. (2015) Taking the “multimodal turn” in interpreting consumption experiences. Consumption Markets & Culture, 18(5): 427–446. https://doi.org/10.1080/10253866.2015.1056167 CrossRefGoogle Scholar
Sacks, H., Schegloff, E. A. & Jefferson, G. (1974) A simplest systematics for the organization of turn-taking for conversation. Language, 50(4): 696–735.CrossRefGoogle Scholar
Schnotz, W. (1999) Introduction. European Journal of Psychology of Education, 14(2): 163–165. https://doi.org/10.1007/BF03172963 CrossRefGoogle Scholar
Tudini, V. (2014) Conversation analysis of computer-mediated interactions. In Chapelle, C. (ed.), The encyclopedia of applied linguistics. Hoboken: John Wiley & Sons, 1–7. https://doi.org/10.1002/9781405198431.wbeal1456 Google Scholar
van Lier, L. (2000) From input to affordance: Social-interactive learning from an ecological perspective. In Lantolf, J. (ed.), Sociocultural theory and second language learning. Oxford: Oxford University Press, 245–259.Google Scholar
Vetter, A. & Chanier, T. (2006) Supporting oral production for professional purposes in synchronous communication with heterogenous learners. ReCALL, 18(1): 5–23. https://doi.org/10.1017/S0958344006000218 CrossRefGoogle Scholar
Vygotsky, L. S. (1981) The instrumental method in psychology. In Wertsch, J. V. (ed.), The concept of activity in Soviet psychology. Armonk: Sharpe, 134–143.Google Scholar
Wertsch, J. V. (1994) The primacy of mediated action in sociocultural studies. Mind, Culture, and Activity, 1(4): 202–208.Google Scholar
Wertsch, J. V. (1998) Mind as action. Oxford: Oxford University Press. https://doi.org/10.1093/acprof:oso/9780195117530.001.0001 Google Scholar
Yamada, M. (2009) The role of social presence in learner-centered communicative language learning using synchronous computer-mediated communication: Experimental study. Computers & Education, 52(4): 820–833. https://doi.org/10.1016/j.compedu.2008.12.007 CrossRefGoogle Scholar