Multimodal Frame Semantics
Expanding the Analytical Categories of FrameNet Brasil Multimodal Datasets
DOI:
https://doi.org/10.35699/2317-2096.2025.57570Palavras-chave:
deixis, joint attention, multimodality, Frame Semantics, filmic narrativeResumo
In this paper, I propose an extension of the joint attention scenes models to include mediated multimodal communicative settings. From this extension, I discuss how the FrameNet analytical model can be used for the annotation of multimodal genres so as to include categories capable of accounting for invited shifts in joint attention in visual narratives. I revisit the literature on deixis and claim that, despite the fact that much of the foundational research on this topic proposes correlations between deixis in verbal language and other semiotic modes, the focus of research lies on the linguistic material in which the expression of deixis is grounded. By bringing together contributions from human communication studies, from the analysis of deixis in narratives and from Frame Semantics, I claim that semiotic devices of different types can be mobilized in filmic narratives to invite viewers to promote shifts in the center of the joint attention scene. To illustrate the proposed typology, I provide example analyses of deictic center shifts in films and describe the annotation methodology incorporated to the FrameNet model to account for them.
Referências
BAKHTIN, M. M. Esthétique de la création verbale. Paris: Gallimard, 1984.
BATEMAN, J. Multimodality and Genre: A Foundation for the Systematic Analysis of Multimodal Documents. Berlin: Springer, 2008.
BATEMAN, J.; WILDFEUER, J.; HIIPALA, T. Multimodality: Foundations, Research and Analysis – A Problem-Oriented Introduction. Berlin: Walter de Gruyter GmbH & Co, 2017.
BELCAVELLO, F. FrameNet Annotation for Multimodal Corpora: Devising a Methodology for the Semantic Representation of Text-Image Interactions in Audiovisual Productions. 2023. 134 f. Thesis (Doctorate in Linguistics) – Faculdade de Letras, Universidade Federal de Juiz de Fora, Juiz de Fora, 2023.
BELCAVELLO, F.; VIRIDIANO, M.; COSTA, A. D. da; MATOS, E. E. da S.; TORRENT, T. T. Frame-Based Annotation of Multimodal Corpora: Tracking (A) Synchronies in Meaning Construction. In: INTERNATIONAL FRAMENET WORKSHOP: TOWARDS A GLOBAL, MULTILINGUAL FRAMENET, 2020, Marseille. Proceedings […]. Marseille: European Language Resources Association (ELRA), 2020. p. 23-30.
BELCAVELLO, F.; VIRIDIANO, M.; MATOS, E.; TORRENT, T. T. Charon: A FrameNet Annotation Tool for Multimodal Corpora. In: LINGUISTIC ANNOTATION WORKSHOP (LAW-XVI), 16. Marseille. Proceedings […]. Workshop presented at LREC2022. Marseille: ELRA, 2022. p. 91-96.
BORDWELL, D.; THOMPSON, K. A arte do cinema: uma introdução. Tradução: Roberta Gregoli. Campinas, SP: Editora da Unicamp; São Paulo: Editora da USP, 2013.
BÜHLER, K. Sprachtheorie. Jena: Fischer, 1934.
CIENKI, A. The Study of Gesture in Cognitive Linguistics: How It Could Inform and Inspire Other Research in Cognitive Science. Cognitive Science, v. 13, n. 6, e1623, 2022.
COHN, N. A Multimodal Parallel Architecture: A Cognitive Framework for Multimodal Interactions. Cognition, v. 146, p. 304-323, 2016a.
COHN, N. From Visual Narrative Grammar to Filmic Narrative Grammar: The Narrative Structure of Static and Moving Images. In: WILDFEUER, J.; BATEMAN, J. (ed.). Film Text Analysis: New Perspectives on the Analysis of Filmic Meaning. London: Routledge, 2016b. p. 94-117.
COHN, N. Visual Narrative Structure. Cognitive Science, v. 37, n. 3, p. 413-452, 2013.
DANNÉLLS, D.; TORRENT, T. T.; SIGILIANO, N. S.; DOBNIK, S. Beyond Strings of Characters: Resources Meet NLP – Again. In: VOLODINA, E.; DANNÉLLS, D.; BERDICEVSKIS, A.; FORSBERG, M.; VIRK, S. (ed.). Live and Learn: Festschrift in Honor of Lars Borin. Gothenburg: Institutionen för Svenska, Flerspråkighet och Språkteknologi – Göteborgs Universitet, 2022. p. 29-36.
DUTRA, L. V.; SIGILIANO, N. S. Ferramenta linguístico-computacional como facilitadora para o ensino de gramática na escola. In: SIMPÓSIO BRASILEIRO DE TECNOLOGIA DA INFORMAÇÃO E DA LINGUAGEM HUMANA (STIL), 2021, [S. l.]. Anais […]. [S. l.]: SBC, 2021. p. 432-436.
FILLMORE, C. J. Frame Semantics and the Nature of Languages. In: ANNALS OF THE NEW YORK ACADEMY OF SCIENCES. Conference on the Origin and Development of Language and Speech, v. 280, p. 20-32, 1976. Work presented at the Conference on the Origin and Development of Language and Speech, 1976, New York.
FILLMORE, C. J. Frame Semantics. In: THE LINGUISTIC SOCIETY OF KOREA (ed.). Linguistics in the Morning Calm. Seoul: Hanshin Publishing Company, 1982. p. 111-137.
FILLMORE, C. J. Frames and the Semantics of Understanding. Quaderni di Semantica, v. 6, n. 2,
p. 222-254, 1985.
FILLMORE, C. J. Santa Cruz Lectures on Deixis, 1971. Bloomington, IN: Indiana University Linguistics Club, 1975.
FILLMORE, C. J. The Case for Case Reopened. In: COLE, P.; SADOCK, J. M. (ed.). Syntax and Semantics: Grammatical Relations. New York: Academic Press, 1977. v. 8, p. 59-81.
FILLMORE, C. J. The Merging of Frames. In: ROSSINI FAVRETTI, R. (ed.). Frames, Corpora, and Knowledge Representation. Bologna: Bononia University Press, 2008. p. 1-12.
FILLMORE, C. J.; PETRUCK, M. R.; RUPPENHOFER, J.; WRIGHT, A. FrameNet in Action: The Case of Attaching. International Journal of Lexicography, v. 16, n. 3, p. 297-332, 2003.
FRAMENET BRASIL LAB. WebTool. c2014-2025. Available at: http://webtool.frame.net.br/. Accessed on: 31 Jan. 2025.
GOLDIN-MEADOW, S. Pointing Sets the Stage for Learning Language – and Creating Language. Child Development, v. 78, n. 3, p. 741-745, 2007.
GRICE, H. P. Meaning. The Philosophical Review, v. 66, n. 3, p. 377-388, 1957.
HANKS, W. F. Língua como prática social: das relações entre língua, cultura e sociedade a partir de Bourdieu e Bakhtin. Textos selecionados por Hanks, traduzidos por Anna Christina Bentes, Marco Antônio Rosa Machado, Marcos Rogério Cintra e Renato C. Rezende. São Paulo: Cortez, 2008.
JACKENDOFF, R. Foundations of Language. Oxford: Oxford University Press, 2002.
JACKENDOFF, R.; LERDAHL, F. The Capacity for Music: What is It, and What’s Special About It? Cognition, v. 100, p. 33-72, 2006.
KALANTZIS, M.; COPE, B.; PINHEIRO, P. Letramentos. Campinas: Unicamp, 2020.
KITA, S. Pointing: Where Language, Culture and Cognition Meet. Mahwah, NJ:
Lawrence Erlbaum, 2003.
LEMKE, J. Multiplying Meaning. In: MARTIN, J. R.; VEEL, R. (ed.). Reading Science: Critical and Functional Perspectives on Discourses of Science. London: Routledge, 1998. p. 87-114.
LEVINSON, S. C. Deixis. In: HORN, L. R.; WARD, G. (ed.). The Handbook of Pragmatics. Malden, MA: Blackwell Publishing, 2004. p. 97-121.
LISZKOWSKI, U.; BROWN, P.; CALLAGHAN, T.; TAKADA, A.; DE VOS, C. A. Prelinguistic Gestural Universal of Human Communication. Cognitive Science, v. 36, p. 698-713, 2012.
RODRÍGUEZ, C.; MORENO-NÚNEZ, A.; BASILIO, M.; SOSA, N. Ostensive Gestures Come First: Their Role in the Beginning of Shared Reference. Cognitive Development, v. 36, p. 142-149, 2015.
SAUSSURE, F de. Cours de linguistique générale. Paris: Payot, 1916.
SEARLE, J. R. Collective Intentions and Actions. In: COHEN, O.; MORGAN, J.; POLLACK, M. (ed.). Intentions in Communication. Cambridge, MA: MIT Press, 1990.
SEGAL, E. M. Narrative Comprehension and the Role of Deictic Shift Theory. In: DUCHAN, J.; BRUDER, G. A.; HEWITT, L. E. (ed.). Deixis in Narrative: A Cognitive Science Perspective. London: Psychology Press, 1995. p. 3-18.
STEEDMAN, M. J. Plans, Affordances, and Combinatory Grammar. Linguistics and Philosophy, v. 25, n. 5-6, p. 723-753, 2002.
STEEN, F.; HOUGAARD, A.; JOO, J.; OLZA, I.; CÁNOVAS, C.; PLESHAKOVA, A.; RAY, S.; UHRIG, P.; VALENZUELA, J.; WOŹNY, J.; TURNER, M. Toward an Infrastructure for Data-Driven Multimodal Communication Research. Linguistics Vanguard, v. 4, n. 1, e20170041, 2018.
SWEETSER, E. Looking at Space to Study Mental Spaces: Co-Speech Gesture as a Crucial Data Source in Cognitive Linguistics. In: GONZALEZ-MARQUEZ, M.; SPIVEY, M. J.; COULSON, S.; MITTELBERG, I. (ed.). Methods in Cognitive Linguistics. Amsterdam: John Benjamins, 2007. p. 201-224.
THE NEIGHBORS Window. Written and directed by Marshall Curry. New York: Marshall Curry Productions, 2019. Available at: http://www.theneighborswindow.com/. Accessed on: 31 Jan. 2025.
THE NEW LONDON GROUP. A Pedagogy of Multiliteracies: Designing Social Futures. Harvard Educational Review, v. 66, n. 1, p. 60-93, 1996.
TOMASELLO, M. Becoming Human: A Theory of Ontogeny. Cambridge, MA: The Belknap Press of Harvard University Press, 2018.
TOMASELLO, M. Constructing a Language: A Usage-Based Theory of Language Acquisition. Cambridge, MA: Harvard University Press, 2003.
TOMASELLO, M. Origins of Human Communication. Cambridge, MA: MIT Press, 2008.
TOMASELLO, M. The Cultural Origins of Human Cognition. Cambridge, MA: MIT Press, 1999.
TOMASELLO, M.; CARPENTER, M.; LISZKOWSKI, U. A New Look at Infant Pointing. Child Development, v. 78, n. 3, p. 705-722, 2007.
TORRENT, T. T.; MATOS, E. E. S.; BELCAVELLO, F.; VIRIDIANO, M.; GAMONAL, M. A.; COSTA, A. D. da; MARIM, M. C. Representing Context in FrameNet: A Multidimensional, Multimodal Approach. Frontiers in Psychology, v. 13, e838441, 2022.
TURNER, M. Multimodal Form-Meaning Pairs for Blended Classic Joint Attention. Linguistics Vanguard, v. 3, n. s1, e20160043, 2017.
TURNER, M. The Role of Creativity in Multimodal Construction Grammar. Zeitschrift für Anglistik und Amerikanistik, v. 66, n. 3, p. 357-370, 2018.
VIRIDIANO, M.; TORRENT, T. T.; CZULO, O.; LORENZI, A.; MATOS, E.; BELCAVELLO, F. The Case for Perspective in Multimodal Datasets. In: WORKSHOP ON PERSPECTIVIST APPROACHES TO NLP, 1., 2022, Marseille. Proceedings […]. Workshop presented at LREC2022. Marseille: European Language Resources Association, 2022. p. 108-116.
WILKINS, D. Why Pointing with the Index Finger is not a Universal (in Sociocultural and Semiotic Terms). In: KITA, S. (ed.). Pointing: Where Language, Culture and Cognition Meet. Mahwah, NJ: Lawrence Erlbaum, 2003. p. 171-216.
WITTGENSTEIN, L. Philosophische Untersuchungen. London: Kegan Paul, 1953.
ZUBIN, D. A.; HEWITT, L. E. The Deictic Center: A Theory of Deixis in Narrative. In: DUCHAN J. F.; BRUDER G. A.; HEWITT, L. E. (ed.). Deixis in Narrative: A Cognitive Science Perspective. New York: Routledge: Taylor & Francis Group, 1995. p. 129-155.
Downloads
Publicado
Edição
Seção
Licença
Copyright (c) 2025 Natália Sathler Sigiliano
Este trabalho está licenciado sob uma licença Creative Commons Attribution 4.0 International License.
