Multimodality and evaluation of machine translation: a proposal for investigating intersemiotic mismatches generated by the use of machine translation in multimodal documents

Thiago Blanch Pires

doi:10.17851/1983-3652.11.1.82-102

Authors

Thiago Blanch Pires Universidade de Brasília (UnB)

DOI:

https://doi.org/10.17851/1983-3652.11.1.82-102

Keywords:

multimodality, machine translation, evaluation of machine translation, intersemiotic mismatches

Abstract

This article aims at proposing an interdisciplinary approach involving the areas of Multimodality and Evaluation of Machine Translation to explore new configurations of text-image semantic relations generated by machine translation results. The methodology consists of a brief contextualization of the research problem, followed by the presentation and study of concepts and possibilities of Multimodality and Evaluation of Machine Translation, with an emphasis on the notion of intersemiotic texture, proposed by Liu and O'Halloran (2009), and a study of machine translation error classification, proposed by Vilar et. al. (2006). Finally, the article suggests some potentialities and limitations when combining the application of both areas of investigation.

Author Biography

Thiago Blanch Pires, Universidade de Brasília (UnB)

Professor adjunto no curso bacharelado de Línguas Estrangeiras Aplicadas ao Multilinguismo e à Sociedade da Informação (LEA-MSI) do Departamento de Línguas Estrangeiras e Tradução (LET), vinculado ao Instituto de Letras (IL) da Universidade de Brasília (UnB). Atua no ensino da língua inglesa, dos estudos de corpora, e do tratamento automatizado das línguas naturais. Doutor em Gestão da Informação pelo Programa de Pós-Graduação em Ciência da Informação (PPGCinf) da Universidade de Brasília, com interesse de estudo nas áreas de Organização da Informação e do Conhecimento, Linguística Computacional, e Multimodalidade. Possui mestrado em Letras - Língua Inglesa e Literaturas pelo Programa de Pós-Graduação em Inglês (PPGI) da Universidade Federal de Santa Catarina, tendo como foco a Análise Textual na interface dos campos dos Estudos de Tradução baseados em Corpus e da Linguística Sistêmico-Funcional.

References

BATEMAN, J. A. Multimodality and genre: a foundation for the systematic analysis of multimodal documents. New York: Palgrave MacMillan, 2008.

BATEMAN, J. A. Text and image: a critical introduction to the visual/verbal divide. New York: Routledge, 2014.

BAZERMAN, C. Systems of genres and the enactment of social intentions. In: Genre and the new rhetoric. London: Taylor and Francis, 1994. p. 79–104.

DE BEAUGRANDE, R. A.; DRESSLER, W. U. Einfurung in die Textinguistik. Tubingen: Niemeyer, 1981.

FAIRCLOUGH, N. Discourse and social change. Cambridge: Polity press, 1992.

HALLIDAY, M. A. K. Language as social semiotic: towards a general socialingsocial theory. Columbia: Hornbeam Press, 1975.

HALLIDAY, M. A. K. Language as social semiotic: the social interpretation of language and meaning. London: Arnold, 1978.

HALLIDAY, M. A. K. Introduction to functional grammar. Londres: Arnold, 1985.

HALLIDAY, M. A. K. Introduction to functional grammar. 2. ed. Londres: Arnold, 1994.

HALLIDAY, M. A. K.; HASAN, R. Cohesion in English. London: Longman, 1976.

HALLIDAY, M. A. K.; MATTHIESSEN, C. An introduction to functional grammar. [s.l.] Arnold, 2004.

IEDEMA, R. Multimodality, resemiotization: Extending the analysis of discourse as multisemiotic practice. Visual Communication, v. 2, n. 1, p. 29–57, 2003.

JEWITT, C. The Routledge handbook of multimodal analysis. London: Routledge, 2009.

JONES, J. 2006. Multiliteracies for academic purposes: a metafunctional exploration of intersemiosis and multimodality in university textbook and computer-based learning resources in science. Doctoral thesis, University of Sidney , Sidney, Australia, 2006.

KAMEYAMA, M.; OCHITANI, R.; PETERS, S. Resolving translation mismatches with information flow. In: ANNUAL MEETING OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS ACL91, 1991. Proceedings of the Annual Meeting of the Association for Computational Linguistics ACL91. 1991.

KALTENBACHER, M. Perspectives on multimodality: From the early beginnings to the state of the art. Information Design Journal, v. 12, n. 3, p. 190–207, 2004.

KRESS, G.; VAN LEEUWEN, T. Reading images: the grammar of visual design. 1st. ed. London: Routledge, 1996.

KRESS, G.; VAN LEEUWEN, T. Multimodal discourse: the modes and media of contemporary communication. London: Edward Arnold, 2001.

KRESS, G.; VAN LEEUWEN, T. Reading images: the grammar of visual design. 2nd. ed. London: Routledge, 2006.

LEMKE, J. Multiplying Meaning: Visual and Verbal Semiotics in Scientific Text. In: Reading science. Londres: Routledge, 1998. p. 87–113.

LEMKE, J. Typology, topology, topography: genre semantics. MS. University of Michigan, 1999. Disponível em: http://academic.brooklyn.cuny.edu/education/jlemke/papers/Genre-topology-revised.htm. Acesso em: 7 abr. 2018.

LIU, Y.; O’HALLORAN, K. L. Intersemiotic texture: analyzing cohesive devices between language and images. Social Semiotics, v. 19, n. 4, p. 367–388, 2009.

LLITJ´OS, A. F.; CARBONELL, J. G.; LAVIE, A. A framework for interactive and automatic refinement of transfer-based machine translation. In: ANNUAL CONFERENCE OF THE EUROPEAN ASSOCIATION FOR MACHINE TRANSLATION (EAMT). 2005, Budapeste. Proc. of the 10th Annual Conf. of the European Association for Machine Translation (EAMT), 2005.

MARTIN, J. R. English text: system and structure. [s.l.] John Benjamins Pub. Co, 1992.

MARTINEC, R.; SALWAY, A. A system for image-text relations in new (and old) media. Visual Communication, v. 4, n. 3, p. 337–371, 2005.

O’HALLORAN, K. L. Mathematical discourse: language, symbolism and visual images. London: Continuum, 2005.

O’TOOLE, M. The language of displayed art. London: Leicester University Press, 1994.

PIRES, T. B.; DUQUE, C. G. Sistemas de gerenciamento de tradução: uma proposta de análise multimodal. DataGramaZero, v. 16, n. 3, 2015.

PIRES, T. B. Ampliando olhares sobre a tradução automática online: um estudo exploratório de categorias de erros de máquina de tradução gerados em documentos multimodais. 2017. 169f. Doctoral thesis – Programa de Pós-Graduação em Ciência da Informação, Unb, Brasília, 2017.

ROYCE, T. Synergy on the Page: Exploring intersemiotic complementarity in page-based multimodal text. JASFL Occasional papers, v. 1, n. 1, p. 25–49, 1998.

ROYCE, T. New directions in the analysis of multimodal discourse. In: New directions in the analysis of multimodal discourse. New York: Routledge, 2007. p. 63–109.

SANTOS, D.; MAIA, B.; SARMENTO, L. Gathering empirical data to evaluate MT from English to Portuguese. In: WORKSHOP ON THE AMAZING UTILITY OF PARALLEL AND COMPARABLE CORPORA. 2004, Lisboa, Portugal. Proceedings of LREC, Lisboa, 2004.

SARMENTO, L. Ferramentas para experimentação, recolha e avaliação de exemplos de tradução automática. In: Avaliação conjunta: um novo paradigma no processamento computacional da língua portuguesa. Lisboa, Portugal: IST press, 2007. p. 193–203.

VILAR, D. et al. Error analysis of statistical machine translation output. In: LREC, 2006, Genoa. Proceedings of LREC. Genoa, 2006.

WHITE, J. S. How to evaluate machine translation. In: Computers and Translation: A translator’s guide. Amsterdam/Philadelphia: John Benjamins Publishing, v. 35, p. 211–244, 2003