Treinamento e análise de um modelo de tradução automática baseado em Transformer

Clóvis Henrique Martins Pimentel; Thiago Blanch Pires

doi:10.1590/1983-3652.2024.49118

Authors

Clóvis Henrique Martins Pimentel Universidade de Brasília, Brasília, DF, Brasil https://orcid.org/0009-0008-7049-7688
Thiago Blanch Pires Universidade de Brasília, Brasília, DF, Brasil https://orcid.org/0000-0002-0060-6075

DOI:

https://doi.org/10.1590/1983-3652.2024.49118

Keywords:

Computational Linguistics, Machine translation, Transformer, Parallel corpus, Machine translation evaluation

Abstract

The objective of this work is to analyze transformer-based machine translation models. It aims to test the feasibility of using trained models based on specialized corpus. For the training of such model, a parallel English-French corpus was built with seven texts related to the Convention of 25 October 1980 on the Civil Aspects of International Child Abduction. The translation results obtained by the trained model were compared with those produced by Google Translate. For the evaluation stage, sacreBLEU automatic evaluation and human evaluation methods were used. The outcome of the automatic evaluation of sentences produced by the trained model was, on average, higher than those generated by the non-trained model. The human evaluation of the sentences revealed that there were adequacy errors in the use of language specific to the subject matter of the 1980 Hague Convention both in sentences generated by the trained model and in sentences generated by the Google Translate model.

Downloads

Download data is not yet available.

Author Biography

Thiago Blanch Pires, Universidade de Brasília, Brasília, DF, Brasil

Professor adjunto do Departamento de Línguas Estrangeiras e Tradução (LET) e do Programa de Pós-Graduação em Linguística (PPGL) vinculados ao Instituto de Letras (IL) da Universidade de Brasília (UnB). Foi coordenador do bacharelado em Línguas Estrangeiras Aplicadas ao Multilinguismo e à Sociedade da Informação (LEA-MSI) de março de 2018 a março de 2020. Atua no ensino da língua inglesa, nos estudos de corpora, e no tratamento automatizado das línguas naturais. Doutor em Gestão da Informação pelo Programa de Pós-Graduação em Ciência da Informação (PPGCinf) da Universidade de Brasília, com período de bolsa doutorado-sanduíche da CAPES na Universität Bremen, Alemanha, sob supervisão do prof. dr. John Bateman. Possui interesse de estudo em Linguística Computacional, Linguística de Corpus, e mais especificamente na construção de ferramentas e recursos para as línguas indígenas brasileiras. Possui graduação e mestrado em Letras - Língua Inglesa e Literaturas pela Universidade Federal de Santa Catarina, sendo o mestrado realizado no Programa de Pós-Graduação em Inglês (PPGI), tendo como foco a Análise Textual na interface dos campos dos Estudos de Tradução baseados em Corpus e da Linguística Sistêmico-Funcional.

References

BANITZ, Brita. Machine translation: a critical look at the performance of rule-based and statistical machine translation. Cadernos de Tradução, v. 40, n. 1, p. 54–71, jan. 2020. ISSN 2175-7968. DOI: 10.5007/2175-7968.2020v40n1p54. Disponível em: https://periodicos.ufsc.br/index.php/traducao/article/view/2175-7968.2020v40n1p54. Acesso em: 22 mar. 2024.

COSTA, Cynthia Beatrice; SILVA, Igor A. Lourenço Da. On the Translation of Literature as a Human Activity par Excellence: Ethical Implications for Literary Machine Translation. Aletria: Revista de Estudos de Literatura, v. 30, n. 4, p. 225–248, dez. 2020. ISSN 2317-2096, 1679-3749. DOI: 10.35699/2317-2096.2020.22047. Disponível em: https://periodicos.ufmg.br/index.php/aletria/article/view/22047. Acesso em: 22 mar. 2024.

DOUGLAS, Arnold; BALKAN, L.; LEE HUMPHREYS, R.; MEIJER, S.; SADLER, L. Machine Translation: An Introductory Guide. [S. l.]: NCC Blackwell, 1994. ISBN 9781855542464.

FREITAG, Markus; GRANGIER, David; CASWELL, Isaac. BLEU might be Guilty but References are not Innocent. In: WEBBER, Bonnie; COHN, Trevor; HE, Yulan; LIU, Yang (ed.). Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Online: Association for Computational Linguistics, nov. 2020. p. 61–71. DOI: 10.18653/v1/2020.emnlp-main.5. Disponível em: https://aclanthology.org/2020.emnlp-main.5. Acesso em: 22 mar. 2024.

FREITAS, Cláudia. Linguística Computacional. São Paulo: Parábola Editorial, 2022.

HCCH. Convention of 25 October 1980 on the Civil Aspects of International Child Abduction. [S. l.: s. n.], 1980. Disponível em: https://assets.hcch.net/docs/e86d9f72-dc8d-46f3-b3bf-e102911c8532.pdf. Acesso em: 22 mar. 2024.

HCCH. Guide to Good Practice under the Hague Convention of 25 October 1980 on the Civil Aspects of International Child Abduction: Part I – Central Authority Practice. [S. l.: s. n.], 2003. Disponível em: https://www.hcch.net/en/instruments/conventions/publications1/?dtid=3&cid=24. Acesso em: 22 mar. 2024.

HCCH. Guide to Good Practice under the Hague Convention of 25 October 1980 on the Civil Aspects of International Child Abduction: Part II – Implementing Measures. [S. l.: s. n.], 2003. Disponível em: https://www.hcch.net/en/instruments/conventions/publications1/?dtid=3&cid=24. Acesso em: 12 jul. 2023.

HCCH. Guide to Good Practice under the Hague Convention of 25 October 1980 on the Civil Aspects of International Child Abduction: Part III - Preventive Measures. [S. l.: s. n.], 2005. Disponível em: https://www.hcch.net/en/instruments/conventions/publications1/?dtid=3&cid=24. Acesso em: 12 jul. 2023.

HCCH. Guide to Good Practice under the Hague Convention of 25 October 1980 on the Civil Aspects of International Child Abduction: Part IV - Enforcement. [S. l.: s. n.], 2010. Disponível em: https://www.hcch.net/en/instruments/conventions/publications1/?dtid=3&cid=24. Acesso em: 12 jul. 2023.

HCCH. Guide to Good Practice under the Hague Convention of 25 October 1980 on the Civil Aspects of International Child Abduction: Part V - Mediation. [S. l.: s. n.], 2012. Disponível em: https://www.hcch.net/en/instruments/conventions/publications1/?dtid=3&cid=24. Acesso em: 12 jul. 2023.

IOSIFOVA, Olena; IOSIFOV, Ievgen; ROLIK, Oleksandr; SOKOLOV, Volodymyr. Techniques Comparison for Natural Language Processing. MoMLeT&DS, v. 2631, n. 1, p. 57–67, 2020. Disponível em: https://core.ac.uk/reader/328802590. Acesso em: 22 mar. 2024.

KALYANI, Aditi; KUMUD, Hemant; SINGH, Shashi Pal; KUMAR, Ajai; DARBARI, Hemant. Evaluation and Ranking of Machine Translated Output in Hindi Language using Precision and Recall Oriented Metrics. [S. l.]: arXiv, abr. 2014. DOI: 10.48550/arXiv.1404.1847. Disponível em: http://arxiv.org/abs/1404.1847. Acesso em: 22 mar. 2024.

KIMERA, Richard; RIM, Daniela N.; CHOI, Heeyoul. Building a Parallel Corpus and Training Translation Models Between Luganda and English. Journal of KIISE, v. 49, n. 11, p. 1009–1016, nov. 2022. ISSN 2383-630X, 2383-6296. DOI: 10.5626/JOK.2022.49.11.1009. Disponível em: http://arxiv.org/abs/2301.02773. Acesso em: 22 mar. 2024.

KOGLIN, Arlene; MOURA, Willian Henrique Cândido; MATOS, Morgana Aparecida de; SILVEIRA, João Gabriel Pereira da. Quality assessment of machine-translated post-edited subtitles: an analysis of Brazilian translators’ perceptions. Linguistica Antverpiensia, New Series – Themes in Translation Studies, v. 22, dez. 2023. ISSN 2295-5739. DOI: 10.52034/lans-tts.v22i.765. Disponível em: https://lans-tts.uantwerpen.be/index.php/LANS-TTS/article/view/765. Acesso em: 22 mar. 2024.

KOPONEN, Maarit. Is machine translation post-editing worth the effort? A survey of research into post-editing and effort. The Journal of Specialised Translation, p. 131–148, 2016. Disponível em: https://www.phoenix3.ir/Article?id=34. Acesso em: 22 mar. 2024.

LAKEW, Surafel Melaku; CETTOLO, Mauro; FEDERICO, Marcello. A Comparison of Transformer and Recurrent Neural Networks on Multilingual Neural Machine Translation. In: BENDER, Emily M.; DERCZYNSKI, Leon; ISABELLE, Pierre (ed.). Proceedings of the 27th International Conference on Computational Linguistics. Santa Fe, New Mexico, USA: Association for Computational Linguistics, ago. 2018. p. 641–652. Disponível em: https://aclanthology.org/C18-1054. Acesso em: 22 mar. 2024.

O’HAGAN, Minako (ed.). The Routledge Handbook of Translation and Technology. 1. ed. Abingdon, Oxon ; New York, NY : Routledge, 2020. |: Routledge, ago. 2019. ISBN 9781315311258. DOI: 10.4324/9781315311258. Disponível em: https://www.taylorfrancis.com/books/9781315311241. Acesso em: 22 mar. 2024.

PAPINENI, Kishore; ROUKOS, Salim; WARD, Todd; ZHU, Wei-Jing. Bleu: a Method for Automatic Evaluation of Machine Translation. In: ISABELLE, Pierre; CHARNIAK, Eugene; LIN, Dekang (ed.). Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. Philadelphia, Pennsylvania, USA: Association for Computational Linguistics, jul. 2002. p. 311–318. DOI: 10.3115/1073083.1073135. Disponível em: https://aclanthology.org/P02-1040. Acesso em: 27 mar. 2024.

PEÑA AGUILAR, Argelia. Challenging machine translation engines: Some Spanish-English linguistic problems put to the test. Cadernos de Tradução, v. 43, n. 1, p. 1–26, dez. 2023. ISSN 2175-7968. DOI: 10.5007/2175-7968.2023.e85397. Disponível em: https://periodicos.ufsc.br/index.php/traducao/article/view/85397. Acesso em: 22 mar. 2024.

PÉREZ-VERA, Elisa. HCCH | Explanatory Report on the 1980 HCCH Child Abduction Convention. In: ACTS and Documents of the Fourteenth Session (1980), tome III, Child abduction. [S. l.: s. n.], 1980. Disponível em: https://www.hcch.net/en/publications-and-studies/details4/?pid=2779. Acesso em: 22 mar. 2024.

PIRES, Thiago Blanch. Ampliando olhares sobre a tradução automática online : um estudo exploratório de categorias de erros de máquina de tradução gerados em documentos multimodais. Jun. 2017. Tese de doutorado – Universidade de Brasília, Brasília. Disponível em: http://repositorio2.unb.br/jspui/handle/10482/23727. Acesso em: 22 mar. 2024.

PIRES, Thiago Blanch. A avaliação de tradução automática na atuação do bacharel em LEA-MSI. In: PEREIRA, Fernana Alencar (ed.). Línguas Estrangeiras Aplicadas: trajetórias e possibilidades. Campinas: Pontes Editores, 2020. p. 61–75.

POST, Matt. A Call for Clarity in Reporting BLEU Scores. [S. l.]: arXiv, set. 2018. DOI: 10.48550/arXiv.1804.08771. Disponível em: http://arxiv.org/abs/1804.08771. Acesso em: 22 mar. 2024.

RAFFEL, Colin; SHAZEER, Noam; ROBERTS, Adam; LEE, Katherine; NARANG, Sharan; MATENA, Michael; ZHOU, Yanqi; LI, Wei; LIU, Peter J. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. [S. l.]: arXiv, set. 2020. DOI: 10.48550/arXiv.1910.10683. Disponível em: http://arxiv.org/abs/1910.10683. Acesso em: 22 mar. 2024.

RODAS, João Grandino; MÔNACO, Gustavo Ferraz de Campos. A Conferência da Haia de direito internacional privado: a participação do Brasil. Brasília: Fundação Alexandre de Gusmão, 2007.

TIAN, Taoling; SONG, Chai; TING, Jin; HUANG, Hongyang. A French-to-English Machine Translation Model Using Transformer Network. Procedia Computer Science, v. 199, p. 1438–1443, jan. 2022. ISSN 1877-0509. DOI: 10.1016/j.procs.2022.01.182. Disponível em: https://www.sciencedirect.com/science/article/pii/S1877050922001831. Acesso em: 22 mar. 2024.

VASWANI, Ashish; SHAZEER, Noam; PARMAR, Niki; USZKOREIT, Jakob; JONES, Llion; GOMEZ, Aidan N.; KAISER, Lukasz; POLOSUKHIN, Illia. Attention Is All You Need. [S. l.]: arXiv, ago. 2017. DOI: 10.48550/arXiv.1706.03762. Disponível em: http://arxiv.org/abs/1706.03762. Acesso em: 22 mar. 2024.

VILAR, David; XU, Jia; D’HARO, Luis Fernando; NEY, Hermann. Error Analysis of Statistical Machine Translation Output. In: CALZOLARI, Nicoletta; CHOUKRI, Khalid; GANGEMI, Aldo; MAEGAARD, Bente; MARIANI, Joseph; ODIJK, Jan; TAPIAS, Daniel (ed.). Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06). Genoa, Italy: European Language Resources Association (ELRA), maio 2006. Disponível em: http://www.lrec-conf.org/proceedings/lrec2006/pdf/413_pdf.pdf. Acesso em: 22 mar. 2024.

WHITE, J. S. How to evaluate machine translation. In: SOMERS, Harold (ed.). Computers and Translation: A translator’s guide. Amsterdam/Philadelphia: John Benjamins Publishing, 2003. v. 35. p. 211–244.

WOLF, Thomas; DEBUT, Lysandre; SANH, Victor; CHAUMOND, Julien; DELANGUE, Clement; MOI, Anthony; CISTAC, Pierric; RAULT, Tim; LOUF, Rémi; FUNTOWICZ, Morgan; DAVISON, Joe; SHLEIFER, Sam; PLATEN, Patrick von; MA, Clara; JERNITE, Yacine; PLU, Julien; XU, Canwen; SCAO, Teven Le; GUGGER, Sylvain; DRAME, Mariama; LHOEST, Quentin; RUSH, Alexander M. HuggingFace’s Transformers: State-of-the-art Natural Language Processing. [S. l.]: arXiv, jul. 2020. DOI: 10.48550/arXiv.1910.03771. Disponível em: http://arxiv.org/abs/1910.03771. Acesso em: 22 mar. 2024.

WU, Yonghui; SCHUSTER, Mike; CHEN, Zhifeng; LE, Quoc V.; NOROUZI, Mohammad; MACHEREY, Wolfgang; KRIKUN, Maxim; CAO, Yuan; GAO, Qin; MACHEREY, Klaus; KLINGNER, Jeff; SHAH, Apurva; JOHNSON, Melvin; LIU, Xiaobing; KAISER, Łukasz; GOUWS, Stephan; KATO, Yoshikiyo; KUDO, Taku; KAZAWA, Hideto; STEVENS, Keith; KURIAN, George; PATIL, Nishant; WANG, Wei; YOUNG, Cliff; SMITH, Jason; RIESA, Jason; RUDNICK, Alex; VINYALS, Oriol; CORRADO, Greg; HUGHES, Macduff; DEAN, Jeffrey. Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation. [S. l.]: arXiv, out. 2016. DOI: 10.48550/arXiv.1609.08144. Disponível em: http://arxiv.org/abs/1609.08144. Acesso em: 22 mar. 2024.