Quality assessment of machine translation output: cognitive evaluation approach in an eye tracking experiment

Ramunė  Kasperavičienė; Jurgita  Motiejūnienė; Irena  Patašienė

doi:10.35699/1983-3652.2020.24399

Autores

Ramunė Kasperavičienė Kaunas University of Technology https://orcid.org/0000-0003-0782-3758
Jurgita Motiejūnienė Kaunas University of Technology
Irena Patašienė Kaunas University of Technology https://orcid.org/0000-0002-7569-8314

DOI:

https://doi.org/10.35699/1983-3652.2020.24399

Palavras-chave:

Tradução automática, Abordagem de avaliação cognitiva, Erro(s) de tradução, Rastreamento ocular, Aceitabilidade

Resumo

Apesar do rápido desenvolvimento da tradução automática, a qualidade do texto produzido é bastante pobre em algumas combinações linguísticas. O objetivo deste artigo é determinar os tipos de erros na produção de tradução automática que acarretam dificuldades de compreensão para os potenciais leitores. O estudo é baseado em um experimento que utiliza rastreamento ocular e um questionário retrospetivo como método complementar de forma a acrescentar mais valor à pesquisa, visto que o rastreamento ocular enquanto método é muitas vezes considerado problemático e desafiador (O’BRIEN, 2009; ALVES et al., 2009). A abordagem de avaliação cognitiva é utilizada em um experimento com rastreamento ocular para determinar a complexidade dos erros na combinação linguística inglês-lituano dos mais fáceis aos mais difíceis, conforme visto pelos leitores do texto traduzido automaticamente. Os parâmetros testados (duração do olhar e número de fixações) demonstram que é necessário um esforço cognitivo diferente para processar diferentes tipos de erros em textos traduzidos de forma automática. Este trabalho almeja contribuir para outras pesquisas neste campo, pois fornece análise de avaliação de erros da produção de tradução automática.

Biografia do Autor

Ramunė Kasperavičienė, Kaunas University of Technology

Professor do Instituto de Ciências Sociais, Artes e Humanidades, pesquisador principal do grupo de pesquisa "Linguagem e Tecnologias"

Referências

ALVES, F.; SARTO SZPAK, K.; GONÇALVES, J. L.; SEKINO, K.; AQUINO, M.; ARAUJO E CASTRO, R.; KOGLIN, A.; DE LIMA FONSECA, N. B.; MESA-LAO, B. Investigating cognitive effort in post-editing: A relevance-theoretical approach. In: HANSEN-SCHIRRA, S. and GRUCZA, S. (eds.). Eyetracking and applied linguistics, Berlin: language science press, 2016. p. 109-142.

ALVES, F.; PAGANO, A.; DA SILVA, I. A new window on translators’ cognitive activity: methodological issues in the combined use of eye tracking, key logging and retrospective protocols. In: MEES, I. M.; ALVES, F.; GÖPFERICH, S. (eds.). Methodology, Technology and Innovation in Translation Process Research. Copenhagen: Samfundslitteratur, 2009. p. 267-292.

CARL, M.; DRAGSTED, B.; ELMING, J.; HARDT, D.; LYKKE JAKOBSEN, A. The process of post-editing: a pilot study. In: Proceedings of the 8th International Natural Language Processing and Cognitive Science Workshop, p. 131-142, 2011.

CARL, M.; GUTERMUTH, S.; HANSEN-SCHIRRA, S. Post-editing machine translation: Efficiency, strategies, and revision processes in professional translation settings. Psycholinguistic and Cognitive Inquiries into Translation and Interpreting. In: FERREIRA, A.; SCHWIETER, J. W. (eds.). Psycholinguistic and cognitive inquiries into translation and interpreting. Amsterdam: John Benjamins, 2015. p. 145-174.

CASTILHO, S. Measuring Acceptability of Machine Translated Enterprise Content (PhD thesis). Dublin: Dublin City University, 2016.

CASTILHO, S.; DOHERTY, S.; GASPARI, F.; MOORKENS, J. Approaches to Human and Machine Translation Quality Assessment. In: MOORKENS J.; CASTILHO S.; GASPARI F.; DOHERTY S. Translation Quality Assessment. Machine Translation: Technologies and Applications, 2018. p. 9-38.

CHOMSKY, N. Aspects of the theory of syntax. Cambridge: MIT Press, 1969.

DAEMS, J.; VANDEPITTE, S.; HARTSUIKER, R. J.; MACKEN, L. Identifying the machine translation Error types with the greatest impact on post-editing effort. In: Frontiers in Psychology, v. 8, 2017. p. 1-15.

DE BEAUGRANDE, R.; DRESSLER, W. Introduction to text linguistics. London, New York, 1981.

FLANAGAN, A. M. Error classification for MT evaluation. In: Proceedings of the first Conference of the Association for Machine Translation in the Americas, 1994. p. 65-72.

GONÇALVES, J. L. Investigating Saccades as an index of cognitive effort in post-editing and translation. In: Proceedings of EST Congress. 2016, Aarhus, Denmark, 2016.

GRAHAM, Y. Improving Evaluation of Machine Translation Quality Estimation. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, 2015. p. 1804-1813.

HVELPLUND, K. T. Four fundamental types of reading during translation. In: JAKOBSEN, A. L.; MESA-LAO, B. (eds.), Translation in Transition: Between Cognition, Computing and Technology. Amsterdam / Philadelphia: John Benjamins Publishing Company, 2017. p. 55-77.

HUTCHINS, J.; SOMERS, H. An introduction to machine translation. London: Academic Press, 1992.

JAKOBSEN, A. L.; JENSEN K. T. H. Eye movement behaviour across four different types of reading task. In: GÖPFERICH, S.; JAKOBSEN, A. L.; MEES, I. M. (eds.). Looking at eyes: eye-tracking studies of reading and translation processing, 2008. p. 78-98.

KORNACKI, M. The application of eye-tracking in translator training. inTRAlinea: New Insights into Translator Training, 2019.

MOORKENS, J. Eye tracking as a measure of cognitive effort for post-editing of machine translation. In: WALKER, C.; FEDERICI, F. M. (eds.). Eye tracking and multidisciplinary studies on translation, 2018. p. 55-69.

NITZKE, J. Monolingual post-editing: An exploratory study on research behavior and target text quality. In: HANSEN-SCHIRRA, S.; GRUCZA, S. (eds.). Eyetracking and applied linguistics, Berlin: Language Science Press, 2016. p. 83-109.

O’BRIEN, SH. Towards predicting post-editing productivity. Machine Translation, v. 25, n. 3, p. 197-215, 2011.

O’BRIEN, SH. Eye tracking in translation-process research: methodological challenges and solutions. In: MEES, I. M.; ALVES, F.; GÖPFERICH, S. (eds.). Methodology, Technology and Innovation in Translation Process Research. Copenhagen: Samfundslitteratur, 2009. p. 251-266.

O’BRIEN, SH. Machine-translatability and post-editing effort: An empirical study using translog and choice network analysis. Dublin: Dublin City University, 2006.

PETKEVIČIŪTĖ, I.; TAMULYNAS, B. Kompiuterinis vertimas į lietuvių kalbą: alternatyvos ir jų lingvistinis vertinimas. Studies about Languages, v. 18, Kaunas: Technologija, p. 38-45, 2011.

POPOVIC, M. Error classification and analysis for machine translation quality assessment. In: MOORKENS, J.; CASTILHO, S.; GASPARI, F.; DOHERTY, S. (eds.). Translation quality assessment: from principles to practice, Springer international publishing AG, 2018. p. 129-158.

ROTURIER, J. An investigation into the impact of controlled English rules on the comprehensibility, usefulness and acceptability of machine-translated technical documentation for French and German users (PhD thesis). Dublin: Dublin City University, 2006.

SPECIA, L. Exploiting objective annotations for measuring translation post-editing effort. In: FORCADA, M. L.; DEPRAETERE H.; VANDEGHINSTE V. (eds.). Proceedings of the 15th Conference of the European Association for Machine Translation, 2011. p. 28-37.

STYMNE, S.; DANIELSSON, H.; BREMIN, S.; HU, H.; KARLSSON, J.; PRYTZ LILLKULL, A.; WESTER, M. Eye Tracking as a Tool for Machine Translation Error Analysis. In: CALZOLARI, N.; CHOUKRI, K.; DECLERCK, T.; UĞUR DOĞAN, M.; MAEGAARD, B.; MARIANI, J.; MORENO, A.; ODIJK, J.; PIPERIDIS, S. (eds.). Proceedings of the 8th international conference on language resources and evaluation, 2012. p. 1121-1126.

TEMNIKOVA, I. Cognitive evaluation approach for a controlled language post-editing experiment. In: Proceedings of the 7th international conference on language resources and evaluation, 2010. p. 3485-3490.

TEMNIKOVA, I.; ZAGHOUANI W.; VOGEL, S.; HABASH, N. Applying the Cognitive Machine Translation Evaluation Approach to Arabic. In: Proceedings of the International Conference on Language Resources and Evaluation, 2016. p. 3644-3651.

TURENKO, V.; BALTULIONIS, S.; VASILJEVAS, M.; DAMAŠEVIČIUS, R. Analysing program source code reading skills with eye tracking technology, Information Technology. In: Proceedings of IVUS’2019, Vytautas Magnus University, 2019. p. 50-54.

VAN SLYPE, G. Critical study of methods for evaluating the quality of machine translation. Brussels: Buerau Marcel van Dijk, 1979.

VILAR, D.; XU, J.; D’HARO, L. F.; NEY, H. Error analysis of statistical machine translation output. In: Proceedings of the 5th International Conference on Language Resources and Evaluation, 2006. p. 697-702.