Quality assessment of machine translation output: cognitive evaluation approach in an eye tracking experiment

Ramunė  Kasperavičienė; Jurgita  Motiejūnienė; Irena  Patašienė

doi:10.35699/1983-3652.2020.24399

Authors

Ramunė Kasperavičienė Kaunas University of Technology https://orcid.org/0000-0003-0782-3758
Jurgita Motiejūnienė Kaunas University of Technology
Irena Patašienė Kaunas University of Technology https://orcid.org/0000-0002-7569-8314

DOI:

https://doi.org/10.35699/1983-3652.2020.24399

Keywords:

Machine translation, Cognitive evaluation approach, Translation error(s), Eye tracking, Acceptability

Abstract

Despite fast development of machine translation, the output quality is less than acceptable in certain language pairs. The aim of this paper is to determine the types of errors in machine translation output that cause comprehension problems to potential readers. The study is based on a reading task experiment using eye tracking and a retrospective survey as a complementary method to add more value to the research as eye tracking as a method is considered to be problematic and challenging (O’BRIEN, 2009; ALVES et al., 2009). The cognitive evaluation approach is used in an eye tracking experiment to determine the complexity of the errors in the English–Lithuanian language pair from easiest to hardest as seen by the readers of a machine-translated text. The tested parameters – gaze time and fixation count – demonstrate that a different amount of cognitive effort is required to process different types of errors in machine-translated texts. The current work aims at contributing to other research in the Translation Studies field by providing the analysis of error assessment of machine translation output.

Author Biography

Ramunė Kasperavičienė, Kaunas University of Technology

Professor at the Institute of Social Sciences, Arts and Humanities, principal investigator of the research group "Language and Technologies"

References

ALVES, F.; SARTO SZPAK, K.; GONÇALVES, J. L.; SEKINO, K.; AQUINO, M.; ARAUJO E CASTRO, R.; KOGLIN, A.; DE LIMA FONSECA, N. B.; MESA-LAO, B. Investigating cognitive effort in post-editing: A relevance-theoretical approach. In: HANSEN-SCHIRRA, S. and GRUCZA, S. (eds.). Eyetracking and applied linguistics, Berlin: language science press, 2016. p. 109-142.

ALVES, F.; PAGANO, A.; DA SILVA, I. A new window on translators’ cognitive activity: methodological issues in the combined use of eye tracking, key logging and retrospective protocols. In: MEES, I. M.; ALVES, F.; GÖPFERICH, S. (eds.). Methodology, Technology and Innovation in Translation Process Research. Copenhagen: Samfundslitteratur, 2009. p. 267-292.

CARL, M.; DRAGSTED, B.; ELMING, J.; HARDT, D.; LYKKE JAKOBSEN, A. The process of post-editing: a pilot study. In: Proceedings of the 8th International Natural Language Processing and Cognitive Science Workshop, p. 131-142, 2011.

CARL, M.; GUTERMUTH, S.; HANSEN-SCHIRRA, S. Post-editing machine translation: Efficiency, strategies, and revision processes in professional translation settings. Psycholinguistic and Cognitive Inquiries into Translation and Interpreting. In: FERREIRA, A.; SCHWIETER, J. W. (eds.). Psycholinguistic and cognitive inquiries into translation and interpreting. Amsterdam: John Benjamins, 2015. p. 145-174.

CASTILHO, S. Measuring Acceptability of Machine Translated Enterprise Content (PhD thesis). Dublin: Dublin City University, 2016.

CASTILHO, S.; DOHERTY, S.; GASPARI, F.; MOORKENS, J. Approaches to Human and Machine Translation Quality Assessment. In: MOORKENS J.; CASTILHO S.; GASPARI F.; DOHERTY S. Translation Quality Assessment. Machine Translation: Technologies and Applications, 2018. p. 9-38.

CHOMSKY, N. Aspects of the theory of syntax. Cambridge: MIT Press, 1969.

DAEMS, J.; VANDEPITTE, S.; HARTSUIKER, R. J.; MACKEN, L. Identifying the machine translation Error types with the greatest impact on post-editing effort. In: Frontiers in Psychology, v. 8, 2017. p. 1-15.

DE BEAUGRANDE, R.; DRESSLER, W. Introduction to text linguistics. London, New York, 1981.

FLANAGAN, A. M. Error classification for MT evaluation. In: Proceedings of the first Conference of the Association for Machine Translation in the Americas, 1994. p. 65-72.

GONÇALVES, J. L. Investigating Saccades as an index of cognitive effort in post-editing and translation. In: Proceedings of EST Congress. 2016, Aarhus, Denmark, 2016.

GRAHAM, Y. Improving Evaluation of Machine Translation Quality Estimation. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, 2015. p. 1804-1813.

HVELPLUND, K. T. Four fundamental types of reading during translation. In: JAKOBSEN, A. L.; MESA-LAO, B. (eds.), Translation in Transition: Between Cognition, Computing and Technology. Amsterdam / Philadelphia: John Benjamins Publishing Company, 2017. p. 55-77.

HUTCHINS, J.; SOMERS, H. An introduction to machine translation. London: Academic Press, 1992.

JAKOBSEN, A. L.; JENSEN K. T. H. Eye movement behaviour across four different types of reading task. In: GÖPFERICH, S.; JAKOBSEN, A. L.; MEES, I. M. (eds.). Looking at eyes: eye-tracking studies of reading and translation processing, 2008. p. 78-98.

KORNACKI, M. The application of eye-tracking in translator training. inTRAlinea: New Insights into Translator Training, 2019.

MOORKENS, J. Eye tracking as a measure of cognitive effort for post-editing of machine translation. In: WALKER, C.; FEDERICI, F. M. (eds.). Eye tracking and multidisciplinary studies on translation, 2018. p. 55-69.

NITZKE, J. Monolingual post-editing: An exploratory study on research behavior and target text quality. In: HANSEN-SCHIRRA, S.; GRUCZA, S. (eds.). Eyetracking and applied linguistics, Berlin: Language Science Press, 2016. p. 83-109.

O’BRIEN, SH. Towards predicting post-editing productivity. Machine Translation, v. 25, n. 3, p. 197-215, 2011.

O’BRIEN, SH. Eye tracking in translation-process research: methodological challenges and solutions. In: MEES, I. M.; ALVES, F.; GÖPFERICH, S. (eds.). Methodology, Technology and Innovation in Translation Process Research. Copenhagen: Samfundslitteratur, 2009. p. 251-266.

O’BRIEN, SH. Machine-translatability and post-editing effort: An empirical study using translog and choice network analysis. Dublin: Dublin City University, 2006.

PETKEVIČIŪTĖ, I.; TAMULYNAS, B. Kompiuterinis vertimas į lietuvių kalbą: alternatyvos ir jų lingvistinis vertinimas. Studies about Languages, v. 18, Kaunas: Technologija, p. 38-45, 2011.

POPOVIC, M. Error classification and analysis for machine translation quality assessment. In: MOORKENS, J.; CASTILHO, S.; GASPARI, F.; DOHERTY, S. (eds.). Translation quality assessment: from principles to practice, Springer international publishing AG, 2018. p. 129-158.

ROTURIER, J. An investigation into the impact of controlled English rules on the comprehensibility, usefulness and acceptability of machine-translated technical documentation for French and German users (PhD thesis). Dublin: Dublin City University, 2006.

SPECIA, L. Exploiting objective annotations for measuring translation post-editing effort. In: FORCADA, M. L.; DEPRAETERE H.; VANDEGHINSTE V. (eds.). Proceedings of the 15th Conference of the European Association for Machine Translation, 2011. p. 28-37.

STYMNE, S.; DANIELSSON, H.; BREMIN, S.; HU, H.; KARLSSON, J.; PRYTZ LILLKULL, A.; WESTER, M. Eye Tracking as a Tool for Machine Translation Error Analysis. In: CALZOLARI, N.; CHOUKRI, K.; DECLERCK, T.; UĞUR DOĞAN, M.; MAEGAARD, B.; MARIANI, J.; MORENO, A.; ODIJK, J.; PIPERIDIS, S. (eds.). Proceedings of the 8th international conference on language resources and evaluation, 2012. p. 1121-1126.

TEMNIKOVA, I. Cognitive evaluation approach for a controlled language post-editing experiment. In: Proceedings of the 7th international conference on language resources and evaluation, 2010. p. 3485-3490.

TEMNIKOVA, I.; ZAGHOUANI W.; VOGEL, S.; HABASH, N. Applying the Cognitive Machine Translation Evaluation Approach to Arabic. In: Proceedings of the International Conference on Language Resources and Evaluation, 2016. p. 3644-3651.

TURENKO, V.; BALTULIONIS, S.; VASILJEVAS, M.; DAMAŠEVIČIUS, R. Analysing program source code reading skills with eye tracking technology, Information Technology. In: Proceedings of IVUS’2019, Vytautas Magnus University, 2019. p. 50-54.

VAN SLYPE, G. Critical study of methods for evaluating the quality of machine translation. Brussels: Buerau Marcel van Dijk, 1979.

VILAR, D.; XU, J.; D’HARO, L. F.; NEY, H. Error analysis of statistical machine translation output. In: Proceedings of the 5th International Conference on Language Resources and Evaluation, 2006. p. 697-702.