The eye-ear relationship: nvestigating auditory impacts on subtitle reading and comprehension

Hussein Abu-Rayyash; Shatha Alhawamdeh; Yuri Ringomon

doi:10.1590/1983-3652.2024.52687

Authors

Hussein Abu-Rayyash Kent State University, College of Arts and Sciences, Department of Modern and ClassicalLanguage Studies, Kent, Ohio, USA https://orcid.org/0000-0002-9695-4030
Shatha Alhawamdeh Kent State University, College of Arts and Sciences, Department of Modern and ClassicalLanguage Studies, Kent, Ohio, USA / Oberlin College, College of Arts and Sciences, Department of Middle East and North Africa Studies, Oberlin, Ohio, USA https://orcid.org/0009-0001-6342-7242
Yuri Ringomon Kent State University, College of Arts and Sciences, Department of Modern and ClassicalLanguage Studies, Kent, Ohio, USA https://orcid.org/0009-0000-6508-8769

DOI:

https://doi.org/10.1590/1983-3652.2024.52687

Keywords:

Eye Movement, Audio, Subtitles, Reading, Comprehension

Abstract

The study employs eye-tracking measures and comprehension quizzes to investigate linguistic processing during subtitle reading and comprehension in Arabic and Spanish. Specifically, it examines the impact of native English (L1) audio, foreign Spanish or Arabic (L2) audio, or the absence of audio on subtitle reading strategies and the processing of embedded linguistic units in novice learners. The study utilizes a 2x3 within- subjects design, manipulating audio condition (L1 audio, L2 audio, no audio) and subtitle language (Spanish, Arabic). Global eye-tracking analyses reveal subtitle reading patterns, while local analyses focus on the processing of specific linguistic units. Comprehension quizzes assess learning outcomes after the viewing of the subtitled videos. The findings suggest that the absence of audio leads to increased fixation durations, indicating greater attentional demand. Spanish subtitles result in higher fixation counts and prolonged reading times compared to Arabic, reflecting deeper cognitive processing. The study highlights the complex interplay between audio presence, subtitle language, and individual proficiency in shaping attentional allocation and lexical processing during language acquisition through audio-visual media. Keywords: Eye Movement. Audio. Subtitles. Reading. Comprehension.

Downloads

Download data is not yet available.

References

ALABOUD, Amal. The Positive Effect of Translation on Improving Reading Comprehension among Female Arabic Learners of English as Foreign Language. Arab World English Journal (AWEJ), v. 13, n. 2, p. 424–436, 2022. DOI: 10.24093/awej/vol13no2.29.

ANIS, Muhammad Yunus; MALIK ABDUL ANDS ETIYADI, Alif Cahya. Developing Creative Writing Process Based on the Translation of Arabic Phrases: Case Study in Arabic Prose and Poetry. Journal of Narrative and Language Studies (NALANS), v. 10, n. 20, p. 173–187, 2022. DOI: 10.59045/nalans.2023.3..

BAFNA, Tanya; HANSEN, John Paulin. Mental fatigue measurement using eye metrics: A systematic literature review. Psychophysiology, v. 58, n. 6, e13828, 2021. DOI: 10.1111/psyp.13828.

BARANOWSKA, Karolina. Learning most with least effort: subtitles and cognitive load. ELT Journal, v. 74, n. 2, p. 105–115, Mar. 2020. ISSN 0951-0893. DOI: 10.1093/elt/ccz060.

BISSON, Marie-Josée; HEUVEN, Walter J B van; CONKLIN, Kathy; TUNNEYR, Richard. Processing of native and foreign language subtitles in films: An eye tracking study. Applied Psycholinguistics, v. 35, n. 2, p. 399–418, 2014. DOI: 10.1017/s0142716412000434.

BLYTHE, Hazel I.; LIVERSEDGE, Simon P.; JOSEPH, Holly S.S.L.; WHITE, Sarah J.; FINDLAY, John M.; RAYNER, Keith. The binocular coordination of eye movements during reading in children and adults. Vision Research, v. 46, n. 22, p. 3898–3908, 2006. ISSN 0042-6989. DOI: https://doi.org/10.1016/j.visres.2006.06.006. Available from: https://www.sciencedirect.com/science/article/pii/S0042698906002586.

BOLAÑOS GARCÍA-ESCRIBANO, Alejandro; OGEA POZO, María del Mar. Didactic audiovisual translation. Translation and Translanguaging in Multilingual Contexts, John Benjamins, v. 9, n. 2, p. 187–215, 2023. ISSN 2352-1805. DOI: https://doi.org/10.1075/ttmc.00108.bol. Available from: https://www.jbe-platform.com/content/journals/10.1075/ttmc.00108.bol.

BORRÁS, ISABEL; LAFAYETTE, ROBERT C. Effects of Multimedia Courseware Subtitling on the Speaking Performance of College Students of French. The Modern Language Journal, v. 78, n. 1, p. 61–75, 1994. DOI: https://doi.org/10.1111/j.1540-4781.1994.tb02015.x. Available from: https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1540-4781.1994.tb02015.x.

CONDINHO BRAVO, Maria Da Conceiçao. Putting the reader in the picture. Screen translation and foreign-language learning. 2008. PhD thesis – Universitat Rovira i Virgili. ISBN 9788469198667.

DÍAZ-CINTAS, Jorge; HAYES, Lydia. Role Reversal: An Overview of Audiovisual Translation into English. Íkala, Revista de Lenguaje y Cultura, v. 28, n. 2, p. 1–18, May 2023. DOI: 10.17533/udea.ikala.v28n2a01. Available from: https://revistas.udea.edu.co/index.php/ikala/article/view/353770.

FERNÁNDEZ-COSTALES, Alberto; TALAVAN, Noa; TINEDO, Antonio. Didactic audiovisual translation in language teaching: Results from TRADILEX. Comunicar, Media Education Research Jounral. DOI: 10.3916/c77-2023-02.

GRAESSER, Arthur C.; MCNAMARA, Danielle S.; CAI, Zhiqang; CONLEY, Mark; LI, Haiying; PENNEBAKER, James. Coh-Metrix Measures Text Characteristics at Multiple Levels of Language and Discourse. The Elementary School Journal, v. 115, n. 2, p. 210–229, 2014. DOI: 10.1086/678293. Available from: https://doi.org/10.1086/678293.

HAIDER, Ahmad S.; AL-SALMAN, Saleh. The Effects of Intralingual Subtitles on Jordanian University Students’ Foreign Language Learning. International Journal of Instruction, v. 15, n. 4, p. 57–76, Jan. 2022. Available from: https://e-iji.net/ats/index.php/pub/article/view/252.

HOLMQVIST, K.; NYSTRÖM, M.; ANDERSSON, R.; DEWHURST, R.; JARODZKA, H.; VAN DE WEIJER, J. Eye tracking: A comprehensive guide to methods and measures. [S. l.]: Oxford University Press, 2011.

HORNERO CORISCO, Ana María; GONZALEZ-VERA, Pilar; BELTRÁN, Paula Buil. Becoming a techie and improving your English with audiovisual translation. Translation and Translanguaging in Multilingual Contexts, John Benjamins, v. 9, n. 2, p. 160–186, 2023. ISSN 2352-1805. DOI: https://doi.org/10.1075/ttmc.00107.hor. Available from: https://www.jbe-platform.com/content/journals/10.1075/ttmc.00107.hor.

KIM, Hye-Jeong. The Effect of Subtitle Correction Activities on Learner’s English Reading Achievement. Korean Association For Learner-Centered Curriculum And Instruction, v. 23, p. 325–336, 2023. DOI: 10.22251/jlcci.2023.23.7.325.

LEWANDOWSKA, B. RealEye Eye-tracking system Technology Whitepaper. Retrieved August, n. 29, 2019.

LIAO, Sixin; YU, Lili; KRUGER, Jan-Louis; REICHLE, Erik D. The impact of audio on the reading of intralingual versus interlingual subtitles: Evidence from eye movements. Applied Psycholinguistics, v. 43, n. 1, p. 237–269, 2022. DOI: 10.1017/s0142716421000527.

LIAO, Sixin; YU, Lili; KRUGER, Jean-Louis. Using Eye Movements to Study the Reading of Subtitles in Video. Scientific Studies of Reading, v. 25, n. 5, p. 417–435, 2020. DOI: 10.1080/10888438.2020.1823986.

OMAR, Khadija Saleem; MOHAMMED SALMAN, Nahida; VADIVEL, Balachandran. The Impact of English Movies with Standard Subtitles on Enhancing the Writing Skills of EFL Students at Cihan University – Duhok. International Journal of Humanities and Education Development (IJHED), v. 5, p. 45–52, May 2023. DOI: 10.22161/jhed.5.3.7. Available from: https://theshillonga.com/index.php/jhed/article/view/44.

QAZI, Aisha; KHAN, Zahra. Impact on English Language Learners’ Listening Comprehension and Acquisition of Vocabulary Through the Use of Subtitled Videos. Education and Social Sciences Review, v. 3, n. 1, p. 32–43, Feb. 2023. DOI: 10.48112/aessr.v3i1.400.

ROSS, Nicholas M.; KOWLER, Eileen. Eye movements while viewing narrated, captioned, and silent videos. Journal of Vision, v. 13, n. 4, p. 1–1, Mar. 2013. ISSN 1534-7362. DOI: 10.1167/13.4.1.

RUSHWAN, Ismail Mohammed Hamid. The Role of Translation in Developing ESP Learners’ Reading Comprehension Skills- A Case Study of Medical Students at Najran University-KSA. International Journal of Applied Linguistics and English Literature, v. 6, n. 3, p. 243–253, 2017. ISSN 2200-3452. DOI: 10.7575/aiac.ijalel.v.6n.3p.243. Available from: https://journals.aiac.org.au/index.php/IJALEL/article/view/3097.

SCHOTTER, E. R.; RAYNER, K. Eye movements and word recognition during reading. In: ADELMAN, J.S. (ed.). isual word recognition: Meaning and context, individuals and development. [S. l.]: Psychology Press, 2012. p. 73–101.

SZARKOWSKA, Agnieszka; GERBER-MORÓN, Olivia. Viewers can keep up with fast subtitles: Evidence from eye movements. Plos One, Public Library of Science, v. 13, n. 6, e0199331, June 2018. DOI: 10.1371/journal.pone.0199331.

RAGNI, Valentina. More than meets the eye: An eye-tracking study on the effects of translation on the processing and memorisation of reversed subtitles. JoSTrans: The Journal of Specialised Translation, University of Bristol, v. 33, Special issue ”Experimental research and cognition in AVT”, p. 99–128, Jan. 2020. Available from: http://www.jostrans.org/issue33/art_ragni.pdf.

WINKE, Paula; GASS, Susan; SYDORENKO, Tetyana. Factors Influencing the Use of Captions by Foreign Language Learners: An Eye-Tracking Study. The Modern Language Journal, v. 97, n. 1, p. 254–275, 2013. DOI: 10.1111/j.1540-4781.2013.01432.x. Available from: https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1540-4781.2013.01432.x.

XU, Yilan; JORDAN, Eoin; LI, Yuanzhe. Chinese EFL Teachers’ Perceptions of Subtitles’ Effects on English Comprehension and Vocabulary Acquisition: Subtitles’ Effects on English Capability. International Journal of Computer-Assisted Language Learning and Teaching, v. 12, n. 1, p. 1–19, 2022. ISSN 2155-7098. DOI: 10.4018/ijcallt.315622.