The eye-ear relationship: nvestigating auditory impacts on subtitle reading and comprehension

Hussein Abu-Rayyash; Shatha Alhawamdeh; Yuri Ringomon

doi:10.1590/1983-3652.2024.52687

Autores

Hussein Abu-Rayyash Kent State University, College of Arts and Sciences, Department of Modern and ClassicalLanguage Studies, Kent, Ohio, USA https://orcid.org/0000-0002-9695-4030
Shatha Alhawamdeh Kent State University, College of Arts and Sciences, Department of Modern and ClassicalLanguage Studies, Kent, Ohio, USA / Oberlin College, College of Arts and Sciences, Department of Middle East and North Africa Studies, Oberlin, Ohio, USA https://orcid.org/0009-0001-6342-7242
Yuri Ringomon Kent State University, College of Arts and Sciences, Department of Modern and ClassicalLanguage Studies, Kent, Ohio, USA https://orcid.org/0009-0000-6508-8769

DOI:

https://doi.org/10.1590/1983-3652.2024.52687

Palavras-chave:

Movimento Ocular, Áudio, Legendas, Leitura, Compreensão

Resumo

O estudo emprega medidas de rastreamento ocular e questionários de compreensão para investigar o processamento linguístico durante a leitura e compreensão de legendas em árabe e espanhol. Especificamente, examina o impacto do áudio nativo em inglês (L1), áudio estrangeiro em espanhol ou árabe (L2) ou a ausência de áudio nas estratégias de leitura de legendas e no processamento de unidades linguísticas inseridas em aprendizes iniciantes. O estudo utiliza um delineamento intra-sujeitos 2x3, manipulando a condição de áudio (áudio L1, áudio L2, sem áudio) e a língua das legendas (espanhol, árabe). As análises globais de rastreamento ocular revelam padrões de leitura de legendas, enquanto as análises locais se concentram no processamento de unidades linguísticas específicas. Os questionários de compreensão avaliam os resultados de aprendizagem após assistir aos vídeos legendados. Os resultados sugerem que a ausência de áudio leva a um aumento na duração das fixações, indicando uma maior demanda atencional. As legendas em espanhol resultam em um maior número de fixações e tempos de leitura prolongados em comparação com o árabe, refletindo um processamento cognitivo mais profundo. O estudo destaca a complexa interação entre a presença de áudio, a língua das legendas e a proficiência individual na alocação de atenção e no processamento lexical durante a aquisição de línguas através de mídia audiovisual.

Referências

ALABOUD, Amal. The Positive Effect of Translation on Improving Reading Comprehension among Female Arabic Learners of English as Foreign Language. Arab World English Journal (AWEJ), v. 13, n. 2, p. 424–436, 2022. DOI: 10.24093/awej/vol13no2.29.

ANIS, Muhammad Yunus; MALIK ABDUL ANDS ETIYADI, Alif Cahya. Developing Creative Writing Process Based on the Translation of Arabic Phrases: Case Study in Arabic Prose and Poetry. Journal of Narrative and Language Studies (NALANS), v. 10, n. 20, p. 173–187, 2022. DOI: 10.59045/nalans.2023.3..

BAFNA, Tanya; HANSEN, John Paulin. Mental fatigue measurement using eye metrics: A systematic literature review. Psychophysiology, v. 58, n. 6, e13828, 2021. DOI: 10.1111/psyp.13828.

BARANOWSKA, Karolina. Learning most with least effort: subtitles and cognitive load. ELT Journal, v. 74, n. 2, p. 105–115, Mar. 2020. ISSN 0951-0893. DOI: 10.1093/elt/ccz060.

BISSON, Marie-Josée; HEUVEN, Walter J B van; CONKLIN, Kathy; TUNNEYR, Richard. Processing of native and foreign language subtitles in films: An eye tracking study. Applied Psycholinguistics, v. 35, n. 2, p. 399–418, 2014. DOI: 10.1017/s0142716412000434.

BLYTHE, Hazel I.; LIVERSEDGE, Simon P.; JOSEPH, Holly S.S.L.; WHITE, Sarah J.; FINDLAY, John M.; RAYNER, Keith. The binocular coordination of eye movements during reading in children and adults. Vision Research, v. 46, n. 22, p. 3898–3908, 2006. ISSN 0042-6989. DOI: https://doi.org/10.1016/j.visres.2006.06.006. Available from: https://www.sciencedirect.com/science/article/pii/S0042698906002586.

BOLAÑOS GARCÍA-ESCRIBANO, Alejandro; OGEA POZO, María del Mar. Didactic audiovisual translation. Translation and Translanguaging in Multilingual Contexts, John Benjamins, v. 9, n. 2, p. 187–215, 2023. ISSN 2352-1805. DOI: https://doi.org/10.1075/ttmc.00108.bol. Available from: https://www.jbe-platform.com/content/journals/10.1075/ttmc.00108.bol.

BORRÁS, ISABEL; LAFAYETTE, ROBERT C. Effects of Multimedia Courseware Subtitling on the Speaking Performance of College Students of French. The Modern Language Journal, v. 78, n. 1, p. 61–75, 1994. DOI: https://doi.org/10.1111/j.1540-4781.1994.tb02015.x. Available from: https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1540-4781.1994.tb02015.x.

CONDINHO BRAVO, Maria Da Conceiçao. Putting the reader in the picture. Screen translation and foreign-language learning. 2008. PhD thesis – Universitat Rovira i Virgili. ISBN 9788469198667.

DÍAZ-CINTAS, Jorge; HAYES, Lydia. Role Reversal: An Overview of Audiovisual Translation into English. Íkala, Revista de Lenguaje y Cultura, v. 28, n. 2, p. 1–18, May 2023. DOI: 10.17533/udea.ikala.v28n2a01. Available from: https://revistas.udea.edu.co/index.php/ikala/article/view/353770.

FERNÁNDEZ-COSTALES, Alberto; TALAVAN, Noa; TINEDO, Antonio. Didactic audiovisual translation in language teaching: Results from TRADILEX. Comunicar, Media Education Research Jounral. DOI: 10.3916/c77-2023-02.

GRAESSER, Arthur C.; MCNAMARA, Danielle S.; CAI, Zhiqang; CONLEY, Mark; LI, Haiying; PENNEBAKER, James. Coh-Metrix Measures Text Characteristics at Multiple Levels of Language and Discourse. The Elementary School Journal, v. 115, n. 2, p. 210–229, 2014. DOI: 10.1086/678293. Available from: https://doi.org/10.1086/678293.

HAIDER, Ahmad S.; AL-SALMAN, Saleh. The Effects of Intralingual Subtitles on Jordanian University Students’ Foreign Language Learning. International Journal of Instruction, v. 15, n. 4, p. 57–76, Jan. 2022. Available from: https://e-iji.net/ats/index.php/pub/article/view/252.

HOLMQVIST, K.; NYSTRÖM, M.; ANDERSSON, R.; DEWHURST, R.; JARODZKA, H.; VAN DE WEIJER, J. Eye tracking: A comprehensive guide to methods and measures. [S. l.]: Oxford University Press, 2011.

HORNERO CORISCO, Ana María; GONZALEZ-VERA, Pilar; BELTRÁN, Paula Buil. Becoming a techie and improving your English with audiovisual translation. Translation and Translanguaging in Multilingual Contexts, John Benjamins, v. 9, n. 2, p. 160–186, 2023. ISSN 2352-1805. DOI: https://doi.org/10.1075/ttmc.00107.hor. Available from: https://www.jbe-platform.com/content/journals/10.1075/ttmc.00107.hor.

KIM, Hye-Jeong. The Effect of Subtitle Correction Activities on Learner’s English Reading Achievement. Korean Association For Learner-Centered Curriculum And Instruction, v. 23, p. 325–336, 2023. DOI: 10.22251/jlcci.2023.23.7.325.

LEWANDOWSKA, B. RealEye Eye-tracking system Technology Whitepaper. Retrieved August, n. 29, 2019.

LIAO, Sixin; YU, Lili; KRUGER, Jan-Louis; REICHLE, Erik D. The impact of audio on the reading of intralingual versus interlingual subtitles: Evidence from eye movements. Applied Psycholinguistics, v. 43, n. 1, p. 237–269, 2022. DOI: 10.1017/s0142716421000527.

LIAO, Sixin; YU, Lili; KRUGER, Jean-Louis. Using Eye Movements to Study the Reading of Subtitles in Video. Scientific Studies of Reading, v. 25, n. 5, p. 417–435, 2020. DOI: 10.1080/10888438.2020.1823986.

OMAR, Khadija Saleem; MOHAMMED SALMAN, Nahida; VADIVEL, Balachandran. The Impact of English Movies with Standard Subtitles on Enhancing the Writing Skills of EFL Students at Cihan University – Duhok. International Journal of Humanities and Education Development (IJHED), v. 5, p. 45–52, May 2023. DOI: 10.22161/jhed.5.3.7. Available from: https://theshillonga.com/index.php/jhed/article/view/44.

QAZI, Aisha; KHAN, Zahra. Impact on English Language Learners’ Listening Comprehension and Acquisition of Vocabulary Through the Use of Subtitled Videos. Education and Social Sciences Review, v. 3, n. 1, p. 32–43, Feb. 2023. DOI: 10.48112/aessr.v3i1.400.

ROSS, Nicholas M.; KOWLER, Eileen. Eye movements while viewing narrated, captioned, and silent videos. Journal of Vision, v. 13, n. 4, p. 1–1, Mar. 2013. ISSN 1534-7362. DOI: 10.1167/13.4.1.

RUSHWAN, Ismail Mohammed Hamid. The Role of Translation in Developing ESP Learners’ Reading Comprehension Skills- A Case Study of Medical Students at Najran University-KSA. International Journal of Applied Linguistics and English Literature, v. 6, n. 3, p. 243–253, 2017. ISSN 2200-3452. DOI: 10.7575/aiac.ijalel.v.6n.3p.243. Available from: https://journals.aiac.org.au/index.php/IJALEL/article/view/3097.

SCHOTTER, E. R.; RAYNER, K. Eye movements and word recognition during reading. In: ADELMAN, J.S. (ed.). isual word recognition: Meaning and context, individuals and development. [S. l.]: Psychology Press, 2012. p. 73–101.

SZARKOWSKA, Agnieszka; GERBER-MORÓN, Olivia. Viewers can keep up with fast subtitles: Evidence from eye movements. Plos One, Public Library of Science, v. 13, n. 6, e0199331, June 2018. DOI: 10.1371/journal.pone.0199331.

RAGNI, Valentina. More than meets the eye: An eye-tracking study on the effects of translation on the processing and memorisation of reversed subtitles. JoSTrans: The Journal of Specialised Translation, University of Bristol, v. 33, Special issue ”Experimental research and cognition in AVT”, p. 99–128, Jan. 2020. Available from: http://www.jostrans.org/issue33/art_ragni.pdf.

WINKE, Paula; GASS, Susan; SYDORENKO, Tetyana. Factors Influencing the Use of Captions by Foreign Language Learners: An Eye-Tracking Study. The Modern Language Journal, v. 97, n. 1, p. 254–275, 2013. DOI: 10.1111/j.1540-4781.2013.01432.x. Available from: https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1540-4781.2013.01432.x.

XU, Yilan; JORDAN, Eoin; LI, Yuanzhe. Chinese EFL Teachers’ Perceptions of Subtitles’ Effects on English Comprehension and Vocabulary Acquisition: Subtitles’ Effects on English Capability. International Journal of Computer-Assisted Language Learning and Teaching, v. 12, n. 1, p. 1–19, 2022. ISSN 2155-7098. DOI: 10.4018/ijcallt.315622.