Exploring learner prompting and AI feedback quality in L2 Portuguese writing

Mu You; Jing  Zhang; Chuihui Lu; Ana Cristina Ferreira de Almeida Rodrigues Alves; Jiaxu Zuo

doi:10.1590/1983-3652.2026.63188

Autores

Mu You Macau Millennium College, Institute of International Language Services Studies, Macau SAR, China https://orcid.org/0000-0001-5029-3903
Jing Zhang University of Macau, Faculty of Arts and Humanities, Department of Portuguese, Macau SAR, China https://orcid.org/0000-0003-4801-6354
Chuihui Lu University of Macau, Faculty of Arts and Humanities, Department of Portuguese, Macau SAR, China https://orcid.org/0000-0001-7744-9197
Ana Cristina Ferreira de Almeida Rodrigues Alves Macau Scientific and Cultural Centre, Lisbon, Portugal https://orcid.org/0000-0002-1461-4746
Jiaxu Zuo University of Macau, Faculty of Science and Technology, Department of Computer and Information Science, Macau SAR, China https://orcid.org/0009-0009-9950-3348

DOI:

https://doi.org/10.1590/1983-3652.2026.63188

Palavras-chave:

Revisão de escrita assistida por IA, Comportamento de prompting, Precisão das revisões, Português L2

Resumo

Para aprendizes de Português como L2 em Macau, a inteligência artificial (IA) tornou-se uma fonte principal de feedback imediato na escrita para estudantes universitários chineses. Este estudo investiga as práticas autênticas de prompting dos aprendizes e a precisão das revisões geradas por IA na escrita em português L2. Quarenta e seis estudantes do segundo ano da licenciatura em Estudos Portugueses, divididos num grupo de controle e num grupo com condição de justificação (que exigia que racionalizassem cada alteração), revisaram uma composição de uma estudante do primeiro ano utilizando as suas ferramentas de IA preferidas. Os resultados revelam que os estudantes utilizaram predominantemente o chinês nos prompts, empregaram linguagem altamente antropomórfica e cortês, recorreram a pedidos genéricos ou centrados na gramática e, frequentemente, apresentaram desalinhamento entre os objetivos pretendidos e os prompts} efetivamente formulados. A obrigatoriedade de justificar as revisões aumentou o número de turnos de interação e deslocou a atenção para segmentos textuais mais reduzidos e para a busca de explicações, embora também tenha gerado taxas de erro mais elevadas nas revisões. Experiências com prompts desenhados pelos investigadores demonstraram o potencial de estratégias simples de engenharia de prompt, sendo a técnica Rephrase-and-Respond a que produziu as revisões mais precisas. Os resultados sublinham a imaturidade do prompting não treinado, a natureza sensível ao contexto da engenharia de prompt e o valor pedagógico prático das atuais ferramentas de IA mesmo quando usados sem treinamento prévio. Para promover uma colaboração humano–IA crítica e reflexiva na escrita em L2, recomendam-se intervenções pedagógicas como a instrução explícita em estratégias de prompting e a utilização de tarefas que exijam justificação ou busca de explicações.

Downloads

Os dados de download ainda não estão disponíveis.

Referências

MMARI, Tawfiq; CHEN, Meilun; ZAMAN, S M Mehedi; GARIMELLA, Kiran. How Students (Really) Use ChatGPT: Uncovering Experiences Among Undergraduate Students. [S. l.: s. n.], 2026. arXiv: 2505.24126 [cs.HC]. Available from: https://arxiv.org/abs/2505.24126.

BARROT, Jessie S. Using ChatGPT for Second Language Writing: Pitfalls and Potentials. Assessing Writing, Elsevier, v. 57, p. 100745, 2023. ISSN 1075-2935. DOI: 10.1016/j.asw.2023.100745.

BLACK, R.W.; TOMLINSON, B. University Students Describe How They Adopt AI for Writing and Research in a General Education Course. Scientific Reports, Nature Portfolio, v. 15, n. 1, p. 8799, 2025. DOI: 10.1038/s41598-025-92937-2.

CROMPTON, Helen; BURKE, Diane. Artificial Intelligence in Higher Education: The State of the Field. International Journal of Educational Technology in Higher Education, Springer, v. 20, n. 1, p. 22, 2023. DOI: 10.1186/s41239-023-00392-8.

DAI, Wei; TSAI, Yi-Shan; LIN, Jionghao; ALDINO, Ahmad; JIN, Hua; LI, Tongguang; GAŠEVIĆ, Dragan; CHEN, Guanliang. Assessing the Proficiency of Large Language Models in Automatic Feedback Generation: An Evaluation Study. Computers and Education: Artificial Intelligence, Elsevier, v. 7, p. 100299, 2024. ISSN 2666-920X. DOI: 10.1016/j.caeai.2024.100299.

DENG, Yihe; ZHANG, Weitong; CHEN, Zixiang; GU, Quanquan. Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves, 2024. arXiv: 2311.04205 [cs.CL]. Available from: https://arxiv.org/abs/2311.04205.

EAGER, Bronwyn; BRUNTON, Ryan. Prompting Higher Education Towards AI-Augmented Teaching and Learning Practice. Journal of University Teaching and Learning Practice, v. 20, n. 5, Sept. 2023. DOI: 10.53761/1.20.5.02.

EFRON, Bradley. Bootstrap Methods: Another Look at the Jackknife. In: Breakthroughs in Statistics: Methodology and Distribution. Ed. by Samuel Kotz and Norman L. Johnson. New York, NY: Springer, 1992. v. 2, p. 569–593. ISBN 978-1-4612-4380-9. DOI: 10.1007/978-1-4612-4380-9_41.

GUO, Kai; WANG, Deliang. To Resist It or To Embrace It? Examining ChatGPT’s Potential to Support Teacher Feedback in EFL Writing. Education and Information Technologies, Springer, v. 29, n. 7, p. 8435–8463, 2023. DOI: 10.1007/s10639-023-12146-0.

HSIEH, Hsiu-Fang; SHANNON, Sarah E. Three Approaches to Qualitative Content Analysis. Qualitative Health Research, Sage Journals, v. 15, p. 1277–1288, 9 2005. DOI: 10.1177/1049732305276687.

HWANG, Myunghwan; JEENS, Robert; LEE, Hee-Kyung. Exploring Learner Prompting Behavior and Its Effect on ChatGPT-Assisted English Writing Revision. The Asia-Pacific Education Researcher, Springer, v. 34, n. 3, p. 1157–1167, 2024. DOI: 10.1007/s40299-024-00930-6.

HWANG, Yohan; LEE, Jang Ho; SHIN, Dongkwang. What is Prompt Literacy? An Exploratory Study of Language Learners’ Development of New Literacy Skill Using Generative AI, 2023. arXiv: 2311.05373 [cs.HC]. Available from: https://arxiv.org/abs/2311.05373.

JELSON, Andrew; MANESH, Daniel; JANG, Alice; DUNLAP, Daniel; KIM, Young-Ho; LEE, Sang Won. An Empirical Study to Understand How Students Use ChatGPT for Writing Essays. [S. l.: s. n.], 2025. arXiv: 2501.10551 [cs.HC]. Available from: https://arxiv.org/abs/2501.10551.

KNOTH, Nils; TOLZIN, Antonia; JANSON, Andreas; LEIMEISTER, Jan Marco. AI Literacy and Its Implications for Prompt Engineering Strategies. Computers and Education: Artificial Intelligence, Elsevier, v. 6, p. 100225, 2024. ISSN 2666-920X. DOI: 10.1016/j.caeai.2024.100225.

KOHNKE, Lucas; MOORHOUSE, Benjamin Luke; ZOU, Di. ChatGPT for Language Teaching and Learning. RELC Journal, Sage Journals, v. 54, n. 2, p. 537–550, 2023. DOI: 10.1177/00336882231162868.

KOLTOVSKAIA, Svetlana; RAHMATI, Payam; SAELI, Hooman. Graduate Students’ Use of ChatGPT for Academic Text Revision: Behavioral, Cognitive, and Affective Engagement. Journal of Second Language Writing, v. 65, p. 101130, 2024. ISSN 1060-3743. DOI: 10.1016/j.jslw.2024.101130.

LIN, Shiming; CROSTHWAITE, Peter. The Grass is not Always Greener: Teacher vs. GPT-Assisted Written Corrective Feedback. System, Elsevier, v. 127, p. 103529, 2024. ISSN 0346-251X. DOI: 10.1016/j.system.2024.103529.

LO, Chung Kwan. What Is the Impact of ChatGPT on Education? A Rapid Review of the Literature. Education Sciences, MDPI, v. 13, n. 4, 2023. ISSN 2227-7102. DOI: 10.3390/educsci13040410.

LU, Qi; YAO, Yuan; XIAO, Longhai; YUAN, Mingzhu; WANG, Jue; ZHU, Xinhua. Can ChatGPT Effectively Complement Teacher Assessment of Undergraduate Students’ Academic Writing? Assessment & Evaluation in Higher Education, Taylor & Francis, v. 49, n. 5, p. 616–633, 2024. DOI: 10.1080/02602938.2024.2301722.

LUO, Ying; XIA, Yuwei; LU, Xiaofei. A Systematic Review of the Role of Artificial Intelligence in Second Language Writing Education. Digital Studies in Language and Literature, De Gruyter, 2025. DOI: 10.1515/dsll-2025-0001.

REIGSTAD, Thomas J.; MCANDREW, Donald A. Training tutors for writing conferences. [S. l.]: ERIC Clearinghouse on Reading and Communication Skills, 1984.

SAHOO, Pranab; SINGH, Ayush Kumar; SAHA, Sriparna; JAIN, Vinija; MONDAL, Samrat; CHADHA, Aman. A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications. [S. l.: s. n.], 2025. arXiv: 2402.07927 [cs.AI]. Available from: https://arxiv.org/abs/2402.07927.

SCHULHOFF, Sander; ILIE, Michael; BALEPUR, Nishant; KAHADZE, Konstantine; LIU, Amanda; SI, Chenglei; LI, Yinheng; GUPTA, Aayush; HAN, HyoJung; SCHULHOFF, Sevien; DULEPET, Pranav Sandeep; VIDYADHARA, Saurav; KI, Dayeon; AGRAWAL, Sweta; PHAM, Chau; KROIZ, Gerson; LI, Feileen; TAO, Hudson; SRIVASTAVA, Ashay; COSTA, Hevander Da; GUPTA, Saloni; ROGERS, Megan L.; GONCEARENCO, Inna; SARLI, Giuseppe; GALYNKER, Igor; PESKOFF, Denis; CARPUAT, Marine; WHITE, Jules; ANADKAT, Shyamal; HOYLE, Alexander; RESNIK, Philip. The Prompt Report: A Systematic Survey of Prompt Engineering Techniques. [S. l.: s. n.], 2025. arXiv: 2406.06608 [cs.CL]. Available from: https://arxiv.org/abs/2406.06608.

STEISS, Jacob; TATE, Tamara; GRAHAM, Steve; CRUZ, Jazmin; HEBERT, Michael; WANG, Jiali; MOON, Youngsun; TSENG, Waverly; WARSCHAUER, Mark; OLSON, Carol Booth. Comparing the Quality of Human and ChatGPT Feedback of Students’ Writing. Learning and Instruction, Elsevier, v. 91, p. 101894, 2024. ISSN 0959-4752. DOI: 10.1016/j.learninstruc.2024.101894.

TIAN, Haoye; WANG, Chong; YANG, BoYang; ZHANG, Lyuye; LIU, Yang. A Taxonomy of Prompt Defects in LLM Systems. [S. l.: s. n.], 2025. arXiv: 2509.14404 [cs.SE]. Available from: https://arxiv.org/abs/2509.14404.

YOU, Mu; ZHANG, Jing; WONG, Derek F.; LAN, Kaixin. UMPLC: the First Longitudinal Learner Corpus of Portuguese. Language Resources and Evaluation, Springer, v. 59, p. 3353–3372, 2025. DOI: 10.1007/s10579-025-09811-w.

ZAMFIRESCU-PEREIRA, J.D.; WONG, Richmond Y.; HARTMANN, Bjoern; YANG, Qian. Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts. In: PROCEEDINGS of the 2023 CHI Conference on Human Factors in Computing Systems. Hamburg, Germany: Association for Computing Machinery, 2023. (CHI ’23). ISBN 9781450394215. DOI: 10.1145/3544548.3581388.

ZHENG, Mingqian; PEI, Jiaxin; LOGESWARAN, Lajanugen; LEE, Moontae; JURGENS, David. When “A Helpful Assistant” Is Not Really Helpful: Personas in System Prompts Do Not Improve Performances of Large Language Models. Ed. by Yaser Al-Onaizan, Mohit Bansal and Yun-Nung Chen. Miami, Florida, USA: Association for Computational Linguistics, Nov. 2024. p. 15126–15154. DOI: 10.18653/v1/2024.findings-emnlp.888.