Evaluation of writing quality in argumentative essays

comparison between Large Language Models (LLM) and human reviews based on a rubric

Abstract

Automated handwriting assessment has established itself as a promising alternative to streamline and improve feedback in learning due to the development of Generative Artificial Intelligence (GAI) with Large Language Models (LLMs) such as OpenAI's ChatGPT. Thus, the formative feedback provided by AI has the potential to influence self-regulation and continuous improvement in student academic performance. In this context, the objective of this study is to determine the degree of inter-rater reliability in the evaluation of argumentative essays performed by language models (GPTo3-mini-high and GPT-4), comparing it with the evaluation of a human expert using the RUBRIAR analytical rubric. A quantitative approach was used to determine the degree of agreement between the evaluations made by the expert and those made by the customized ChatGPT models on 46 argumentative essays written by first-year university students. The findings reveal that the evaluation performed by the LLMs is similar to that of the human expert, especially in the dimension of gender adjustment in the subdimension of communicative purpose. However, there is low accuracy in cohesion and coherence relationships, as well as in conformity with language norms. It is concluded that it is essential to incorporate a pedagogical approach that promotes an intentional, reflective, and ethical use of AI tools, particularly during the early years of higher education, when students are building the foundations of their academic competencies.