Coherence in the Parediction of Dicendi Verbs in Abstracts

A Comparison Between Human Capabilities and Current Language Models

Abstract

The relationship between technology and human language is a topic of growing debate, especially with the emergence of large-scale language models (LLMs) and their textual production capacity. This research is justified by the importance of verifying the proximity between the linguistic productions of LLMs and humans, based on the assumption that human language is shaped by epilinguistic knowledge. To this end, we conducted a comparative study in which we asked students and LLMs to suggest verbs (dicendi) to coherently fill gaps in a summary of a source text. The choice of a school summary was motivated by the need for a deep understanding of the original text and textual coherence, which represents a challenge for both humans and LLMs. The methodology included the comparison of the students’ and 5 LLMs suggestions via chat, seeking to assess coherence with the source text. The results show that LLMs and humans presented good suggestions in the gaps. However, the results contradict the initial hypothesis that humans would present greater lexical diversity. On the other hand, the results also indicate that LLMs perform better in terms of local and global coherence. These facts suggest that LLMs can be a support tool for studies related to the intersection between language and technology, including for teaching.