对比4种ChatGPT与真人写作,发现大模型文本词多样性不似人类,新模型反而更不像。
Do LLMs produce texts with "human-like" lexical diversity?
- 从6个维度测量文本词多样性,对比AI与真人写作差异
- 新模型(ChatGPT-4.5)词多样性更高但更不像人类,旧模型反而更接近
- 结果对语言教学与AI应用有重要启示
尽管该问题已受广泛实证关注,大语言模型(LLMs)生成文本是否真正具有人类特征仍不明确。本研究从词汇多样性视角切入,比较四种ChatGPT模型(ChatGPT-3.5、ChatGPT-4、ChatGPT-o4 mini、ChatGPT-4.5)与240名母语(L1)及二语(L2)英语写作者(涵盖四个教育水平)的文本表现。每篇文本测量六个词汇多样性维度:总量、丰度、多样-重复、均匀性、差异性和分散性。单因素MANOVA、ANOVA及支持向量机分析显示,所有ChatGPT生成文本在各变量上均显著区别于真人写作,其中ChatGPT-o4 mini与ChatGPT-4.5差异最大。尽管生成字数更少,ChatGPT-4.5在词汇多样性上高于旧模型。真人写作者在教育水平或语言身份上无显著差异。总体表明,当前ChatGPT模型无法生成具有人类特征的词汇多样性文本,且新模型反而更偏离人类模式。研究讨论其对语言教学与相关应用的影响。
原文摘要 · Abstract (English)
The degree to which large language models (LLMs) produce writing that is truly human-like remains unclear despite the extensive empirical attention that this question has received. The present study addresses this question from the perspective of lexical diversity. Specifically, the study investigates patterns of lexical diversity in LLM-generated texts from four ChatGPT models (ChatGPT-3.5, ChatGPT-4, ChatGPT-o4 mini, and ChatGPT-4.5) in comparison with texts written by L1 and L2 English participants (n = 240) across four education levels. Six dimensions of lexical diversity were measured in each text: volume, abundance, variety-repetition, evenness, disparity, and dispersion. Results from one-way MANOVAs, one-way ANOVAs, and Support Vector Machines revealed that the ChatGPT-generated texts differed significantly from human-written texts for each variable, with ChatGPT-o4 mini and ChatGPT-4.5 differing the most. Within these two groups, ChatGPT-4.5 demonstrated higher levels of lexical diversity than older models despite producing fewer tokens. The human writers' lexical diversity did not differ across subgroups (i.e., education, language status). Altogether, the results indicate that ChatGPT models do not produce human-like texts in relation to lexical diversity, and the newer models produce less human-like text than older models. We discuss the implications of these results for language pedagogy and related applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。