分析万篇论文发现,ChatGPT让学术写作更花哨但更难读。
Examining Linguistic Shifts in Academic Writing Before and After the Launch of ChatGPT: A Study on Preprint Papers
- 对比十年论文,检测词汇、句法等语言特征变化
- 新词增多、句子变简单,但连贯性和可读性下降
- 英语弱的学者更依赖工具,计算机领域变化最明显
大型语言模型(如ChatGPT)引发了学术界对其对学术写作影响的担忧。现有研究多采用词频统计与概率分析等量化方法,但很少系统考察其对学术写作语言特征的影响。为此,我们基于arXiv数据集,对过去十年间823,798篇论文摘要进行了大规模语言学分析,考察了模型偏好词汇频率、词汇复杂度、句法复杂度、连贯性、可读性和情感倾向等特征。结果显示,摘要中模型偏好词汇比例显著上升,表明大模型已广泛影响学术写作风格。此外,词汇复杂度和情感表达上升,但句法复杂度下降,说明大模型引入更多新词并简化句子结构。然而,连贯性与可读性显著降低,表现为连接词减少,文本更难理解。分析还发现,英语能力较弱的研究者更倾向于使用大模型以提升摘要逻辑与流畅性。在学科层面,计算机科学领域的写作风格变化最为显著,而数学领域变化较小。
原文摘要 · Abstract (English)
Large Language Models (LLMs), such as ChatGPT, have prompted academic concerns about their impact on academic writing. Existing studies have primarily examined LLM usage in academic writing through quantitative approaches, such as word frequency statistics and probability-based analyses. However, few have systematically examined the potential impact of LLMs on the linguistic characteristics of academic writing. To address this gap, we conducted a large-scale analysis across 823,798 abstracts published in last decade from arXiv dataset. Through the linguistic analysis of features such as the frequency of LLM-preferred words, lexical complexity, syntactic complexity, cohesion, readability and sentiment, the results indicate a significant increase in the proportion of LLM-preferred words in abstracts, revealing the widespread influence of LLMs on academic writing. Additionally, we observed an increase in lexical complexity and sentiment in the abstracts, but a decrease in syntactic complexity, suggesting that LLMs introduce more new vocabulary and simplify sentence structure. However, the significant decrease in cohesion and readability indicates that abstracts have fewer connecting words and are becoming more difficult to read. Moreover, our analysis reveals that scholars with weaker English proficiency were more likely to use the LLMs for academic writing, and focused on improving the overall logic and fluency of the abstracts. Finally, at discipline level, we found that scholars in Computer Science showed more pronounced changes in writing style, while the changes in Mathematics were minimal.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。