用风格分析法识别短文本中人类与大模型生成内容，准确率达98%。
Stylometry recognizes human and LLM-generated texts in short samples
- 基于词法、语法和标点等特征，构建文本风格分析模型。
- 在维基百科短文本上，二分类准确率达79%至98%。
- 适合关注AI内容溯源与版权保护的研究者使用。
本文探讨了风格分析法在区分大型语言模型（LLMs）与人类生成文本中的应用，涉及模型归属、知识产权及伦理AI使用等问题。通过构建基于维基百科的基准数据集，包含人类撰写的词条摘要、纯大模型生成文本（GPT-3.5/4、LLaMa 2/3、Orca、Falcon）、多种摘要方法（T5、BART、Gensim、Sumy）处理结果以及重述方法（Dipper、T5）生成的内容，采用树模型（决策树与LightGBM）结合人工设计特征（StyloMetrix）与自研n-gram特征进行分类。交叉验证结果显示，在7类多分类任务中，最高达0.87的马修斯相关系数；二分类任务中准确率在0.79至1.0之间，其中维基百科与GPT-4组合在平衡数据集上达到0.98准确率。沙普利值解释表明，百科类文本特征、特定高频词及大模型更高的语法一致性是关键判别因素。结果表明，即便面对日益复杂的模型，对特定类型文本仍可有效区分机器与人类生成内容。
原文摘要 · Abstract (English)
The paper explores stylometry as a method to distinguish between texts created by Large Language Models (LLMs) and humans, addressing issues of model attribution, intellectual property, and ethical AI use. Stylometry has been used extensively to characterise the style and attribute authorship of texts. By applying it to LLM-generated texts, we identify their emergent writing patterns. The paper involves creating a benchmark dataset based on Wikipedia, with (a) human-written term summaries, (b) texts generated purely by LLMs (GPT-3.5/4, LLaMa 2/3, Orca, and Falcon), (c) processed through multiple text summarisation methods (T5, BART, Gensim, and Sumy), and (d) rephrasing methods (Dipper, T5). The 10-sentence long texts were classified by tree-based models (decision trees and LightGBM) using human-designed (StyloMetrix) and n-gram-based (our own pipeline) stylometric features that encode lexical, grammatical, syntactic, and punctuation patterns. The cross-validated results reached a performance of up to .87 Matthews correlation coefficient in the multiclass scenario with 7 classes, and accuracy between .79 and 1. in binary classification, with the particular example of Wikipedia and GPT-4 reaching up to .98 accuracy on a balanced dataset. Shapley Additive Explanations pinpointed features characteristic of the encyclopaedic text type, individual overused words, as well as a greater grammatical standardisation of LLMs with respect to human-written texts. These results show -- crucially, in the context of the increasingly sophisticated LLMs -- that it is possible to distinguish machine- from human-generated texts at least for a well-defined text type.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。