对比多种模型检测AI生成学术文本，发现轻量Transformer效果最佳。
Assessing Classical Machine Learning and Transformer-based Approaches for Detecting AI-Generated Research Text
- 用250对摘要数据，测试经典与Transformer模型的检测能力。
- DistilBERT准确率最高，逻辑回归和自定义BERT表现均衡。
- 集成模型不如单一最优模型，说明表征质量比模型数量更重要。
大型语言模型（如ChatGPT）的快速普及模糊了人类与AI生成文本的界限，引发学术诚信、知识产权及虚假信息传播的担忧。因此，可靠的文字生成检测对保障人类原创性与数字交流信任至关重要。本研究通过一个包含250对摘要的标注数据集，评估现有机器学习方法区分ChatGPT-3.5生成文本与人工撰写文本的能力。我们测试并比较了经典方法（基于词袋、词性标注、TF-IDF特征的逻辑回归）与Transformer模型（BERT+N-gram、DistilBERT、BERT轻量自定义分类器、LSTM-N-gram）。目标是评估各模型在检测学术类AI文本上的表现，并检验模型集成是否优于单个模型。结果显示，DistilBERT整体表现最佳，逻辑回归与BERT-Custom提供稳健平衡的替代方案；而LSTM与BERT-N-gram性能较弱。三个最优模型的投票集成未能超越单独的DistilBERT，凸显单一高质量表示的重要性。该研究系统评估了各类检测方法的优劣，为未来基于更大更丰富数据集的鲁棒型Transformer框架奠定基础，以应对持续进化的生成式AI。
原文摘要 · Abstract (English)
The rapid adoption of large language models (LLMs) such as ChatGPT has blurred the line between human and AI-generated texts, raising urgent questions about academic integrity, intellectual property, and the spread of misinformation. Thus, reliable AI-text detection is needed for fair assessment to safeguard human authenticity and cultivate trust in digital communication. In this study, we investigate how well current machine learning (ML) approaches can distinguish ChatGPT-3.5-generated texts from human-written texts employing a labeled data set of 250 pairs of abstracts from a wide range of research topics. We test and compare both classical (Logistic Regression armed with classical Bag-of-Words, POS, and TF-IDF features) and transformer-based (BERT augmented with N-grams, DistilBERT, BERT with a lightweight custom classifier, and LSTM-based N-gram models) ML detection techniques. As we aim to assess each model's performance in detecting AI-generated research texts, we also aim to test whether an ensemble of these models can outperform any single detector. Results show DistilBERT achieves the overall best performance, while Logistic Regression and BERT-Custom offer solid, balanced alternatives; LSTM- and BERT-N-gram approaches lag. The max voting ensemble of the three best models fails to surpass DistilBERT itself, highlighting the primacy of a single transformer-based representation over mere model diversity. By comprehensively assessing the strengths and weaknesses of these AI-text detection approaches, this work lays a foundation for more robust transformer frameworks with larger, richer datasets to keep pace with ever-improving generative AI models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。