arXiv:2412.17242cs.AIcs.CL2024-12KDD被引 13

针对学术写作中机器生成文本检测的泛化与适应能力,构建大规模数据集并评测多种检测方法。

On the Generalization and Adaptation Ability of Machine-Generated Text Detectors in Academic Writing

  • 构建包含3360万词元的MGT-Acedemic数据集,覆盖多学科领域。
  • 发现跨领域检测性能下降明显,且归属识别任务更具挑战性。
  • 提出新型持续学习场景,验证不同适配策略在少样本下的表现。

大型语言模型(LLMs)的普及引发了对机器生成文本(MGT)的担忧,尤其在学术领域,如抄袭和虚假信息等问题日益突出。因此,开发具有高度泛化性和适应性的MGT检测系统成为迫切需求。鉴于LLMs常被滥用于学术写作,本文从三个关键方面研究MGT检测器在学术写作中的泛化与适应能力:首先,构建MGT-Acedemic数据集,包含超过336百万词元和74.9万样本,涵盖理工、人文与社会科学领域的真人写作(HWT)与机器生成文本,并提供可扩展的代码框架以支持高效基准测试。其次,在域内与跨域设置下,对多种检测器进行二分类与归属任务的性能评估,揭示了归属任务常被忽视的挑战。第三,引入一种新型归属任务,要求模型在无(或极少)历史训练数据条件下,通过少量或大量样本实现对新类别的适应。我们实现了八种不同适配技术,显著提升了性能,并凸显该任务的内在复杂性。研究结果为理解MGT检测器在多样化场景下的泛化与适应能力提供了洞见,为构建鲁棒、自适应的检测系统奠定基础。代码框架已开源:https://github.com/Y-L-LIU/MGTBench-2.0。

原文摘要 · Abstract (English)

The rising popularity of large language models (LLMs) has raised concerns about machine-generated text (MGT), particularly in academic settings, where issues like plagiarism and misinformation are prevalent. As a result, developing a highly generalizable and adaptable MGT detection system has become an urgent priority. Given that LLMs are most commonly misused in academic writing, this work investigates the generalization and adaptation capabilities of MGT detectors in three key aspects specific to academic writing: First, we construct MGT-Acedemic, a large-scale dataset comprising over 336M tokens and 749K samples. MGT-Acedemic focuses on academic writing, featuring human-written texts (HWTs) and MGTs across STEM, Humanities, and Social Sciences, paired with an extensible code framework for efficient benchmarking. Second, we benchmark the performance of various detectors for binary classification and attribution tasks in both in-domain and cross-domain settings. This benchmark reveals the often-overlooked challenges of attribution tasks. Third, we introduce a novel attribution task where models have to adapt to new classes over time without (or with very limited) access to prior training data in both few-shot and many-shot scenarios. We implement eight different adapting techniques to improve the performance and highlight the inherent complexity of the task. Our findings provide insights into the generalization and adaptation ability of MGT detectors across diverse scenarios and lay the foundation for building robust, adaptive detection systems. The code framework is available at https://github.com/Y-L-LIU/MGTBench-2.0.

文本检测学术写作持续学习大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。