构建维基百科专用文本生成检测基准,提升真实场景下机器文本识别能力。
WETBench: A Benchmark for Detecting Task-Specific Machine-Generated Text on Wikipedia
- 针对维基编辑常见任务设计三类生成场景,覆盖段落写作、摘要与风格转换。
- 基于多语言、多生成器数据测试发现,训练型检测器平均准确率78%,零样本仅58%。
- 为编辑辅助场景下的生成内容检测提供可复现评估标准,适合可信内容研究者使用。
鉴于维基百科作为高质量可靠内容来源的重要地位,由大语言模型生成的低质量机器文本(MGT)泛滥问题日益引发关注。可靠的MGT检测至关重要,但现有研究主要在通用生成任务上评估检测器,与维基百科编辑的实际使用场景存在偏差,导致实际应用中泛化能力不足。为此,我们提出WETBench,一个涵盖多语言、多生成器且任务特定的MGT检测基准。基于编辑真实使用场景,定义了三种编辑任务:段落写作、摘要生成和文本风格转换,并构建两个新数据集覆盖三种语言。对每项任务测试三种提示策略,采用最优提示生成多模型文本,评估多种检测器表现。结果显示,在各类设置下,训练型检测器平均准确率为78%,而零样本检测器仅为58%。结果表明,现有检测器在真实编辑场景中仍面临挑战,强调必须在多样化的任务特定数据上评估检测模型,以衡量其在编辑驱动环境中的可靠性。
原文摘要 · Abstract (English)
Given Wikipedia's role as a trusted source of high-quality, reliable content, concerns are growing about the proliferation of low-quality machine-generated text (MGT) produced by large language models (LLMs) on its platform. Reliable detection of MGT is therefore essential. However, existing work primarily evaluates MGT detectors on generic generation tasks rather than on tasks more commonly performed by Wikipedia editors. This misalignment can lead to poor generalisability when applied in real-world Wikipedia contexts. We introduce WETBench, a multilingual, multi-generator, and task-specific benchmark for MGT detection. We define three editing tasks, empirically grounded in Wikipedia editors' perceived use cases for LLM-assisted editing: Paragraph Writing, Summarisation, and Text Style Transfer, which we implement using two new datasets across three languages. For each writing task, we evaluate three prompts, generate MGT across multiple generators using the best-performing prompt, and benchmark diverse detectors. We find that, across settings, training-based detectors achieve an average accuracy of 78%, while zero-shot detectors average 58%. These results show that detectors struggle with MGT in realistic generation scenarios and underscore the importance of evaluating such models on diverse, task-specific data to assess their reliability in editor-driven contexts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。