构建新数据集揭示:换人重写文本后,检测器效果大幅下降。
ARB: A Matched Authorship-Rewriting Benchmark Dataset for AI-Text Detector Evaluation

- 用真实人类文本+LLM重写生成四类数据,模拟真实场景。
- 重写后检测器准确率从93%暴跌至15%,降幅超70个百分点。
- 适合关注检测器实际应用效果的研究者和开发者。
现有AI文本检测基准多对比人类文本与直接生成的LLM文本。然而,重写和改写会显著降低检测性能。为填补这一空白,我们构建了作者重写基准数据集ARB,包含1,800篇人类原文(各600篇来自XSum、WritingPrompts、OpenWebText)及四种开源模型(Llama-3.2-3B、Qwen2.5-7B、Mistral-7B、Gemma-2-9B)。每条原文生成四类变体:人类原文(HUMAN)、直接生成(Free-LLM)、LLM重写的原文(H2L)、同模型重写的生成文本(LLM2L)。在严格1%假阳性率下评估五种检测器(FastDetectGPT、Binoculars-falcon-7b、RADAR、BERT-Defense、RoBERTa-Defense)。结果显示,对直接生成文本检测率达91.2%和93.5%,但对经LLM重写的原文仅能检测30.8%和15.1%,下降60-78个百分点;而对同模型重写生成文本仍保持78.3%和83.0%召回率,下降仅10-13点。BERT-Defense与RoBERTa-Defense在所有情形下均低于3%召回率。结果表明,传统基准性能无法反映检测器在人类内容被重写时的表现,尽管其对纯生成文本重写仍具鲁棒性。
原文摘要 · Abstract (English)
Standard AI-text detection benchmarks compare human-written text against text generated directly by large language models (LLMs). While prior work has shown that rewriting and paraphrasing can degrade detector performance, it remains unclear whether performance measured on this conventional benchmark predicts detector behavior when human-authored content is rewritten by an LLM. To address this gap, we introduce Authorship-Rewriting Benchmark (ARB), built from 1,800 human source texts (600 each from XSum, WritingPrompts, and OpenWebText) and four open-weight generators (Llama-3.2-3B, Qwen2.5-7B, Mistral-7B, Gemma-2-9B). Each source item yields four matched variants: human-written (HUMAN), direct LLM generation (Free-LLM), LLM-rewritten human text (H2L), and same-generator LLM-rewritten LLM text (LLM2L). We evaluated five detectors (FastDetectGPT, Binoculars-falcon-7b, RADAR, BERT-Defense, RoBERTa-Defense) at a strict 1%-false-positive operating point (TPR@1%FPR). FastDetectGPT and Binoculars-falcon-7b detected 91.2% and 93.5\% of direct LLM text, but only 30.8% and 15.1% of human text an LLM had rewritten, a drop of 60-78 percentage points. The same detectors retained 78.3% and 83.0% recall when LLM text was rewritten by the same model, a much smaller decline of 10-13 points. RADAR followed the same pattern (66.8% to 12.2%), while BERT-Defense and RoBERTa-Defense stayed below 3% recall across all regimes. These results show that detector performance measured on the conventional human-vs-LLM benchmark does not transfer to human-authored text revised by an LLM, even though the same detectors remain largely robust to LLM-only rewriting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。