arXiv:2605.02620cs.CLcs.LG2026-05被引 1

用智能体重做一篇风格检测论文,发现大模型能有效骗过检测器。

Beating the Style Detector: Three Hours of Agentic Research on the AI-Text Arms Race

论文配图:Beating the Style Detector: Three Hours of Agentic Research on the AI-Text Arms Race
图 1 · 摘自论文原文
  • 让智能体自动研究并复现原实验,人类仅作监督。
  • 大模型在324个任务中减少71%~75%的风格差距,远超人工修改。
  • 只需20轮反馈,就能让模型伪装成人类,降低被检测概率。

以往复现实验需数周,如今借助现代智能体研究框架与已公开数据,我们重新执行了2026年ACL论文中关于大模型草稿风格后编辑的所有实验,并新增三项。人类仅作为审阅者介入。成功复现全部七项预注册假设,恢复感知自相似性与嵌入自相似性间的相关性(r=+0.244, p<10⁻⁸, n=648)。在无泄露的保留协议下,GPT-5.5与Claude Opus 4.7在324对任务中缩小71%~75%的风格差距,接近同作者上限,而人工编辑仅达24%,且在约80%任务中胜出。进一步将数据视为对抗检测的军备竞赛:基于LUAR-MUD嵌入的留作者外线性SVM达到AUC 0.93~1.00;六项诊断显示,GPT-5.5检测主要受文本长度干扰,而Opus检测反映真实风格特征。在固定检测器下进行T=20轮反馈,一个Opus智能体成功将五组测试模仿样本中的两组翻转至人类分布区,并使所有边距缩小一个数量级。表明在适度努力下,前沿大模型已可高效降低自身被检测概率。所有代码、648份模仿草稿、训练好的检测器、诊断结果及对抗轨迹均已开源。

原文摘要 · Abstract (English)

Reproducing an empirical NLP study used to take weeks. Given the released data and a modern agentic-research harness, we redo every experiment of a recent ACL\,2026 study on personal-style post-editing of LLM drafts -- and add three new ones -- with the human investigator acting only as a reviewer-in-the-loop. We reproduce all seven preregistered hypotheses and recover the paper's headline correlation between perceived self-similarity and embedding-measured self-similarity to three decimal places ($r{=}{+}0.244$, $p{<}10^{-8}$, $n{=}648$). Under a leakage-free held-out protocol, GPT-5.5 and Claude\,Opus\,4.7 close $71$--$75\,\%$ of the style gap to the same-author ceiling on $324$ paired tasks, against $24\,\%$ for the human post-edit, and beat the human post-edit on $\sim$$80\,\%$ of tasks. We then frame the same data as an AI-text detection arms race. A leave-authors-out linear SVM on LUAR-MUD embeddings reaches AUC $0.93$--$1.00$ across approaches; six diagnostics show that GPT-5.5 detection is mostly a length confound while Opus detection is a genuine stylistic signature. Given $T{=}20$ feedback iterations against the frozen detector, an Opus agent flips two of five held-out test mimics to the human half-space and shrinks every margin by an order of magnitude. With moderate effort against a known detector, a frontier LLM can already efficiently lower its own AI-detection probability. All code, $648$ mimic drafts, trained detectors, diagnostics, and adversarial trajectories are released.

AI检测风格伪造智能体研究

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。