采样策略变化让机器生成文本检测失效，最高导致准确率降至1%
How Sampling Affects the Detectability of Machine-written texts: A Comprehensive Study
- 系统测试37种采样参数组合，研究其对检测效果的影响
- 温度、top-p等微调使检测器AUROC从99%暴跌至1%
- 揭示现有检测方法脆弱性，适合检测研究者参考
随着大语言模型生成文本日益普遍且与人类写作难以区分，自动文本检测研究备受关注。许多近期检测器报告近乎完美的准确率，常达99%以上AUROC。然而，这些结论通常基于固定生成设置，未考察解码策略变化的影响。本文系统研究采样解码对检测可识别性的影响，重点关注模型（子）词级分布的细微变化如何影响检测性能。结果发现，即使微调温度、top-p或核采样等参数，也能严重削弱检测准确率，部分情况下AUROC骤降至1%。研究揭示当前检测方法的关键盲点，强调需建立更全面的评估体系。为促进后续研究，我们发布包含37种解码配置的大规模数据集，附代码与评估框架：https://github.com/BaggerOfWords/Sampling-and-Detection
原文摘要 · Abstract (English)
As texts generated by Large Language Models (LLMs) are ever more common and often indistinguishable from human-written content, research on automatic text detection has attracted growing attention. Many recent detectors report near-perfect accuracy, often boasting AUROC scores above 99\%. However, these claims typically assume fixed generation settings, leaving open the question of how robust such systems are to changes in decoding strategies. In this work, we systematically examine how sampling-based decoding impacts detectability, with a focus on how subtle variations in a model's (sub)word-level distribution affect detection performance. We find that even minor adjustments to decoding parameters - such as temperature, top-p, or nucleus sampling - can severely impair detector accuracy, with AUROC dropping from near-perfect levels to 1\% in some settings. Our findings expose critical blind spots in current detection methods and emphasize the need for more comprehensive evaluation protocols. To facilitate future research, we release a large-scale dataset encompassing 37 decoding configurations, along with our code and evaluation framework https://github.com/BaggerOfWords/Sampling-and-Detection
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。