测试多种模型在不同场景下识别AI文本的能力，发现无一能全面通用。
Detecting the Machine: A Comprehensive Benchmark of AI-Generated Text Detectors Across Architectures, Domains, and Adversarial Conditions
- 对比20余种检测方法，涵盖传统模型与大模型提示
- 跨领域迁移时性能显著下降，无模型保持稳定表现
- 基于风格的检测器可解释性强，适合需要透明性的场景
大型语言模型（LLMs）的快速普及带来了对机器生成文本检测器的迫切需求。现有基准通常仅在理想条件下评估单一检测器，难以反映真实复杂场景下的表现。本文构建了全面评测基准，涵盖两个语料库：HC3（23,363对人类-ChatGPT文本）和ELI5（15,000对人类-Mistral-7B文本）。评估方法包括经典分类器、微调的Transformer编码器（BERT、RoBERTa、ELECTRA、DistilBERT、DeBERTa-v3）、CNN、XGBoost风格化模型、基于困惑度的检测器，以及以LLM作为检测器的提示方法。结果显示，变压器模型在分布内表现接近完美，但在领域偏移下性能急剧下降。XGBoost风格化模型表现相当且具备可解释性。以大模型为检测器的方法表现不佳，且受生成器-检测器身份偏差影响。困惑度方法出现极性反转现象——现代大模型输出的困惑度反而低于人类文本，但经修正后仍有效。没有任何方法能在不同领域和大模型来源间实现稳健泛化。
原文摘要 · Abstract (English)
The rapid proliferation of large language models (LLMs) has created an urgent need for robust and generalizable detectors of machine-generated text. Existing benchmarks typically evaluate a single detector on a single dataset under ideal conditions, leaving open questions about cross-domain transfer, cross-LLM generalization, and adversarial robustness. We present a comprehensive benchmark evaluating diverse detection approaches across two corpora: HC3 (23,363 human-ChatGPT pairs) and ELI5 (15,000 human-Mistral-7B pairs). Methods include classical classifiers, fine-tuned transformer encoders (BERT, RoBERTa, ELECTRA, DistilBERT, DeBERTa-v3), a CNN, an XGBoost stylometric model, perplexity-based detectors, and LLM-as-detector prompting. Results show that transformer models achieve near-perfect in-distribution performance but degrade under domain shift. The XGBoost stylometric model matches performance while remaining interpretable. LLM-based detectors underperform and are affected by generator-detector identity bias. Perplexity-based methods exhibit polarity inversion, with modern LLM outputs showing lower perplexity than human text, but remain effective when corrected. No method generalizes robustly across domains and LLM sources.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。