常使用ChatGPT写作的人能精准识别AI生成文本
People who frequently use ChatGPT for writing tasks are accurate and robust detectors of AI-generated text
- 高频使用LLM写作的人无需培训即可准确辨识AI文本
- 五人专家集体判断仅错1篇，远超多数检测工具
- 擅长捕捉词汇特征与文本风格等复杂线索，适合研究者参考
本文研究人类对商业大模型（GPT-4o、Claude、o1）生成文本的识别能力。我们招募标注员阅读300篇非虚构英文文章，判断其为人工或AI生成，并提供段落级解释。结果表明，经常使用LLM进行写作的标注员在无任何训练或反馈的情况下，仍能高效识别AI文本。五位此类“专家”集体决策仅错误分类1篇，显著优于我们评估的多数商用及开源检测器，即使面对改写和人工化等规避策略也表现稳健。对专家自由解释的定性分析显示，他们不仅依赖特定词汇线索（如‘AI词汇’），还能察觉形式化、原创性、清晰度等复杂文本特征，这些对自动检测器而言尤为困难。我们公开标注数据集与代码，以推动人类与自动化检测方法的进一步研究。
原文摘要 · Abstract (English)
In this paper, we study how well humans can detect text generated by commercial LLMs (GPT-4o, Claude, o1). We hire annotators to read 300 non-fiction English articles, label them as either human-written or AI-generated, and provide paragraph-length explanations for their decisions. Our experiments show that annotators who frequently use LLMs for writing tasks excel at detecting AI-generated text, even without any specialized training or feedback. In fact, the majority vote among five such "expert" annotators misclassifies only 1 of 300 articles, significantly outperforming most commercial and open-source detectors we evaluated even in the presence of evasion tactics like paraphrasing and humanization. Qualitative analysis of the experts' free-form explanations shows that while they rely heavily on specific lexical clues ('AI vocabulary'), they also pick up on more complex phenomena within the text (e.g., formality, originality, clarity) that are challenging to assess for automatic detectors. We release our annotated dataset and code to spur future research into both human and automated detection of AI-generated text.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。