无需训练即可检测文本是否由特定大模型生成,误判率极低。
A Training-free Method for LLM Text Attribution
- 基于文本生成的序列依赖性,设计零样本统计检验方法。
- 文本越长,误判率指数级下降,理论保证强。
- 适用于教育、平台等场景,应对伪造内容与合规监管。
验证内容来源对教育机构、社交媒体平台和企业至关重要。随着大语言模型(LLMs)生成的文本几乎无法与人工内容区分,这一问题日益严峻。许多机构使用自研模型,需防止外部非授权模型生成内容。本文回答:给定一段文本,能否在保证极低假阳性率的前提下,识别其是否由特定模型生成?我们将LLM文本建模为完全依赖历史的随机过程,设计零样本统计检验,用于(i)区分两组已知模型A(非授权)与B(内部)生成的文本,(ii)判断文本是否由已知模型或任意未知模型生成。理论上证明,该检验的类型I和类型II错误随文本长度指数衰减。我们还扩展至黑盒采样场景,给出达到白盒性能所需的样本量,并通过信息论下界证明上界紧致性。数值实验验证理论结果,并评估对抗后编辑下的鲁棒性。本工作可广泛应用于内容溯源、打击虚假信息及满足新兴AI监管要求。代码与演示见https://github.com/TaraRadvand74/llm-text-detection。
原文摘要 · Abstract (English)
Verifying the provenance of content is crucial to the functioning of many organizations, e.g., educational institutions, social media platforms, and firms. This problem is becoming increasingly challenging as text generated by Large Language Models (LLMs) becomes almost indistinguishable from human-generated content. In addition, many institutions use in-house LLMs and want to ensure that external, non-sanctioned LLMs do not produce content within their institutions. In this paper, we answer the following question: Given a piece of text, can we identify whether it was produced by a particular LLM, while ensuring a guaranteed low false positive rate? We model LLM text as a sequential stochastic process with complete dependence on history. We then design zero-shot statistical tests to (i) distinguish between text generated by two different known sets of LLMs $A$ (non-sanctioned) and $B$ (in-house), and (ii) identify whether text was generated by a known LLM or by any unknown model. We prove that the Type I and Type II errors of our test decrease exponentially with the length of the text. We also extend our theory to black-box access via sampling and characterize the required sample size to obtain essentially the same Type I and Type II error upper bounds as in the white-box setting (i.e., with access to $A$). We show the tightness of our upper bounds by providing an information-theoretic lower bound. We next present numerical experiments to validate our theoretical results and assess their robustness in settings with adversarial post-editing. Our work has a host of practical applications in which determining the origin of a text is important and can also be useful for combating misinformation and ensuring compliance with emerging AI regulations. See https://github.com/TaraRadvand74/llm-text-detection for code, data, and an online demo of the project.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。