发现大模型输出有独特指纹，可精准识别来源模型。
Idiosyncrasies in Large Language Models
- 用生成文本的嵌入表示做分类，无需复杂特征
- 五模型分类准确率达97.1%，跨改写翻译仍有效
- 适合关注模型溯源、合成数据安全的研究者
本文揭示并研究了大型语言模型（LLMs）中的独特输出模式——可区分不同模型的个性化特征。我们设计了一个简单分类任务：给定一段文本输出，判断其来源模型。在多个模型组上评估该任务，发现仅需对文本嵌入模型进行微调，即可实现高精度分类。在包含ChatGPT、Claude、Grok、Gemini和DeepSeek的五类模型分类中，验证集准确率达到97.1%。进一步分析表明，这些独特模式源于词级分布，即使文本经重写、翻译或摘要后依然存在，说明其已编码于语义内容中。我们还利用大模型作为评判者，生成各模型独特性的详细描述。最后讨论了其对合成数据训练、模型相似性推断及鲁棒评估的广泛影响。代码开源于https://github.com/locuslab/llm-idiosyncrasies。
原文摘要 · Abstract (English)
In this work, we unveil and study idiosyncrasies in Large Language Models (LLMs) -- unique patterns in their outputs that can be used to distinguish the models. To do so, we consider a simple classification task: given a particular text output, the objective is to predict the source LLM that generates the text. We evaluate this synthetic task across various groups of LLMs and find that simply fine-tuning text embedding models on LLM-generated texts yields excellent classification accuracy. Notably, we achieve 97.1% accuracy on held-out validation data in the five-way classification problem involving ChatGPT, Claude, Grok, Gemini, and DeepSeek. Our further investigation reveals that these idiosyncrasies are rooted in word-level distributions. These patterns persist even when the texts are rewritten, translated, or summarized by an external LLM, suggesting that they are also encoded in the semantic content. Additionally, we leverage LLM as judges to generate detailed, open-ended descriptions of each model's idiosyncrasies. Finally, we discuss the broader implications of our findings, including training on synthetic data, inferring model similarity, and robust evaluation of LLMs. Code is available at https://github.com/locuslab/llm-idiosyncrasies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。