用特殊标记符替代冗长示例，高效更新大模型事实知识
Persuasion Tokens for Editing Factual Knowledge in LLMs
- 设计可学习的'说服标记'，模拟原有示例效果
- 在多个数据集和模型上表现优于或媲美原方法
- 标记数量越多效果越好，且对邻近信息干扰小
上下文知识编辑（IKE）是一种更新大语言模型（LLMs）新知识的有前景技术。然而，IKE依赖于冗长且针对特定事实的示范，创建成本高，且占用大量上下文窗口空间。本文提出说服标记（P-Tokens）——经过训练的特殊标记，能复现IKE示范的效果，实现无需事实相关示范的高效知识编辑。我们在两个编辑数据集和三个LLM上评估了P-Tokens，结果表明其性能可媲美甚至超过IKE。进一步发现，编辑性能对干扰项具有鲁棒性，仅产生轻微负面影响；增加P-Tokens数量能提升性能。本工作解决了IKE的关键局限，提供了一种更实用、可扩展的模型编辑方案。
原文摘要 · Abstract (English)
In-context knowledge editing (IKE) is a promising technique for updating Large Language Models (LLMs) with new information. However, IKE relies on lengthy, fact-specific demonstrations which are costly to create and consume significant context window space. In this paper, we introduce persuasion tokens (P-Tokens) -- special tokens trained to replicate the effect of IKE demonstrations, enabling efficient knowledge editing without requiring fact-specific demonstrations. We evaluate P-Tokens across two editing datasets and three LLMs, demonstrating performance comparable to, and often exceeding, IKE. We further find that editing performance is robust to distractors with small negative effects to neighboring facts, and that increasing the number of P-Tokens improves performance. Our work addresses key limitations of IKE and provides a more practical and scalable alternative for editing LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。