让大模型学会组合记忆中的多条新知识,解决旧模型无法灵活使用新信息的问题。
Hybrid-Policy Self-Editing for Composable Unstructured Knowledge Editing
- 用模型自身生成的推理轨迹主动学习新知识,而非被动复制输入文本。
- 在4个大模型上验证,新方法能准确回答关于新知识的单跳和多跳问题。
- 适合需要频繁更新知识的场景,如动态问答系统或智能助手。
大型语言模型(LLMs)在自然语言任务中表现卓越,但其知识基于静态语料库,难以应对快速变化的世界。这催生了知识编辑(KE)技术,旨在无需修改无关内容的情况下更新特定知识。近期研究从结构化知识三元组转向非结构化知识编辑(UKE),即通过自由格式段落一次性引入多个事实。然而,现有方法虽能将段落注入模型,却无法有效利用:模型虽能复述该段落,却无法回答其中单个事实相关的问题,也无法将多个事实组合用于多步推理。我们将其归因于编辑器对固定段落的被动依赖。为此,我们提出一种主动自蒸馏机制,利用同一模型在上下文中的特权状态进行无监督学习。我们进一步发现,由于新知识的陌生性,原始模型的推理轨迹很少覆盖这些信息,限制了纯在线策略蒸馏的效果。为弥补这一差距,我们提出混合策略自编辑(HPSE),构建混合推理轨迹,在模型覆盖不足时精准插入缺失事实,其余部分保持在线策略。我们理论上分析了HPSE相比纯在线策略的优势,并在四个大模型骨干和两个编辑器下,多种场景中实证其即插即用的提升效果。
原文摘要 · Abstract (English)
Large language models (LLMs) achieve remarkable performance across natural language tasks, yet they are trained on static corpora and their knowledge quickly becomes outdated in a fast-changing world. This motivates knowledge editing (KE), which updates specific knowledge in an LLM without changing unrelated others. Recent works move from structured knowledge triples toward unstructured KE (UKE), where the edit is a free-form passage that may state multiple facts at once. Nonetheless, existing editors inject such a passage yet fail to use it: the edited model can recall the passage, but can neither answer atomic questions about its facts nor compose them into multi-hop reasoning. We attribute this missing property, which we term composability, to editors' passive reliance on the fixed passage as the sole learning source. In response, we cast editing as a proactive self-distillation from a privileged in-context state of the same model, which requires no external supervision. We further reveal that due to the novelty of the injected knowledge, the pre-edited model's own rollouts rarely cover it, which limits the effectiveness of pure on-policy distillation. To close this gap, we propose HPSE, which builds a hybrid rollout that steps in to place missing facts onto the student's own trajectory precisely where its coverage fails, while staying on-policy elsewhere. We theoretically analyze HPSE's advantage over pure on-policy distillation, and empirically establish its plug-and-play improvements across four LLM backbones and two KE editors under various scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。