arXiv:2505.18774cs.CL2025-05被引 6

提出DiKE方法,让大模型改知识时不会误伤相关但不相关的事实。

Disentangling Knowledge Representations for Large Language Model Editing

  • 将主体表征拆解为相关与无关部分,只更新需要改的部分。
  • 在多个模型上验证,对无关事实的保护效果提升显著。
  • 适合需要精准编辑、避免知识污染的AI应用开发者。

知识编辑已成为高效更新大语言模型(LLMs)内部知识的有前景方案。现有方法虽能有效融入新知识并保留原能力,却无法维持细粒度无关知识——即与编辑知识共享主体但关系或对象不同的事实。这是因为主体表征天然包含多重属性,导致目标知识与细粒度无关知识在表示空间中纠缠,编辑时易被意外修改。为此,我们提出DiKE(Disentangling Knowledge Representations for LLM Editing),包含两个核心组件:知识表征解缠(KRD)模块,将主体表征分解为与目标知识相关和无关的成分;基于解缠的知识编辑(DKE)模块,仅更新相关部分并显式保护无关部分。我们进一步基于矩阵理论推导出闭式、秩一参数更新,实现高效且低侵入性编辑。为严格评估细粒度无关知识的保留能力,我们构建了FINE-KED基准,涵盖不同关系相似度下的细粒度无关知识。跨多个大模型的大量实验表明,DiKE在显著提升细粒度无关知识保留的同时,保持了竞争性的通用编辑性能。

原文摘要 · Abstract (English)

Knowledge Editing has emerged as a promising solution for efficiently updating embedded knowledge in large language models (LLMs). While existing approaches demonstrate effectiveness in integrating new knowledge and preserving the original capabilities of LLMs, they fail to maintain fine-grained irrelevant knowledge, namely facts that share the same subject as edited knowledge but differ in relation and object. This challenge arises because subject representations inherently encode multiple attributes, causing the target and fine-grained irrelevant knowledge to become entangled in the representation space, and thus vulnerable to unintended alterations during editing. To address this, we propose DiKE, a novel approach that Disentangles Knowledge representations for LLM Editing (DiKE). DiKE consists of two key components: a Knowledge Representation Disentanglement (KRD) module that decomposes the subject representation into target-knowledge-related and -unrelated components, and a Disentanglementbased Knowledge Edit (DKE) module that updates only the target-related component while explicitly preserving the unrelated one. We further derive a closedform, rank-one parameter update based on matrix theory to enable efficient and minimally invasive edits. To rigorously evaluate fine-grained irrelevant knowledge preservation, we construct FINE-KED, a new benchmark comprising fine-grained irrelevant knowledge at different levels of relational similarity to the edited knowledge. Extensive experiments across multiple LLMs demonstrate that DiKE substantially improves fine-grained irrelevant knowledge preservation while maintaining competitive general editing performance.

知识编辑大模型解纠缠精准更新

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。