arXiv:2510.27400cs.CLcs.AI2025-10

让大模型同时更新记忆模块和注意力模块,提升知识修改效果。

Balancing Knowledge Updates: Toward Unified Modular Editing in LLMs

  • 联合更新MLP与注意力模块,打破只改记忆单元的局限。
  • 在多个基准上编辑成功率更高,且保留旧知识更优。
  • 适合需要精准、稳定知识更新的研究者与开发者。

知识编辑已成为高效更新大语言模型事实知识的方法,通常定位知识存储模块并修改其参数。然而,现有方法多聚焦于多层感知机(MLP)模块,而忽视了注意力(Attn)模块的作用。本研究对先进大模型进行系统性知识定位实验,发现注意力模块在早期层中对事实知识存储与检索有显著贡献。基于此,我们提出IntAttn-Edit方法,将关联记忆范式扩展至同时更新MLP与注意力模块,并采用知识平衡策略,按各模块对知识存储的实际贡献分配更新强度。在标准基准测试中,IntAttn-Edit在编辑成功率、泛化能力与知识保留方面均优于已有方法。进一步分析表明,该平衡策略使编辑性能在多种场景下保持在最优区间。

原文摘要 · Abstract (English)

Knowledge editing has emerged as an efficient approach for updating factual knowledge in large language models (LLMs). It typically locates knowledge storage modules and then modifies their parameters. However, most existing methods focus on the weights of multilayer perceptron (MLP) modules, which are often identified as the main repositories of factual information. Other components, such as attention (Attn) modules, are often ignored during editing. This imbalance can leave residual outdated knowledge and limit editing effectiveness. We perform comprehensive knowledge localization experiments on advanced LLMs and find that Attn modules play a substantial role in factual knowledge storage and retrieval, especially in earlier layers. Based on these insights, we propose IntAttn-Edit, a method that extends the associative memory paradigm to jointly update both MLP and Attn modules. Our approach uses a knowledge balancing strategy that allocates update magnitudes in proportion to each module's measured contribution to knowledge storage. Experiments on standard benchmarks show that IntAttn-Edit achieves higher edit success, better generalization, and stronger knowledge preservation than prior methods. Further analysis shows that the balancing strategy keeps editing performance within an optimal range across diverse settings.

知识编辑大模型注意力机制模块化更新

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。