arXiv:2502.03748cs.CL2025-02NeurIPS被引 7

改进残差分布机制,让大模型知识编辑更准更稳

Rethinking Residual Distribution in Locate-then-Edit Model Editing

  • 提出边界层更新策略,优化编辑时的残差分配方式
  • 在3个模型2个数据集上平均提升35.59%,精度显著提高
  • 适合需要精准修改大模型知识且不破坏原有能力的研究者

模型编辑可实现对大语言模型(LLMs)知识的精准更新,而定位-编辑方法通过识别关键层、计算目标编辑对应的残差,并基于最小二乘法进行多层残差分布更新。然而我们发现,该核心机制存在反直觉的误差:残差分布会引入权重偏移,且误差随分布距离、批量大小和编辑序列长度增加,导致编辑不准确。为此,我们提出边界层更新(BLUE)策略。在三个大模型和两个数据集上的顺序批量编辑实验表明,BLUE不仅使平均性能提升35.59%,达到当前最佳水平,还有效保持了模型的通用能力。代码已开源。

原文摘要 · Abstract (English)

Model editing enables targeted updates to the knowledge of large language models (LLMs) with minimal retraining. Among existing approaches, locate-then-edit methods constitute a prominent paradigm: they first identify critical layers, then compute residuals at the final critical layer based on the target edit, and finally apply least-squares-based multi-layer updates via $\textbf{residual distribution}$. While empirically effective, we identify a counterintuitive failure mode: residual distribution, a core mechanism in these methods, introduces weight shift errors that undermine editing precision. Through theoretical and empirical analysis, we show that such errors increase with the distribution distance, batch size, and edit sequence length, ultimately leading to inaccurate or suboptimal edits. To address this, we propose the $\textbf{B}$oundary $\textbf{L}$ayer $\textbf{U}$pdat$\textbf{E (BLUE)}$ strategy to enhance locate-then-edit methods. Sequential batch editing experiments on three LLMs and two datasets demonstrate that BLUE not only delivers an average performance improvement of 35.59\%, significantly advancing the state of the art in model editing, but also enhances the preservation of LLMs' general capabilities. Our code is available at https://github.com/xpq-tech/BLUE.

模型编辑残差分布大模型知识更新

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。