arXiv:2503.00035cs.CLcs.AI2025-03NAACL被引 4

通过压缩编辑锚点,缓解大模型多次修改导致的能力退化。

Constraining Sequential Model Editing with Editing Anchor Compression

  • 选取关键编辑锚点压缩信息,减少参数矩阵偏离原状态
  • 在多次编辑后仍保持超70%的通用能力
  • 适合需要持续更新知识的大模型应用

大型语言模型因错误或过时知识易产生幻觉。由于重训练资源消耗高,模型编辑成为研究热点。然而,连续编辑会显著降低模型在下游任务中的通用能力。本文统计发现,随着编辑次数增加,参数矩阵与原始状态偏差加剧,破坏原有知识关联。为此提出编辑锚点压缩(EAC)框架,通过选择对新关系编码重要且偏离原矩阵较小的编辑锚点,约束参数偏差,从而保留通用能力。在三个大模型上对两种主流编辑方法进行四类任务实验,结果表明EAC有效减少不合理偏差,在保留编辑知识的同时,使通用能力维持在70%以上,优于原方法。

原文摘要 · Abstract (English)

Large language models (LLMs) struggle with hallucinations due to false or outdated knowledge. Given the high resource demands of retraining these models, there is an increasing focus on developing model editing. However, the general abilities of LLMs across downstream tasks are prone to significant degradation during sequential editing. This paper statistically observes that the parameter matrix after editing exhibits a significant deviation compared to its previous state as the number of edits increases. This serious deviation affects the original knowledge associations within LLMs and leads to the degradation of their general abilities. To this end, a framework termed Editing Anchor Compression (EAC) is proposed to constrain the deviation of the parameter matrix during sequential editing. It compresses the editing information by selecting editing anchors that are important in encoding new relations without deviating too much from the original matrix, thereby preserving the general abilities. Experiments of applying EAC to two popular editing methods on three LLMs across four tasks are conducted. Evaluation results show that EAC effectively minimizes unreasonable deviations caused by model editing, preserving over 70% of the general abilities while better retaining the editing knowledge compared to the original counterpart methods.

模型编辑大模型知识保持参数优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。