arXiv:2605.11836cs.LGcs.CL2026-05中稿 · ICML

提出稳定编辑框架,让大模型持续更新知识不遗忘。

More Edits, More Stable: Understanding the Lifelong Normalization in Sequential Model Editing

  • 通过动态归一化梯度,构建自我强化的稳定性机制。
  • 实验证明早期编辑能提升后续编辑成功率,且长期性能更稳。
  • 适合需要持续更新知识的大模型应用,如智能助手、知识库系统。

持续模型编辑旨在不断更新大语言模型中的演化事实,同时保留无关知识和通用能力,但常面临灾难性遗忘与模型崩溃问题。我们发现近期编辑器在长周期下表现稳健的共同策略是:终身归一化(Lifelong Normalization, LN),即使用运行统计量对值梯度进行归一化。移除LN会导致性能立即崩溃,且观察到早期编辑可促进未来编辑成功的反直觉正向累积效应。然而,LN的机制仍属‘黑箱’,其在长期稳定性中的作用未被充分理解。本文首次从理论上解析了LN在终身编辑场景下的作用。分析揭示了一个自我强化的稳定性循环,并证明当结合岭回归正则化时,LN能产生渐近正交且范数有界的参数更新,直接缓解遗忘与系统性崩溃。基于此,我们提出StableEdit,通过显式预热阶段和完整白化增强该稳定性循环,在极低开销下显著提升长周期稳定性。大量实验验证了理论,并展现出竞争性性能。代码已开源。

原文摘要 · Abstract (English)

Lifelong Model Editing aims to continuously update evolving facts in Large Language Models while preserving unrelated knowledge and general capabilities, yet it remains plagued by catastrophic forgetting and model collapse. Empirically, we find that recent editors resilient over long horizons share the same core strategy: Lifelong Normalization (LN), which normalizes value gradients using running statistics. Removing LN causes immediate performance collapse, and we observe a counter-intuitive positive cumulative effect where early edits can promote the success of future edits. Yet the mechanism of LN remains a "black box", leaving its precise role in lifelong stability poorly understood. In this work, we provide the first theoretical account of LN in the lifelong regime. Our analysis reveals a self-reinforcing stability loop and proves that, when combined with ridge-regularized regression, LN yields parameter updates with asymptotic orthogonality and bounded norms, directly mitigating forgetting and systemic collapse. Based on these insights, we derive StableEdit, which strengthens this stability loop via an explicit warm-up stage and full whitening, improving long-horizon stability at minimal overhead. Extensive experiments validate our theory and demonstrate competitive performance. Our code is available at https://github.com/MINE-USTC/StableEdit.

模型编辑持续学习大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。