提出新方法提升大模型连续编辑稳定性,防止知识遗忘。
Energy-Regularized Sequential Model Editing on Hyperspheres
- 用超球面能量衡量权重分布均匀性,发现其波动与编辑失败相关。
- 在LLaMA3和Qwen2.5上,编辑能力平均提升16.41%,性能更稳定。
- 适合需要频繁更新知识的大模型应用,如智能助手、问答系统。
大型语言模型需持续更新以匹配不断变化的现实知识。模型编辑提供了一种轻量级替代重训练的方法,但连续编辑常导致表征不稳定并引发灾难性遗忘。本文提出假设:超球面均匀性——即神经元权重在超球面上均匀分布的特性——有助于模型保持稳定、保留旧知识的同时容纳新信息。通过超球面能量(HE)量化编辑过程中的权重均匀性,实证研究表明,HE动态与编辑性能高度相关,编辑失败始终伴随高HE波动。进一步理论证明,HE动态对预训练知识退化的下降具有下界约束,凸显了其稳定性对知识保留的重要性。基于此,提出SPHERE(稀疏投影的超球面能量正则化编辑)策略,通过识别预训练权重矩阵主方向的互补稀疏空间,并将新知识投影至该空间,抑制主方向扰动,稳定权重分布。在LLaMA3(8B)和Qwen2.5(7B)上的大量实验表明,SPHERE相比最佳基线平均提升编辑能力16.41%,同时最大程度保留通用模型性能,为大规模知识编辑提供了可信赖的原理性路径。
原文摘要 · Abstract (English)
Large language models (LLMs) require constant updates to remain aligned with evolving real-world knowledge. Model editing offers a lightweight alternative to retraining, but sequential editing often destabilizes representations and induces catastrophic forgetting. In this work, we seek to better understand and mitigate performance degradation caused by sequential editing. We hypothesize that hyperspherical uniformity, a property that maintains uniform distribution of neuron weights on a hypersphere, helps the model remain stable, retain prior knowledge, while still accommodate new updates. We use Hyperspherical Energy (HE) to quantify neuron uniformity during editing, and examine its correlation with editing performance. Empirical studies across widely used editing methods reveals a strong correlation between HE dynamics and editing performance, with editing failures consistently coinciding with high HE fluctuations. We further theoretically prove that HE dynamics impose a lower bound on the degradation of pretrained knowledge, highlighting why HE stability is crucial for knowledge retention. Motivated by these insights, we propose SPHERE (Sparse Projection for Hyperspherical Energy-Regularized Editing), an HE-driven regularization strategy that stabilizes neuron weight distributions, ultimately preserving prior knowledge while enabling reliable sequential updates. Specifically, SPHERE identifies a sparse space complementary to the principal hyperspherical directions of the pretrained weight matrices and projects new knowledge onto it, attenuating perturbations on the principal directions. Extensive experiments on LLaMA3 (8B) and Qwen2.5 (7B) show that SPHERE outperforms the best baseline in editing capability by an average of 16.41%, while most faithfully preserving general model performance, thereby offering a principled path toward reliable large-scale knowledge editing.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。