arXiv:2601.18699cs.LGcs.CL2026-01被引 9

揭示大模型持续微调中遗忘机制,提出新方法有效缓解能力退化。

Mechanistic Analysis of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning

  • 通过参数轨迹与注意力分析,定位易遗忘的神经通路。
  • 在开放权重模型上,遗忘程度最高达94.2%时仍可有效缓解。
  • 适合关注模型长期学习与稳定性研究的开发者和研究人员。

大型语言模型(LLMs)在顺序微调过程中常出现灾难性遗忘,即学习新任务导致原有能力下降。本文对20个代表2026年中期技术水平的主流模型进行了系统性对比研究,涵盖十款闭源模型(如Claude Fable 5、GPT-5.5 High、Gemini 3.5 Flash)与十款开源模型(如DeepSeek-V4-Pro、Llama 4 Maverick、Qwen 3.6-27B)。通过权重空间轨迹追踪、中心核对齐(CKA)及混合专家(MoE)层路由门漂移计算,定位出易受参数覆盖影响的神经回路。结果表明:浅层注意力头呈现系统性熵扩散,中深层前馈网络或稀疏专家块则发生局部表示坍缩。基于此,提出低秩电路投影(LRCP)干预方法,通过子空间正则化实现稳定训练。实证表明,该方法在开放权重配置下可缓解高达94.2%的祖先能力损失,并保持与标准参数高效微调(PEFT)相当的适应速度。

原文摘要 · Abstract (English)

Sequential fine-tuning of Large Language Models (LLMs) adaptation to target tasks often triggers catastrophic forgetting, where the acquisition of novel target skills degrades ancestral capabilities. This paper presents a systematic comparative study of catastrophic forgetting across twenty premier models representing the state-of-the-art in mid-2026. We categorize our investigation into two primary research lines: (i) a behavioral and semantic output drift analysis of ten leading closed-source models (including Claude Fable 5, GPT-5.5 High, and Gemini 3.5 Flash), and (ii) a deep mechanistic interpretation of ten prominent open-weight architectures (such as DeepSeek-V4-Pro, Llama 4 Maverick, and Qwen 3.6-27B). Through weight-space trajectory tracking, Centered Kernel Alignment (CKA), and routing gate drift calculations in Mixture-of-Experts (MoE) layers, we localize the neural circuits highly susceptible to parameter overwriting. Our findings indicate that early-layer attention heads exhibit systemic entropic dispersion, while mid-to-deep feed-forward networks (or sparse expert blocks) suffer localized representation collapse. Informed by these insights, we introduce Low-Rank Circuit Projection (LRCP), a subspace-regularized training intervention. Empirical evaluations show that LRCP successfully mitigates up to 94.2% of ancestral capabilities in open-weight configurations and matches the adaptation velocity of standard PEFT baselines.

大模型持续学习遗忘缓解参数效率

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。