用费舍尔信息动态屏蔽参数，缓解大模型持续学习中的遗忘问题。
FGGM: Fisher-Guided Gradient Masking for Continual Learning
- 基于对角费舍尔信息生成自适应二值掩码，筛选重要参数更新。
- 在TRACE上比SFT提升9.6%，比MIGU提升4.4%，遗忘更少。
- 无需历史数据，数学原理清晰，适合持续学习场景使用。
灾难性遗忘严重阻碍大语言模型的持续学习。本文提出费舍尔引导梯度掩码（FGGM），通过使用对角费舍尔信息策略性选择更新参数，缓解该问题。FGGM动态生成自适应阈值的二值掩码，保留关键参数，在不依赖历史数据的前提下平衡稳定性和可塑性。与基于权重大小的方法（如MIGU）相比，本方法具有数学上严谨的参数重要性估计。在TRACE基准测试中，FGGM相较于监督微调（SFT）在保持通用能力上实现9.6%的相对提升，并在TRACE任务上较MIGU提升4.4%。代码生成任务的附加分析进一步验证了其优异性能和更低遗忘率，证明其为有效解决方案。
原文摘要 · Abstract (English)
Catastrophic forgetting impairs the continuous learning of large language models. We propose Fisher-Guided Gradient Masking (FGGM), a framework that mitigates this by strategically selecting parameters for updates using diagonal Fisher Information. FGGM dynamically generates binary masks with adaptive thresholds, preserving critical parameters to balance stability and plasticity without requiring historical data. Unlike magnitude-based methods such as MIGU, our approach offers a mathematically principled parameter importance estimation. On the TRACE benchmark, FGGM shows a 9.6% relative improvement in retaining general capabilities over supervised fine-tuning (SFT) and a 4.4% improvement over MIGU on TRACE tasks. Additional analysis on code generation tasks confirms FGGM's superior performance and reduced forgetting, establishing it as an effective solution.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。