用拟牛顿法改进遗忘抑制，让模型学新任务时不丢旧知识。
Continual Learning With Quasi-Newton Methods
- 基于拟牛顿法计算更准确的海森矩阵近似，捕捉参数间关联
- 在四个基准上平均减少50%遗忘，性能提升8%
- 无需修改网络结构，适用于多种任务和模型
神经网络在顺序学习任务时面临灾难性遗忘问题。弹性权重巩固（EWC）通过引入贝叶斯启发的正则化损失来保护已学知识，但其依赖拉普拉斯近似，将海森矩阵简化为费雪信息矩阵的对角线，假设参数之间相互独立，这一过于简化的假设常导致海森矩阵估计不准，限制了其效果。为此，我们提出持续学习中的采样拟牛顿方法（CSQN），利用拟牛顿法计算更精确的海森矩阵近似，无需架构特异性修改即可捕捉参数间的交互关系。在四个基准上的实验表明，CSQN始终优于EWC及其他先进基线方法，包括基于回放的方法。相较于EWC，CSQN平均减少50%遗忘，性能提升8%。尤其在三个出类拔萃的基准上表现优异，凸显其作为鲁棒持续学习解决方案的巨大潜力。
原文摘要 · Abstract (English)
Catastrophic forgetting remains a major challenge when neural networks learn tasks sequentially. Elastic Weight Consolidation (EWC) attempts to address this problem by introducing a Bayesian-inspired regularization loss to preserve knowledge of previously learned tasks. However, EWC relies on a Laplace approximation where the Hessian is simplified to the diagonal of the Fisher information matrix, assuming uncorrelated model parameters. This overly simplistic assumption often leads to poor Hessian estimates, limiting its effectiveness. To overcome this limitation, we introduce Continual Learning with Sampled Quasi-Newton (CSQN), which leverages Quasi-Newton methods to compute more accurate Hessian approximations. CSQN captures parameter interactions beyond the diagonal without requiring architecture-specific modifications, making it applicable across diverse tasks and architectures. Experimental results across four benchmarks demonstrate that CSQN consistently outperforms EWC and other state-of-the-art baselines, including rehearsal-based methods. CSQN reduces EWC's forgetting by 50 percent and improves its performance by 8 percent on average. Notably, CSQN achieves superior results on three out of four benchmarks, including the most challenging scenarios, highlighting its potential as a robust solution for continual learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。