arXiv:2605.19080cs.LGcs.AI2026-05

MANGO通过自适应梯度门控与元学习正则化,缓解在线持续学习中的遗忘问题。

MANGO: Meta-Adaptive Network Gradient Optimization for Online Continual Learning

论文配图:MANGO: Meta-Adaptive Network Gradient Optimization for Online Continual Learning
图 1 · 摘自论文原文
  • 用梯度门控按参数敏感度调节更新,避免破坏性修改
  • 元学习动态调整正则化强度,提升记忆保留效果
  • 在多个数据集上表现最优,支持小缓冲区仍有效

在在线持续学习(OCL)中,神经网络需在单次遍历非平稳数据流时,仅依赖有限记忆缓冲区进行学习,这与依赖大规模数据集多轮训练的离线持续学习形成对比。主要挑战在于:既要防止旧任务遗忘(稳定性),又要高效学习新任务(灵活性)。现有方法通过重放、输出层蒸馏、固定正则化或仅基于流数据的元学习来应对遗忘,但各有缺陷:重放引入样本偏差;蒸馏仅作用于输出分布,不调节参数更新;固定正则化对所有参数一视同仁;流数据元学习缺乏反馈控制的参数更新机制。本文提出元自适应网络梯度优化(MANGO),通过梯度门控与元学习正则化实现稳定-灵活平衡。梯度门控根据参数敏感度缩放更新量,防止有害更新;元学习正则化动态调整稳定性系数,评估参数更新对重放的影响。在MANGO中,重放既作为训练信号,也作为遗忘评估器。我们在三个标准OCL基准数据集上验证了该方法,结果表明,MANGO优于强基线,达到当前最优性能,并在不同重放规模下保持一致表现。在CLEAR-10上的领域增量学习和在CIFAR-100、Tiny-ImageNet上的类别增量学习中,均取得最高准确率,并实现正向后向迁移,成功克服CLEAR-10上的遗忘问题。

原文摘要 · Abstract (English)

In Online Continual Learning (OCL), a neural network sequentially learns from a non-stationary data stream in a single-pass with access only to a limited memory replay buffer. This contrasts sharply with off-line continual learning where training is multiple epoch dependent on large datasets. The main challenge faced by OCL is to overcome catastrophic forgetting of past tasks (stability) while learning new ones efficiently (plasticity). Existing methods counter forgetting via replay-based rehearsal, output level distillation, fixed regularization, or meta-learning on the current data. However, these methods have limitations: rehearsal introduces a stored sample bias; distillation operates on output-distributions without modulating parameter updates; fixed-regularization penalizes parameters irrespective of sensitivity; stream-only meta-learning lacks a feedback controlled parameter update. We propose Meta-Adaptive Network Gradient Optimization (MANGO), an OCL framework that balances stability-plasticity via gradient-gating and meta-learned regularization. Gradient-gating scales parameter updates based on sensitivity, preventing destructive updates. Meta-learned regularization adapts stability coefficients, evaluating the effect of parameter update on replay. In MANGO, replay acts as both a training signal and a forgetting evaluator. We evaluated our method on three standard OCL benchmark datasets. MANGO outperforms strong baselines, achieving state-of-the-art results with consistent performance across replay sizes. In domain incremental learning on CLEAR-10 and class incremental learning on CIFAR-100 and Tiny-ImageNet, it achieves highest accuracy among all baselines and achieves positive Backward Transfer, overcoming forgetting on CLEAR-10.

持续学习梯度门控元学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。