arXiv:2604.11064cs.LGcs.CV2026-04被引 2

提升持续学习效率,训练速度比现有方法快1.25倍

A Faster Path to Continual Learning

论文配图:A Faster Path to Continual Learning
图 1 · 摘自论文原文
  • 通过去除冗余梯度计算,优化持续学习的训练过程
  • 在多种任务上实现与原方法相当甚至更高的准确率
  • 适合追求高效训练的持续学习研究者和应用开发者

持续学习旨在让神经网络在动态任务流中不断学习而不遗忘旧知识。现有基于优化的方法中,C-Flat因其即插即用特性及对新旧任务均保持低损失区域的能力而表现突出,但每轮迭代需额外三次梯度计算,带来显著开销。本文提出C-Flat Turbo,一种更快更强的优化器,显著降低训练成本。我们发现一阶平坦性相关的梯度相对于代理模型梯度具有方向无关性,因此可跳过扰动上升步骤中的冗余计算;此外,观察到这些促进平坦性的梯度随任务逐渐稳定,据此设计线性调度策略并引入自适应触发机制,为后期任务分配更大加速步数。实验表明,C-Flat Turbo在多种持续学习方法下比C-Flat快1.0×至1.25×,同时保持或提升准确率。

原文摘要 · Abstract (English)

Continual Learning (CL) aims to train neural networks on a dynamic stream of tasks without forgetting previously learned knowledge. Among optimization-based approaches, C-Flat has emerged as a promising solution due to its plug-and-play nature and its ability to encourage uniformly low-loss regions for both new and old tasks. However, C-Flat requires three additional gradient computations per iteration, imposing substantial overhead on the optimization process. In this work, we propose C-Flat Turbo, a faster yet stronger optimizer that significantly reduces the training cost. We show that the gradients associated with first-order flatness contain direction-invariant components relative to the proxy-model gradients, enabling us to skip redundant gradient computations in the perturbed ascent steps. Moreover, we observe that these flatness-promoting gradients progressively stabilize across tasks, which motivates a linear scheduling strategy with an adaptive trigger to allocate larger turbo steps for later tasks. Experiments show that C-Flat Turbo is 1.0$\times$ to 1.25$\times$ faster than C-Flat across a wide range of CL methods, while achieving comparable or even improved accuracy.

持续学习优化器高效训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。