梯度下降在连续线性分类中收敛到全局最优解，且遗忘随循环次数消失。
Convergence and Implicit Bias of Gradient Descent on Continual Linear Classification
- 按循环顺序用梯度下降训练多个任务，方向收敛至联合最大间隔解。
- 任务间对齐程度直接影响遗忘和反向知识迁移，循环越多遗忘越少。
- 即使任务不联合可分，循环训练仍收敛到联合损失的唯一最小值。
我们研究通过为每个任务固定迭代次数的梯度下降（GD）进行连续学习。当所有任务联合线性可分且以循环/随机顺序呈现时，训练出的线性分类器方向收敛至联合（离线）最大间隔解。这令人惊讶，因为单任务梯度下降隐式偏向于该任务的最大间隔解，而联合最大间隔解的方向可能与各任务解大相径庭。此外，当任务按循环顺序给出时，我们给出了非渐近分析，揭示：(1) 任务对齐程度与灾难性遗忘及反向知识迁移密切相关；(2) 随着循环重复，平均遗忘量趋于零。最后，我们分析了任务不再联合可分的情况，发现循环训练下模型收敛至联合损失函数的唯一最小值。
原文摘要 · Abstract (English)
We study continual learning on multiple linear classification tasks by sequentially running gradient descent (GD) for a fixed budget of iterations per task. When all tasks are jointly linearly separable and are presented in a cyclic/random order, we show the directional convergence of the trained linear classifier to the joint (offline) max-margin solution. This is surprising because GD training on a single task is implicitly biased towards the individual max-margin solution for the task, and the direction of the joint max-margin solution can be largely different from these individual solutions. Additionally, when tasks are given in a cyclic order, we present a non-asymptotic analysis on cycle-averaged forgetting, revealing that (1) alignment between tasks is indeed closely tied to catastrophic forgetting and backward knowledge transfer and (2) the amount of forgetting vanishes to zero as the cycle repeats. Lastly, we analyze the case where the tasks are no longer jointly separable and show that the model trained in a cyclic order converges to the unique minimum of the joint loss function.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。