揭示了深度网络持续学习的收敛机制与失败原因
Convergence of Continual Learning in Homogeneous Deep Networks
- 将持续学习视为任务间隔的逐次投影，统一分析框架
- 发现即使简单模型也普遍无法全局收敛
- 提出特定正则性条件可保证局部线性收敛，适合研究者参考
我们把弱正则化的连续分类问题在齐次模型中表征为对任务间隔集的逐次投影，该结果推广了以往仅限于静态（单任务）深度模型或连续线性模型的分析。我们证明，即使在数据线性但参数非线性的简单模型中，全局收敛通常也会失败。然而，通过利用非凸投影理论的结果，我们识别出齐次深度网络的正则性特征，在随机和循环的任务序列下能保证局部线性收敛。最后，我们将分析扩展至持续回归，统一了齐次模型的框架。
原文摘要 · Abstract (English)
We characterize weakly regularized continual classification in homogeneous models as sequential projections onto task margin sets. This result generalizes prior analyses restricted to either stationary (single-task) deep models or continual linear models. We show that global convergence generally fails, even for simple models linear in data but nonlinear in parameters. Nevertheless, by leveraging results from nonconvex projection theory, we identify regularity properties of homogeneous deep networks that guarantee local linear convergence under random and cyclic task sequences. Finally, we extend our analysis to continual regression, unifying the framework for homogeneous models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。