通过递增正则化，实现持续线性回归的最优学习速率。
Optimal Rates in Continual Linear Regression via Increasing Regularization
- 用显式或隐式正则化将问题转化为可分析的代理损失优化。
- 固定正则强度下达到近优率 $O(\log k / k)$，递增正则达最优率 $O(1/k)$。
- 适用于关注持续学习理论性能、设计稳定训练策略的研究者。
我们研究在随机任务顺序下的可实现持续线性回归，这是持续学习理论中的常见设置。在此设定中，$k$ 次学习迭代后的最坏情况期望损失存在下界 $Ω(1/k)$。然而，以往采用无正则化方案的工作仅能证明上界 $O(1/k^{1/4})$，存在显著差距。本文证明，通过两种常用正则化方法——（1）显式的各向同性 $\ell_2$ 正则化，和（2）通过有限步数预算实现的隐式正则化——可缩小甚至消除该差距。我们发现，这些方法在实践中用于缓解遗忘，实质等价于对精心定义的代理损失进行随机梯度下降（SGD）。通过这一视角，我们识别出一个固定正则强度，可达到近优率 $O(\log k / k)$。此外，通过形式化并分析针对时变函数的广义 SGD，我们推导出一种递增正则强度调度策略，可严格实现最优率 $O(1/k)$。这表明，在最坏情况下，增加正则系数或减少每任务步数是有益的。
原文摘要 · Abstract (English)
We study realizable continual linear regression under random task orderings, a common setting for developing continual learning theory. In this setup, the worst-case expected loss after $k$ learning iterations admits a lower bound of $Ω(1/k)$. However, prior work using an unregularized scheme has only established an upper bound of $O(1/k^{1/4})$, leaving a significant gap. Our paper proves that this gap can be narrowed, or even closed, using two frequently used regularization schemes: (1) explicit isotropic $\ell_2$ regularization, and (2) implicit regularization via finite step budgets. We show that these approaches, which are used in practice to mitigate forgetting, reduce to stochastic gradient descent (SGD) on carefully defined surrogate losses. Through this lens, we identify a fixed regularization strength that yields a near-optimal rate of $O(\log k / k)$. Moreover, formalizing and analyzing a generalized variant of SGD for time-varying functions, we derive an increasing regularization strength schedule that provably achieves an optimal rate of $O(1/k)$. This suggests that schedules that increase the regularization coefficient or decrease the number of steps per task are beneficial, at least in the worst case.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。