提出通用遗忘率,揭示随机任务顺序可防灾难性遗忘。
From Continual Learning to SGD and Back: Better Rates for Continual Linear Models
- 将连续学习等价为改进的SGD步,建立新上界
- 在随机重复/无重复顺序下分别获最优遗忘率
- 适用于线性分类与更广方法,理论普适性强
我们研究过参数化模型在连续学习场景中逐个拟合一组可共同实现的任务。分析经过k次迭代后的遗忘现象,即先前任务损失。对连续线性模型,证明拟合一个任务等价于在修改目标函数上的单次随机梯度下降(SGD)步骤。我们在可实现最小二乘设置下推导出新的最后一轮迭代SGD上界,并用于连续学习新结果。针对T个任务的随机顺序,建立了无需依赖问题维度或复杂性的通用遗忘率;此前率依赖维度,在高度过参数化情形下失效。在有替换的连续回归中,将最佳现有率从O((d−r̄)/k)提升至O(min(1/⁴√k, √(d−r̄)/k, √(T r̄)/k)),其中d为维度,r̄为平均任务秩。进一步建立首个无替换随机任务顺序的率:O(min(1/⁴√T, (d−r̄)/T)),表明仅靠随机化即可在足够长序列中防止灾难性遗忘。最后,对可分数据上的连续线性分类,证明匹配的O(1/⁴√k)遗忘率。我们的通用率可扩展至块Kaczmarz和POCS等方法,揭示其在i.i.d.与单遍顺序下的损失收敛性。
原文摘要 · Abstract (English)
We study the common continual learning setup where an overparameterized model is sequentially fitted to a set of jointly realizable tasks. We analyze forgetting, defined as the loss on previously seen tasks, after $k$ iterations. For continual linear models, we prove that fitting a task is equivalent to a single stochastic gradient descent (SGD) step on a modified objective. We develop novel last-iterate SGD upper bounds in the realizable least squares setup and leverage them to derive new results for continual learning. Focusing on random orderings over $T$ tasks, we establish universal forgetting rates, whereas existing rates depend on problem dimensionality or complexity and become prohibitive in highly overparameterized regimes. In continual regression with replacement, we improve the best existing rate from $O((d-\bar{r})/k)$ to $O(\min(1/\sqrt[4]{k}, \sqrt{(d-\bar{r})}/k, \sqrt{T\bar{r}}/k))$, where $d$ is the dimensionality and $\bar{r}$ the average task rank. Furthermore, we establish the first rate for random task orderings without replacement. The resulting rate $O(\min(1/\sqrt[4]{T},\, (d-\bar{r})/T))$ shows that randomization alone, without task repetition, prevents catastrophic forgetting in sufficiently long task sequences. Finally, we prove a matching $O(1/\sqrt[4]{k})$ forgetting rate for continual linear classification on separable data. Our universal rates extend to broader methods, such as block Kaczmarz and POCS, illuminating their loss convergence under i.i.d. and single-pass orderings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。