贪心任务顺序比随机顺序收敛更快，但重复学习能避免灾难性失败。
Are Greedy Task Orderings Better Than Random in Continual Linear Regression?
- 基于任务间差异最大化设计贪心顺序，提升学习效率。
- 贪心顺序在平均损失上收敛速度优于随机顺序，尤其允许重复时达1/∛k速率。
- 适合研究持续学习中任务调度机制的学者，尤其关注线性回归场景。
本文研究持续学习中线性回归的任务顺序问题，假设训练数据可联合实现。聚焦于贪心最大化连续任务间差异的顺序，虽此前已有初步探索，但仍有未解之谜。借助Kaczmarz方法工具，我们形式化此类顺序并建立几何与代数直觉。实验表明，无论随机数据还是CIFAR-100上的线性探针任务，贪心顺序的平均损失收敛均快于随机顺序。理论分析显示，在高秩回归下，贪心顺序的损失界与随机顺序类似；但在一般秩下，存在依赖重复次数的差异：单遍贪心顺序可能灾难性失败，而允许多次重复的版本收敛速率达𝒪(1/∛k)，优于随机顺序的𝒪(1/√k)。整体揭示了贪心与随机顺序间的细微差别。
原文摘要 · Abstract (English)
We analyze task orderings in continual learning for linear regression, assuming joint realizability of training data. We focus on orderings that greedily maximize dissimilarity between consecutive tasks, a concept briefly explored in prior work but still surrounded by open questions. Using tools from the Kaczmarz method literature, we formalize such orderings and develop geometric and algebraic intuitions around them. Empirically, we demonstrate that greedy orderings converge faster than random ones in terms of the average loss across tasks, both for linear regression with random data and for linear probing on CIFAR-100 classification tasks. Analytically, in a high-rank regression setting, we prove a loss bound for greedy orderings analogous to that of random ones. However, under general rank, we establish a repetition-dependent separation. Specifically, while prior work showed that for random orderings, with or without replacement, the average loss after $k$ iterations is bounded by $\mathcal{O}(1/\sqrt{k})$, we prove that single-pass greedy orderings may fail catastrophically, whereas those allowing repetition converge at rate $\mathcal{O}(1/\sqrt[3]{k})$. Overall, we reveal nuances within and between greedy and random orderings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。