研究渐进适应中每任务训练时间如何随任务数变化，发现最优训练时间与任务数成反比。
Optimal Training-Time Scaling in Gradual Adaptation
- 通过连续任务线性回归建模，分析训练时长对学习进度的影响。
- 小总时长时进度与τ成正比，大总时长时与τ的倒数成正比。
- 适合研究持续学习、在线学习及训练调度的学者参考。
在渐进适应中，当中间任务数量增加时，每个任务的训练时间应如何调整？我们研究了过参数化线性回归任务在平滑变化且共享零损失解的情况。设共有N个任务，每个任务训练时间为s_N，当Ns_N→τ时，最终学习进度收敛到一个连续曲线。当τ较小时，进度为Θ(τ)；当τ较大时，进度为Θ(τ⁻¹)，表明极短或极长的训练时间均导致进展有限。因此，最优每任务训练时间满足s_N⋆=Θ(N⁻¹)，即Ns_N⋆=Θ(1)。在逐步旋转的MNIST数据集和自然年鉴时间迁移实验中，结果均显示：路径划分越细，所需每任务训练时间越少。
原文摘要 · Abstract (English)
In gradual adaptation, how should the training time on each task change as the number of intermediate tasks increases? We study this question for overparameterized linear regression tasks that change smoothly and share a zero-loss solution. With $N$ tasks and training time $s_N$ on each, the final learning progress converges to a continuum curve when $Ns_N\toτ$. The limiting progress is $Θ(τ)$ for small $τ$ and $Θ(τ^{-1})$ for large $τ$, so both very short and very long training produce little progress. It follows that optimal per-task training times scale as $s_N^\star=Θ(N^{-1})$, equivalently $Ns_N^\star=Θ(1)$. Experiments on gradually rotated MNIST and a natural Yearbook time shift are consistent with less per-task training as the path is divided more finely.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。