自动发现让序列计算更容易训练的生成顺序。
Discovering Learning-Friendly Generation Orders for Sequential Computation
- 通过早期损失下降速度评估候选顺序，快速筛选学习友好型生成顺序。
- 在六项任务中，成功率从约10%提升至近100%，最高可处理40个元素。
- 适用于依赖关系复杂的序列任务，如乘法和动态系统建模。
通过自回归生成进行序列计算可使复杂任务变得可学习，但中间状态的生成顺序强烈影响训练是否成功。本文提出自动发现学习友好目标顺序的方法，无需依赖特定任务设计。核心观察是：学习友好顺序在训练初期能带来更快的损失下降。为此，我们采用“损失剖面分析”，通过单次短时运行的早期损失对候选顺序进行排序。为应对阶乘级候选空间，将损失剖面分析嵌入分层全局-局部搜索，分别优化块间与块内顺序。在六项顺序敏感任务中，该方法可从随机初始化发现高达L=13的有效顺序，从结构化初始化可达L=40，成功率从约10%提升至接近100%。在整数乘法任务中，重新发现了先前研究报道高效的逆位数顺序；在延迟动力系统（多变量递推案例）中，即使合法拓扑排序之间学习性差异显著，该方法仍能识别出学习友好顺序，且全局搜索甚至发现了优于人工设计的顺序。
原文摘要 · Abstract (English)
Sequential computation via autoregressive generation can make difficult tasks learnable, but the generation order of intermediate states strongly affects whether training succeeds. We address the problem of discovering a learning-friendly target order automatically, rather than relying on task-specific design. Our key observation is that learning-friendly orders cause a faster loss drop in the early stage of training. We exploit this by \emph{loss profiling}, which ranks candidate orders by the early-stage loss of a single short run. To handle the factorial candidate space, we wrap loss profiling in a hierarchical global -- local search over block- and within-block-level orderings. On six order-sensitive tasks, the method discovers effective orders up to $L=13$ from random initialization and up to $L=40$ from structured initialization, lifting success rates from about 10\% to near 100\%. On integer multiplication, it rediscovers the reverse-digit order that was reported to be efficient in prior studies. On delay dynamical systems, as a case study of multi-variate recurrences, learnability varies sharply even among valid topological sorts of the dependency graph: loss profiling identifies a learning-friendly one, and the global search even discovers orders surpassing hand-designed candidates.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。