揭示Transformer模型在不同计算资源下的学习与泛化规律，提出双阶段衰减模型。
Unifying Learning Dynamics and Generalization in Transformers Scaling Law
- 将Transformer训练过程建模为微分方程，推导出优化与统计两阶段的泛化误差行为。
- 计算资源增加时，误差先指数下降，越过阈值后转为约C⁻¹/⁷的幂律衰减。
- 理论严格证明上下界，适用于真实数据分布，对模型、训练、数据量分别给出独立规律。
缩放定律是大语言模型发展的基石，预测计算资源增加可提升模型性能。然而其理论基础仍不清晰。本文将基于Transformer的语言模型学习动态形式化为常微分方程（ODE）系统，并近似为核行为。不同于以往简化模型分析，我们严格分析了在任意数据分布下多层Transformer在序列到序列任务上使用随机梯度下降（SGD）的训练过程，贴近真实场景。分析表明，随着计算资源与数据规模同步增长，泛化误差收敛至不可约风险。我们建立了超额风险的匹配上下界，呈现显著相变特征：初期优化阶段，超额风险随计算成本${ extsf{C}}$指数衰减；一旦超过特定资源阈值，系统进入统计阶段，泛化误差遵循$Θ( extsf{C}^{-1/7})$的幂律衰减。这些速率由互补的下界保证——统计侧通过信息论两点缩减，优化侧通过一阶预言机论证，使双阶段定律在常数、对数因子及条件数间隙内严格成立。此外，本理论还导出了模型大小、训练时间与数据集大小的独立缩放规律，阐明各变量如何分别决定泛化边界。
原文摘要 · Abstract (English)
The scaling law, a cornerstone of Large Language Model (LLM) development, predicts improvements in model performance with increasing computational resources. Yet, while empirically validated, its theoretical underpinnings remain poorly understood. This work formalizes the learning dynamics of transformer-based language models as an ordinary differential equation (ODE) system, then approximates this process to kernel behaviors. Departing from prior toy-model analyses, we rigorously analyze stochastic gradient descent (SGD) training for multi-layer transformers on sequence-to-sequence data with arbitrary data distribution, closely mirroring real-world conditions. Our analysis characterizes the convergence of generalization error to the irreducible risk as computational resources scale with data, especially during the optimization process. We establish matching upper and lower bounds on the excess risk, characterized by a distinct phase transition. In the initial optimization phase, the excess risk decays exponentially relative to the computational cost ${\sf C}$. However, once a specific resource allocation threshold is crossed, the system enters a statistical phase, where the generalization error follows a power-law decay of $Θ(\mathsf{C}^{-1/7})$. These rates are certified by complementary lower bounds -- statistical, via an information-theoretic two-point reduction, and optimization-side, via a first-order oracle argument -- rendering the two-stage law tight up to constants, logarithmic factors, and a condition-number gap. Beyond this unified framework, our theory derives isolated scaling laws for model size, training time, and dataset size, elucidating how each variable independently governs the bounds of generalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。