首次证明权重衰减让Transformer损失函数具备快速收敛的数学基础
Weight-Decay Turns Transformer Loss Landscapes Villani: Functional-Analytic Foundations for Optimization and Generalization
- 用泛函分析证明损失函数满足维拉尼条件，可保证优化稳定性
- 推导出明确的收敛速率与泛化界，与正则强度λ和模型维度有关
- 提出可扩展的诊断工具，实测验证了理论预测的二次增长特性
权重衰减在大语言模型中广泛使用，但其对Transformer损失景观的影响尚缺乏理论解析。本文首次严格刻画标准Transformer目标函数——交叉熵损失加L²正则——满足维拉尼关于强制能量函数的条件。具体而言，我们证明正则化损失ℱ是无限可微的，至少以二次方式增长，具有高斯可积尾部，并满足微分增长条件：当‖θ‖→∞时，-Δℱ + (1/s)‖∇ℱ‖² → ∞（∀s>0）。由此，我们推导出显式的对数索博列夫和泊松常数，满足C_LS ≤ λ⁻¹ + d/λ²，将正则强度λ与模型维度d关联至噪声随机梯度下降的有限时间收敛保证及紧缩的PAC-Bayesian泛化界。为验证理论，引入可扩展的维拉尼诊断Ψ_s(θ) = -Δℱ + s⁻¹‖∇ℱ‖²，通过Hutchinson迹探针高效估计超过1亿参数模型。在GPT-Neo-125M上对Penn Treebank与WikiText-103的实验验证了Ψ_s的预测二次增长、海森矩阵谱膨胀及符合对数索博列夫分析的指数收敛行为。结果表明，权重衰减不仅提升泛化性能，还建立了深度学习中快速朗之万混合与曲率感知优化的理论基础。
原文摘要 · Abstract (English)
Weight decay is widely used as a regularizer in large language models, yet its precise role in shaping Transformer loss landscapes remains theoretically underexplored. This paper provides the first rigorous functional-analytic characterization of the standard Transformer objective--cross-entropy loss with $L^2$ regularization--by proving it satisfies Villani's criteria for coercive energy functions. Specifically, we show that the regularized loss $\mathcal{F}$ is infinitely differentiable, grows at least quadratically, has Gaussian-integrable tails, and satisfies the differential growth condition $-Δ\mathcal{F} + \tfrac{1}{s}\|\nabla\mathcal{F}\|^{2} \to \infty$ as $\|θ\| \to \infty$ for all $s>0$. From this structure, we derive explicit log-Sobolev and Poincaré constants $C_{\mathrm{LS}} \leq λ^{-1} + d/λ^{2}$, linking the regularization strength $λ$ and model dimension $d$ to finite-time convergence guarantees for noisy stochastic gradient descent and PAC-Bayesian generalization bounds that tighten with increasing $λ$. To validate our theory, we introduce a scalable Villani diagnostic $Ψ_s(θ) = -Δ\mathcal{F} + s^{-1}\|\nabla \mathcal{F}\|^2$ and estimate it efficiently using Hutchinson trace probes in models with over 100M parameters. Experiments on GPT-Neo-125M across Penn Treebank and WikiText-103 confirm the predicted quadratic growth of $Ψ_s$, spectral inflation of the Hessian, and exponential convergence behavior consistent with our log-Sobolev analysis. These results demonstrate that weight decay not only improves generalization empirically but also establishes the mathematical conditions required for fast Langevin mixing and theoretically grounded curvature-aware optimization in deep learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。