提出可扩展的曲率测量方法,揭示大模型训练中的动态变化规律。
A Scalable Measure of Loss Landscape Curvature for Analyzing the Training Dynamics of LLMs
- 用少于10次前向传播计算曲率,实现对大模型的高效分析
- 首次在70亿参数模型上验证了损失曲面逐渐变尖和稳定边缘现象
- 适合关注训练稳定性与数据策略的大模型研发人员
理解损失曲面曲率演化是分析神经网络训练动态的基础。最常用的度量是海塞尖锐度(λ_{max}^H),即损失海塞矩阵的最大特征值,它决定局部训练稳定性,并与学习率共同影响训练过程。尽管其重要性显著,但直接计算海塞尖锐度对大语言模型(LLMs)而言因计算成本过高而不可行。本文分析了临界尖锐度(λ_c),一种仅需少于10次前向传播即可完成、依赖更新方向Δθ的高效度量。该方法能有效捕捉已知的海塞尖锐度现象,包括渐进式变尖和稳定边缘(Edge of Stability)。利用此方法,我们首次在高达70亿参数的OLMo-2模型上实现了大规模展示,覆盖预训练与中段训练阶段。我们进一步引入相对临界尖锐度(λ_c^{1→2}),用于量化在优化一个损失时另一损失的曲率,以分析从预训练到微调的转变,并指导数据混合策略。临界尖锐度为从业者提供了诊断曲率动态和制定大规模数据组合策略的实用工具。更广泛地,本工作表明,可扩展的曲率度量能够为大规模训练提供可操作的洞察。
原文摘要 · Abstract (English)
Understanding the curvature evolution of the loss landscape is fundamental to analyzing the training dynamics of neural networks. The most commonly studied measure, Hessian sharpness ($λ_{\max}^H$) -- the largest eigenvalue of the loss Hessian -- determines local training stability and interacts with the learning rate throughout training. Despite its significance in analyzing training dynamics, direct measurement of Hessian sharpness remains prohibitive for Large Language Models (LLMs) due to high computational cost. We analyze $\textit{critical sharpness}$ ($λ_c$), a computationally efficient measure requiring fewer than $10$ forward passes given the update direction $Δ\mathbfθ$. Critically, this measure captures well-documented Hessian sharpness phenomena, including progressive sharpening and Edge of Stability. Using this measure, we provide the first demonstration of these sharpness phenomena at scale, up to $7$B parameters, spanning both pre-training and mid-training of OLMo-2 models. We further introduce $\textit{relative critical sharpness}$ ($λ_c^{1\to 2}$), which quantifies the curvature of one loss landscape while optimizing another, to analyze the transition from pre-training to fine-tuning and guide data mixing strategies. Critical sharpness provides practitioners with a practical tool for diagnosing curvature dynamics and informing data composition choices at scale. More broadly, our work shows that scalable curvature measures can provide actionable insights for large-scale training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。