发现Transformer训练中优化器导致的低维漂移轨迹，揭示其动态结构。
Optimizer-Induced Low-Dimensional Drift and Transverse Dynamics in Transformer Training
- 通过分析AdamW下参数轨迹，发现主导漂移方向占长期位移60%-80%。
- 该方向在滚动窗口内稳定，但随训练阶段渐变，尤其在目标重加权后。
- 换用SGD或降低β₂会破坏此结构，适合研究优化器机制的学者。
我们分析了AdamW优化器下Transformer训练的累积参数轨迹，发现一个主导的低维漂移方向（'骨干'），可解释初始化后60--80%的长期位移。该方向在滚动训练窗口内高度稳定，但在不同训练阶段逐渐重定向，尤其在目标函数重新加权后。每批次梯度与骨干方向近乎噪声级对齐，而优化器更新则强烈对齐，表明该结构源于累积优化器动力学而非瞬时梯度几何。改用SGD族优化器会消除该结构，降低β₂值会逐步削弱骨干主导性及重启恢复能力。重加热实验显示，横向探测模式可短暂重新激发，而不显著改变累积骨干漂移。这些结果提供了对变压器训练中优化器诱导几何结构的轨迹级表征，将关注点从瞬时梯度属性转向累积更新动力学。
原文摘要 · Abstract (English)
We analyze cumulative parameter trajectories of transformer training under AdamW and identify a dominant low-dimensional drift direction ("backbone") that captures 60--80% of long-horizon displacement from initialization. This direction is highly stable over rolling training windows yet reorients gradually across phases, particularly following objective reweighting. Per-batch gradients exhibit near-noise-floor alignment with the backbone, whereas optimizer-integrated updates align strongly with it, indicating that the structure emerges from accumulated optimizer dynamics rather than instantaneous gradient geometry. Replacing AdamW with SGD-family optimizers eliminates this structure, and reducing $β_2$ smoothly degrades backbone dominance and reheating recoverability. Reheating experiments show that transverse probe modes can be transiently re-excited without substantially altering accumulated backbone drift. These results provide a trajectory-level characterization of optimizer-induced geometric structure in transformer training and shift attention from instantaneous gradient properties to cumulative update dynamics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。