优化器与归一化层搭配不当会拖慢大模型训练，尤其动态Erf配合特定优化器时性能显著下降。
Does Your Optimizer Care How You Normalize? Normalization-Optimizer Coupling in LLM Training
- 发现动态Erf归一化在特定优化器下出现严重性能下降，与标准归一化差距扩大三倍
- 关键问题在于激活值饱和和尺度感知失效，导致信息损失
- 通过调整参数或引入滑动平均可恢复80%以上性能，适合调参与训练稳定性研究者
在10亿参数模型上进行3×2因子实验（1000步训练），我们发现归一化层与优化器通常被视为独立设计的选择这一假设可能失效：动态Erf（Derf；Chen & Liu, 2025）与Muon（Jordan, 2024）结合时，其与RMSNorm的差距从AdamW下的+0.31 nats扩大至Muon下的+0.97 nats，约三倍。作为有界归一化的对照组，Dynamic Tanh（DyT；Zhu et al., 2025）无此惩罚。分析表明，erf在Muon更快的谱范数增长下出现两种失效模式：饱和（损失压缩）和尺度盲视（忽略激活幅度）。引入指数移动平均（EMA）重建运行尺度估计，可恢复约84%的性能差距。将Derf的alpha从默认值0.5降至0.3，使其保持近线性区间，可恢复约80%性能，但该设置非原论文推荐。使用默认alpha配合Muon时，虽未引发NaN或发散，仍存在0.66 nats的交互惩罚，短周期试验易被忽略。
原文摘要 · Abstract (English)
In LLM training, normalization layers and optimizers are typically treated as independent design choices. In a 3x2 factorial at 1B parameters and 1000 training steps, we show this assumption can fail: Dynamic Erf (Derf; Chen & Liu, 2025) suffers a large negative interaction with Muon (Jordan, 2024), with its gap to RMSNorm growing from +0.31 nats under AdamW to +0.97 under Muon, approximately three times larger. Dynamic Tanh (DyT; Zhu et al., 2025), included as a bounded-normalizer control, shows no such penalty. Our evidence points to two failure modes of erf under Muon's faster spectral-norm growth: saturation (lossy compression) and scale blindness (discarding activation magnitude). An EMA-blend that reintroduces running scale estimates recovers ~84% of the gap. Separately, reducing Derf's alpha from its published default (0.5 to 0.3) recovers ~80% by keeping erf in its near-linear regime, where it approximately preserves relative scale; this setting is not the published default of Chen & Liu (2025). Using Derf's published default alpha with Muon incurs a 0.66-nat interaction penalty without producing NaNs or divergence, making the failure easy to miss in short pilot runs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。