针对大模型训练中梯度方向不均衡问题，提出新优化器Spectra提升训练效率与精度。
Spectra: Rethinking Optimizers for LLMs Under Spectral Anisotropy
- 通过追踪主导谱子空间，动态抑制低秩主峰对更新的干扰。
- 在LLaMA3 8B上比AdamW快30%，显存占用减少近一半。
- 适合追求高效低显存训练的大模型研发人员使用。
大模型训练中梯度信号高度各向异性：语言重复结构使能量集中在少数主导谱方向（仅占约1.5%方向），而上下文信息则分布于长尾。我们发现这种尖峰-尾部分离现象贯穿训练全过程，尖峰虽占比极小却主导优化器统计量。该主导性通过二阶矩归一化抑制尾部学习，并收紧全局稳定学习率范围。受此启发，我们提出Spectra——一种感知尖峰的优化器，能有效抑制主导低秩尖峰子空间，而不放大对噪声敏感的尾部。Spectra通过缓存预热的幂迭代追踪尖峰子空间，采用低秩谱形变，开销极低且显著降低优化器状态内存。在500亿词上训练的LLaMA3 8B模型上，Spectra相较AdamW实现相同目标损失提速30%，每步端到端开销降低0.7%，优化器状态内存减少49.25%，平均下游准确率提升1.62%；相比Muon，其优化器处理速度提升5.1倍，最终损失更低，平均准确率提高0.66%。
原文摘要 · Abstract (English)
Gradient signals in LLM training are highly anisotropic: recurrent linguistic structure concentrates energy into a small set of dominant spectral directions, while context specific information resides in a long tail. We show that this spike tail separation persists throughout training, with the spike occupying only about 1.5% of directions yet dominating optimizer statistics. This dominance suppresses tail learning by contracting tail updates through second moment normalization and tightening the globally stable learning rate bound. Motivated by this analysis, we propose Spectra, a spike aware optimizer that suppresses the dominant low rank spike subspace without amplifying the noise sensitive spectral tail. Spectra tracks the spike subspace via cached, warm started power iteration and applies low rank spectral shaping with negligible overhead and substantially reduced optimizer state memory. On LLaMA3 8B trained on 50B tokens, Spectra reaches the same target loss 30% faster than AdamW, reduces per step end to end overhead by 0.7%, cuts optimizer state memory by 49.25%, and improves average downstream accuracy by 1.62%. Compared to Muon, Spectra is 5.1x faster in optimizer processing time, achieves a lower final loss, and improves average accuracy by 0.66%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。