不同优化器让相同模型展现截然不同的学习能力,揭示优化是影响模型表现的关键因素。
Same Architecture, Different Capacity: Optimizer-Induced Spectral Scaling Laws
- 用谱秩分析发现:同一模型用不同优化器,谱容量扩展效率差异显著。
- AdamW在稀有词上仅弱线性扩展(β=0.44),而Muon实现接近线性(β=1.02),提升2.3倍。
- 优化器影响远超架构设计,推动优化与架构协同设计新思路。
缩放定律使语言模型性能可由模型规模、数据量和算力预测,但通常将优化器视为固定训练细节。我们发现,这一假设忽略了表征缩放的一个根本维度:优化器将增加的前馈网络宽度转化为实际利用的谱容量的效率。通过测量前馈网络表征的特征谱(软谱秩与硬谱秩),我们发现:相同Transformer架构在不同优化器下呈现出显著不同的谱缩放规律。在保持架构与宽度调度不变的情况下,AdamW在稀有词(TAIL)表征上的硬谱秩缩放较弱(β=0.44),而Muon在同一条件下实现近似线性缩放(β=1.02),缩放指数提高2.3倍。该差异无法由验证损失解释:尽管部分AdamW配置在延长训练后能匹配低秩Dion变体的困惑度,其谱几何结构仍显著不同,表明相似损失不等于相似表征结构。硬-软谱秩不对称性进一步表明,优化器不仅影响容量利用率,还影响各特征模式的结构分布。通过与注意力秩、位置编码等架构干预对比,我们发现优化器引起的谱移位常超过架构效应。这些结果表明,优化应作为表征缩放的第一性轴,推动优化器-架构联合设计。
原文摘要 · Abstract (English)
Scaling laws have made language-model performance predictable from model size, data, and compute, but they typically treat the optimizer as a fixed training detail. We show that this assumption misses a fundamental axis of representation scaling: how effectively the optimizer converts added FFN width into utilized spectral capacity. Using eigenspectra of feed-forward network representations, measured through soft and hard spectral-ranks, we find that \emph{the same Transformer architecture realizes markedly different spectral scaling laws when trained with different optimizers}. Holding architecture and width schedule fixed, AdamW exhibits weak hard-rank scaling ($β$=0.44) on rare-token (TAIL) representations where learning is known to be hardest, whereas Muon achieves linear scaling ($β$=1.02) in the same regimes, a $2.3\times$ increase in the scaling exponent. This difference is not reducible to validation loss: AdamW configurations can match low-rank Dion variants in perplexity, under extended training, while exhibiting sharply different spectral geometry, demonstrating that matched loss does not imply matched representation structure. Hard--soft rank asymmetry further reveals that optimizers differ not only in how much capacity is realized, but also in how that capacity is structured across eigenmodes. To disentangle optimizer effects from architectural ones, we compare against architectural interventions (e.g., attention rank and positional encoding), and find that optimizer-induced spectral shifts often exceed the architectural effects. These results suggest optimization as a first-class axis of representation scaling, motivating optimizer--architecture co-design.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。