arXiv:2609.04577cs.LG2026-09

训练越长,优化器表现越变,选对学习率和权重衰减很关键。

Optimizer Memory Schedules for Outscaling the Overtraining Axis

论文配图:Optimizer Memory Schedules for Outscaling the Overtraining Axis
图 1 · 摘自论文原文
  • 研究不同优化器在长期训练下的表现变化,发现最佳参数随训练时长改变。
  • ADANA在长训练下超越AdamW,优势随训练时间指数增长,接近理论预测。
  • 适合超长训练场景的模型调优,尤其关注权重衰减与动量调度策略。

我们研究优化器在过训练轴(overtraining axis)上的扩展特性,发现相对性能和最优超参数随训练周期显著变化。以51M到253M参数模型为对象,考察了从1倍到256倍过训练因子的场景,系统比较了矩阵预条件方法(Muon、SOAP)与动量调度方法(ADANA)相对于AdamW的表现。结果表明:最优学习率调度可能反转,最佳权重衰减系数近似随√OT增长,更长训练通常偏好更长固定内存。即使单独为各训练周期调优AdamW的固定内存,ADANA仍保持优势。对ADANA采用对数时间权重衰减与动量冷却可带来显著增益,且随训练延长而累积。在该处理下,ADANA在功率律随机特征模型上实现接近DANA理论预测的指数级超越。Muon与SOAP则在整个测量范围内保持约恒定的每令牌效率优势,但在最高过训练因子下可能进一步提升。尽管初始落后于两种矩阵预条件方法,但随着训练时间增加,ADANA逐渐缩小差距,最终在最高过训练因子下超越Muon并逼近SOAP。这些结果确立训练周期为优化器评估与设计的核心维度。

原文摘要 · Abstract (English)

We investigate how optimizers scale across the overtraining axis and show that relative optimizer performance and optimal hyperparameters change substantially with training horizon. In particular, we study how matrix-preconditioned methods (Muon and SOAP) and a momentum-scheduled method (ADANA) scale relative to AdamW. We compare these four optimizers across models from 51M to 253M parameters and overtraining (OT) factors from 1x to 256x, sweeping the base learning rate at every setting. The preferred learning rate schedule can reverse across the overtraining axis, the best weight decay coefficient scales approximately as sqrt(OT), and longer horizons generally favor longer fixed memory. ADANA's scaling advantage over AdamW persists after tuning AdamW's fixed memory separately at each horizon. Log-time weight decay and momentum cooldown provide substantial gains for ADANA that compound as training increases. With this treatment, ADANA outscales AdamW with an exponent advantage close to that predicted by DANA theory on power-law random features. Muon and SOAP instead provide roughly constant token-efficiency advantages over AdamW across most of the measured range, although SOAP may gain further at the highest overtraining factors. ADANA begins behind both matrix-preconditioned optimizers but closes the gaps as training increases, surpassing Muon and becoming competitive with SOAP at our highest OT factors. These results establish training horizon as an essential axis for optimizer evaluation and design.

优化器长训练超训练动态调度

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。