提出新型自适应优化器，训练随时可暂停并获得高质量模型。
Anytime Training with Schedule-Free Spectral Optimization
- 无需学习率调度，基于谱特性动态调整更新策略。
- 在125M至772M参数模型上，性能媲美甚至超越调优的AdamW。
- 适合需要灵活训练时长或持续学习的场景，降低调参成本。
标准神经网络训练依赖于固定周期的学习率调度，导致路径依赖严重，数据量变化时需重新调参。现有无调度（Schedule-Free）方法虽消除显式调度，但其最优方案SF-AdamW仍显著落后于调优后的AdamW。本文提出SF-NorMuon——一种无调度谱优化器，在仅配置单一超参数的情况下，可在1–8倍Chinchilla训练周期内，使125M与772M参数语言模型的表现达到或超过调优的AdamW。理论上，我们证明了无调度谱动态的平稳性，并指出快速迭代中的权重衰减对长周期稳定性至关重要。该方法使训练过程中任意时刻均可获得高质量检查点，无需预先设定训练周期。通过弥合与调优基线的性能差距，提升了无周期优化的实用性，推动真正开放、持续学习的发展。
原文摘要 · Abstract (English)
Standard neural network training relies on learning-rate schedules tied to a fixed horizon, leading to strong path dependence and costly re-tuning as data availability changes. Schedule-Free (SF) methods address this by removing explicit schedules, yet SF-AdamW, the current state-of-the-art anytime optimizer, consistently underperforms well-tuned AdamW baselines. We propose SF-NorMuon, a schedule-free spectral optimizer that closes this gap: with a single hyperparameter configuration, SF-NorMuon matches or exceeds tuned AdamW on 125M and 772M parameter language models across $1$--$8\times$ Chinchilla horizons. On the theoretical side, we prove a stationarity guarantee for schedule-free spectral dynamics and identify weight decay at the fast iterate as essential for long-horizon stability. SF-NorMuon enables practitioners to obtain high-quality checkpoints at any point during training without committing to a horizon in advance. By closing the performance gap with tuned baselines, SF-NorMuon makes horizon-free optimization more practical, taking a step towards truly open-ended, continual learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。