跨模型验证了学习率调度的通用动力学规律。
Universal Dynamics of Warmup Stable Decay: understanding WSD beyond Transformers
- 对比CNN与语言模型，发现不同架构下优化路径相似。
- 仅部分训练期使用衰减学习率，效果媲美完整周期余弦退火。
- 适合研究高维优化几何与训练动态的学者参考。
Warmup Stable Decay（WSD）学习率调度器因其在训练大语言模型时的优异表现和灵活性而受到关注。目前尚不清楚其卓越性能是否仅限于基于Transformer的语言模型，能否为理解其训练动态提供新的理论视角。受学习率调度器作为探索损失曲面几何结构（如河谷、连通极小值、渐进锐化）新工具的启发，本文比较了Adam优化器在类Pythia语言模型与小型CNN分类CIFAR10图像时的WSD路径。结果发现，两类模型在多数训练信号、优化器路径特征及尖锐度动态上均呈现定性相似性。这一一致性表明，传统与新兴非凸优化问题的损失曲面具有共享的几何特性，也提示未来研究应聚焦高维优化问题的几何本质。
原文摘要 · Abstract (English)
The Warmup Stable Decay (WSD) learning rate scheduler has recently become popular, largely due to its good performance and flexibility when training large language models. It remains an open question whether the remarkable performance of WSD - using a decaying learning rate for only a fraction of training compared to cosine decay - is a phenomenon specific to transformer-based language models that can potentially offer new theoretical insights into their training dynamics. Inspired by the usage of learning rate schedulers as a new lens into understanding landscape geometry (e.g., river valley, connected minima, progressive sharpening), in this work we compare the WSD path of the Adam optimizer on a Pythia-like language model to that of a small CNN trained to classify CIFAR10 images. We observe most training signals, optimizer path features, and sharpness dynamics to be qualitatively similar in such architectures. This consistency points to shared geometric characteristics of the loss landscapes of old and new nonconvex problems, and hints to future research questions around the geometry of high dimensional optimization problems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。