只对更新方向一致的参数做权重衰减，提升大模型训练效果
Cautious Weight Decay
- 仅在参数更新方向一致时应用权重衰减，保持原始损失函数不变
- 在百万至十亿参数规模下，显著降低语言模型和图像分类的损失与误差
- 无需调参即可接入AdamW、Lion等优化器，适合大规模模型训练场景
我们提出谨慎权重衰减（Cautious Weight Decay, CWD），一种只需一行代码、与优化器无关的修改：仅对符号与优化器更新方向一致的参数坐标施加权重衰减。不同于标准解耦衰减隐含正则化目标，CWD保留原始损失函数，并具有双层优化解释：当达到稳定流形时，可实现滑模行为，从而搜索未修改目标函数的局部帕累托最优驻点。实践中，CWD可直接集成至AdamW、Lion、Muon等优化器，无需新增超参数或额外调优。在语言模型预训练和ImageNet分类任务中，无论在百万还是十亿参数规模，均一致提升最终损失与准确率。
原文摘要 · Abstract (English)
We introduce Cautious Weight Decay (CWD), a one-line, optimizer-agnostic modification that applies weight decay only to parameter coordinates whose signs align with the optimizer update. Unlike standard decoupled decay, which implicitly optimizes a regularized or constrained objective, CWD preserves the original loss and admits a bilevel interpretation: it induces sliding-mode behavior upon reaching the stationary manifold, allowing it to search for locally Pareto-optimal stationary points of the unmodified objective. In practice, CWD is a drop-in change for optimizers such as AdamW, Lion, and Muon, requiring no new hyperparameters or additional tuning. For language model pre-training and ImageNet classification, CWD consistently improves final loss and accuracy at million- to billion-parameter scales.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。