提出分层学习率优化框架DALS,兼顾训练与微调效果。
Learning Rate Engineering: From Coarse Single Parameter to Layered Evolution

- 构建五代学习率策略演化体系,揭示分层适配必要性。
- DALS在合成数据上达98.0%准确率,3轮内速达90%。
- 适合需要跨任务适应的模型训练,尤其从头训练场景。
学习率调度已从早期SGD的全局固定率演进为精细的分层自适应策略。本文系统归纳为五代:(Gen1) 全局固定率,(Gen2) 全局调度,(Gen3) 参数级适应,(Gen4) 层级差异化,(Gen5) 层-时联合调度。我们追溯每代演变动因,揭示从通用到按层与时定制的必要性——低层需小更新保通用知识,高层需大更新适配新任务。基于此,提出统一框架DALS,融合相位自适应余弦调度、深度感知Grokfast梯度过滤与LARS式信任比。在合成、CIFAR-10(从头训练)、RTE、TREC-6和IMDb(微调)五个数据集上评测18种策略,包括三种DALS变体。在合成数据上,DALS达到98.0%最高准确率;DALS-Fast仅3轮即达90%。跨数据集分析显示策略表现高度依赖任务范式——无单一最优策略。关键发现:STLR+Discriminative(ULMFiT冠军)在从头训练中表现崩塌(TREC-6仅43.6%,远低于预训练下的96.8%),证实方向衰减偏差有害于无预训练场景。DALS避免极端,既在合成数据领先,又保持良好微调性能。
原文摘要 · Abstract (English)
Learning rate scheduling has evolved from the single global fixed rate of early SGD to sophisticated layer-wise adaptive strategies. We systematize this evolution into five generations: (Gen1) global fixed learning rates, (Gen2) global scheduling, (Gen3) parameter-level adaptation, (Gen4) layer-level differentiation, and (Gen5) joint layer-time scheduling. We trace the fundamental motivation behind each transition, showing how the shift from one-size-fits-all to tailoring by layer and time addresses the impossible trinity of transfer learning: lower layers require small updates to preserve general knowledge while higher layers need large updates to adapt to new tasks. Building on this taxonomy, we propose Discriminative Adaptive Layer Scaling (DALS), a unified framework that integrates phase-adaptive cosine scheduling, depth-aware Grokfast gradient filtering, and LARS-style trust ratios into a single coherent optimizer. We benchmark 18 strategies including three DALS variants across all five generations on five datasets: synthetic, CIFAR-10 (from scratch), RTE, TREC-6, and IMDb (fine-tuning). On synthetic, DALS achieves the best accuracy at 98.0%, while DALS-Fast reaches 90% in just 3 epochs. The cross-dataset analysis reveals striking regime-dependent patterns -- no single strategy wins across all regimes. Critically, STLR+Discriminative, the ULMFiT champion, catastrophically fails on from-scratch tasks (43.6% on TREC-6 from scratch vs. 96.8% with RAdam), confirming that directional decay biases are harmful without pretrained features. DALS avoids either extreme, achieving the best synthetic result while maintaining competitive fine-tuning performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。