为Transformer模型不同组件设计可分离的学习率，显著加速训练。
Decoupled Relative Learning Rate Schedules
- 按组件分别调整学习率，打破全网统一速率的限制。
- 复杂模型中训练速度最快提升23%，支持跨规模复用超参。
- 适合大规模模型训练者，降低算力成本且易于部署。
本文提出一种优化大语言模型训练的新方法——解耦相对学习率调度（RLRS）。传统方法对所有网络层使用统一学习率，可能忽略各组件的独特动态。而RLRS通过为Transformer模型不同组件分别设置学习率，显著加速训练过程，最大提速达23%，尤其在混合专家（MoE）等复杂结构中效果突出。该方法可在小模型上高效调参，并有效迁移至规模大27倍的模型。该方法简单高效，大幅减少训练时间和计算资源，为大规模神经网络优化提供实用且可扩展的解决方案。
原文摘要 · Abstract (English)
In this work, we introduce a novel approach for optimizing LLM training by adjusting learning rates across weights of different components in Transformer models. Traditional methods often apply a uniform learning rate across all network layers, potentially overlooking the unique dynamics of each part. Remarkably, our introduced relative learning rates, RLRS, method accelerates the training process by up to $23\%$, particularly in complex models such as Mixture of Experts (MoE). Hyperparameters of RLRS can be efficiently tuned on smaller models and then effectively reused on models up to $27\times$ larger. This simple and effective method results in a substantial reduction in training time and computational resources, offering a practical and scalable solution for optimizing large-scale neural networks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。