通过分组动态调整学习率，让大模型训练更稳定高效
Taming LLMs by Scaling Learning Rates with Gradient Grouping
- 按层分组梯度统计，对每组参数做专属学习率校准
- 在多种大模型上实现更快收敛，且不依赖固定超参数
- 特别适合参数高效微调，兼容性好、稳定性强
大规模语言模型（LLM）的训练因规模庞大和架构异质而面临挑战。尽管自适应优化器如AdamW能缓解梯度差异，但在参数级学习率估计上仍存在效率与效果不足的问题，导致训练不稳定、收敛慢，并与参数高效微调（PEFT）技术兼容性差。本文提出梯度分组缩放（SGG），一种优化器封装方法，通过动态分组和组内特定缩放改进自适应学习率估计。SGG将每层梯度统计划分为聚类，并对每个聚类应用特定缩放，以施加组级约束的同时保持精细的逐参数适应。在多种（M）LLM基准上的实验表明，SGG可无缝集成至现有优化器，对不同模型规模均带来一致性能提升与更快收敛；其在不同批量大小和学习率下的稳定性，使其成为大模型优化的可靠选择。
原文摘要 · Abstract (English)
Training large language models (LLMs) poses challenges due to their massive scale and heterogeneous architectures. While adaptive optimizers like AdamW help address gradient variations, they still struggle with efficient and effective parameter-wise learning rate estimation, resulting in training instability, slow convergence, and poor compatibility with parameter-efficient fine-tuning (PEFT) techniques. This work introduces Scaling with Gradient Grouping (SGG), an optimizer wrapper that improves adaptive learning rate estimation by dynamic grouping and group-specific scaling. SGG first groups gradient statistics in each layer into clusters and then applies cluster-specific scaling to calibrate learning rates for each parameter, thus imposing collective group-wise constraints while maintaining precise per-parameter adaptation. Experiments on diverse (M)LLM benchmarks show that SGG integrates seamlessly with existing optimizers, and offers consistent gains and faster convergence over baselines, with various model sizes. Its stability across varying batch sizes and learning rates establishes SGG as a robust choice for LLM optimization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。