通过信号噪声比自动调节大模型各模块学习率，提升训练效率与稳定性。
Revealing Modular Gradient Noise Imbalance in LLMs: Calibrating Adam via Signal-to-Noise Ratio
- 基于模块级信噪比动态调整Adam优化器的学习率。
- 在多个大模型基准上实现更快收敛和接近人工调优的性能。
- 无需手动调参，兼容高效训练算法，适合大规模模型训练者。
大语言模型（LLMs）的强大表现源于其巨大规模和异构模块结构，但这种结构异质性带来了额外的优化挑战。尽管自适应优化器如Adam(W)具备参数级自适应能力，却未显式考虑模块级梯度异质性，导致收敛变慢、性能欠佳或训练不稳定。现有方法多依赖人工设定的模块专属学习率或特定优化策略，计算开销大且难以跨任务通用。为此，我们首先分析了Adam在高噪声模块中的噪声抑制行为，提出一种基于信噪比的模块级学习率缩放方法（MoLS）。MoLS通过估计模块级信噪比来缩放Adam更新，实现无需人工调参的自动化模块级学习率分配。多组大模型训练基准的实证结果表明，MoLS显著提升收敛速度与泛化能力，性能可媲美精心调优的模块专属学习率，同时兼容内存高效的训练算法。
原文摘要 · Abstract (English)
The impressive performance of large language models (LLMs) arises from their massive scale and heterogeneous module composition. However, this structural heterogeneity introduces additional optimization challenges. While adaptive optimizers such as Adam(W) provide per-parameter adaptivity, they do not explicitly account for module-level gradient heterogeneity, resulting in slower convergence, suboptimal performance, or training instability. Existing approaches typically rely on manually tuned module-specific learning rates or specific optimization strategies, which are computationally costly and difficult to generalize across tasks or models. To establish a more principled approach, we first analyze the noise-damping behavior of Adam in high-noise modules and introduce \textbf{Module-wise Learning Rate Scaling via SNR (MoLS)}. MoLS estimates module-level SNRs to scale Adam updates, allowing automated module-wise learning rate allocation without manual tuning. Empirical results through multiple LLM training benchmarks demonstrate that MoLS improves convergence speed and generalization, achieving performance comparable to carefully tuned module-specific learning rates, while remaining compatible with memory-efficient training algorithms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。