arXiv:2601.05049cs.AI2026-01ACL被引 4

提出两种学习率配置方法,提升大规模预训练效率与稳定性。

How to Set the Learning Rate for Large-Scale Pre-training?

  • 构建缩放规律模型,将搜索复杂度从O(n³)降至O(n×C_D×C_η)
  • 实验证明μTransfer在大规模场景下不可靠,模块级调参效果差
  • 为工业级预训练提供可落地的调参指南和理论新视角

大规模预训练中学习率(LR)的最优配置是基础但棘手的问题。在训练成本与模型性能的严格权衡下,核心问题是能否通过低成本实验准确外推最优学习率。本文将该问题形式化为两种研究范式:拟合(Fitting)与迁移(Transfer)。在拟合范式中,创新性地提出搜索因子的缩放规律,通过预测建模将搜索复杂度从O(n³)降低至O(n×C_D×C_η)。在迁移范式中,将μTransfer原理扩展至混合专家(MoE)架构,覆盖模型深度、权重衰减和标记时序等维度。通过大规模对比实验,发现现有广泛采用的μTransfer在超大规模预训练中不具备可扩展性。进一步通过训练稳定性和特征学习双重视角,揭示了模块级参数调优在大规模设置中表现不佳的根本原因。本工作为工业级预训练提供了系统性实践指导与全新理论视角。

原文摘要 · Abstract (English)

Optimal configuration of the learning rate (LR) is a fundamental yet formidable challenge in large-scale pre-training. Given the stringent trade-off between training costs and model performance, the pivotal question is whether the optimal LR can be accurately extrapolated from low-cost experiments. In this paper, we formalize this investigation into two distinct research paradigms: Fitting and Transfer. Within the Fitting Paradigm, we innovatively introduce a Scaling Law for search factor, effectively reducing the search complexity from O(n^3) to O(n*C_D*C_η) via predictive modeling. Within the Transfer Paradigm, we extend the principles of $μ$Transfer to the Mixture of Experts (MoE) architecture, broadening its applicability to encompass model depth, weight decay, and token horizons. By pushing the boundaries of existing hyperparameter research in terms of scale, we conduct a comprehensive comparison between these two paradigms. Our empirical results challenge the scalability of the widely adopted $μ$ Transfer in large-scale pre-training scenarios. Furthermore, we provide a rigorous analysis through the dual lenses of training stability and feature learning to elucidate the underlying reasons why module-wise parameter tuning underperforms in large-scale settings. This work offers systematic practical guidelines and a fresh theoretical perspective for optimizing industrial-level pre-training.

学习率预训练超参优化MoE

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。