提出可预测大模型训练效果的超参规律，省去大量试错成本。
Optimization Hyper-parameter Laws for Large Language Models
- 基于随机微分方程分析，建立学习率调度与训练损失的关系模型。
- 在未见配置上94%概率选出接近最优的调度方案，对发散训练检测准确率92%。
- 适合需要高效调参的大模型训练者，尤其适用于资源受限场景。
大语言模型推动了人工智能重大进展，但其训练资源消耗大且对超参数高度敏感。尽管缩放定律为模型规模和数据需求提供了指导，却难以确定训练过程中动态变化的超参数（如学习率调度）。为此，我们提出优化超参数规律（Opt-Laws），一个能将最终训练损失预测为学习率调度、模型规模和数据规模函数的框架。该框架基于基于随机微分方程的收敛与逃逸分析，生成可解释的收敛与逃逸特征，可在不同模型规模上预测训练损失，从而实现从小规模实验预选调度方案。实证表明，Opt-Laws在保留配置中达到94%的前二命中率，五个跨族设置中均正确识别最佳调度家族，并以F1=0.92检测训练发散。
原文摘要 · Abstract (English)
Large Language Models have driven significant AI advancements, yet their training is resource-intensive and highly sensitive to hyper-parameter selection. While scaling laws provide valuable guidance on model size and data requirements, they fall short in choosing dynamic hyper-parameters, such as learning-rate (LR) schedules, that evolve during training. To bridge this gap, we present Optimization Hyper-parameter Laws (Opt-Laws), a framework that predicts final training loss as a function of LR schedule, model size, and data size. Grounded in SDE-based convergence and escape analyses, Opt-Laws yield interpretable convergence and escape features that predict final training loss across model scales, enabling schedule pre-selection from small-scale experiments. Empirically, Opt-Laws achieve a 94% Top-2 hit rate for identifying near-optimal schedule candidates on held-out configurations, correctly identify the best-performing schedule family in all five evaluated out-of-family settings, and detect training divergence with F1 = 0.92.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。