提出联合调整模型规模与训练预算的高效超参优化方法，大幅降低大模型强化学习调参成本。
Efficient Hyperparameter Optimization for LLM Reinforcement Learning
- 通过小代理模型和动态早停策略，同步控制模型大小与训练预算作为精度指标。
- 单次试验效率提升最高达14.9倍，在相同时间内表现优于或相当现有方法。
- 适合需快速调优大模型强化学习超参的研究者与工程团队使用。
大型语言模型（LLMs）的强化学习对超参数配置极为敏感，超参数优化（HPO）虽关键却计算开销巨大。现有多保真度HPO方法因模型规模庞大及训练周期资源密集，难以高效应用于LLM RL。本文提出联合保真度超参数优化（JF-HPO），同时将模型规模与训练预算作为保真度进行调节。JF-HPO依托：(i) 使用目标LLM的小型代理模型实现每次HPO试验中的高效训练与评估；(ii) 基于训练动态设计的精准早停策略；(iii) 高效检查点机制以消除冗余计算。相比现有方法，JF-HPO显著提升单次试验的计算效率（最高达14.9倍），且在相同时间预算下达到更优或相当的预测准确性。值得注意的是，相较于使用VeRL Recipe提供的超参配置，JF-HPO性能提升范围为5.8%至111.6%。
原文摘要 · Abstract (English)
Reinforcement learning (RL) for large language models (LLMs) is highly sensitive to hyperparameter configurations, making hyperparameter optimization (HPO) essential yet computationally expensive. Existing multi-fidelity HPO methods remain inefficient for LLM RL due to the massive model scale and resource-intensive training cycles. In this paper, we propose Joint Fidelity Hyperparameter Optimization (JF-HPO), which simultaneously adapts both model size and training budget as fidelity. JF-HPO is empowered by: (i) it leverages a small proxy model of the target LLM for efficient training and evaluation in each HPO trial; (ii) it integrates carefully designed early-stopping strategies based on training dynamics; (iii) it introduces an efficient checkpointing mechanism to eliminate redundant computations. Compared with existing HPO methods, JF-HPO significantly improves the computational efficiency of each trial (up to 14.9 times), while achieving better or competitive predictive accuracy under the same time budget. Notably, compared with utilizing hyperparameter configurations from the VeRL Recipe, JF-HPO delivers performance improvements ranging from 5.8% to 111.6%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。