arXiv:2604.28056cs.AI2026-04

让大模型生成的奖励函数在合适时机被验证和使用,提升强化学习训练效果。

RHyVE: Competence-Aware Verification and Phase-Aware Deployment for LLM-Generated Reward Hypotheses

论文配图:RHyVE: Competence-Aware Verification and Phase-Aware Deployment for LLM-Generated Reward Hypotheses
图 1 · 摘自论文原文
  • 根据策略能力与训练阶段,动态验证并部署大模型生成的奖励假设。
  • 低能力阶段奖励排名不可靠,超过任务阈值后才具备指导意义。
  • 适合研究大模型奖励生成与强化学习协同优化的团队参考。

大型语言模型(LLMs)使强化学习中的奖励设计更具可扩展性,但生成的奖励并非自动可靠。现有工作多关注奖励候选的生成、演化或选择,而较少关注何时可验证与部署这些候选。我们通过将生成的奖励视为依赖于当前策略能力与训练阶段的奖励假设,研究这一部署时机问题。提出RHyVE协议,利用短时程分叉验证,从共享策略检查点中比较少量奖励假设。实验表明,低能力阶段奖励排名不可靠,超过任务相关阈值后才具信息量。在稀疏操作任务中,基于阶段的部署提升了峰值性能与保持性能。更新后的LLM生成奖励候选实验显示:候选家族行为具有阶段依赖性,且不存在通用最优预热调度。保留调度选择、保守选择基线、计算量匹配控制及规模控制进一步表明,RHyVE应理解为基于验证信息的部署协议,而非通用调度器。密集与全失败边界实验明确了该方法适用范围。结果共同表明,奖励生成与部署需作为耦合问题研究:生成的奖励必须在策略能力变化下进行验证与部署。

原文摘要 · Abstract (English)

Large language models (LLMs) make reward design in reinforcement learning substantially more scalable, but generated rewards are not automatically reliable training objectives. Existing work has focused primarily on generating, evolving, or selecting reward candidates, while paying less attention to when such candidates can be verified and deployed during policy optimization. We study this deployment-time problem by treating generated rewards as reward hypotheses whose utility depends on the competence of the current policy and the phase of training. We propose \textsc{RHyVE}, a competence-aware verification and phase-aware deployment protocol that compares small sets of reward hypotheses from shared policy checkpoints using short-horizon fork verification. Our experiments show that reward rankings are unreliable at low competence but become informative after task-dependent thresholds. On a sparse manipulation task, phase-aware deployment improves peak and retained performance under a locked protocol. Updated LLM-generated reward-candidate experiments show candidate-family-dependent behavior: generated pools can exhibit phase-dependent winner changes, but no fixed warm-up schedule is universally optimal. Held-out schedule selection, conservative selector baselines, compute-matched controls, and scale controls further show that \textsc{RHyVE} is best understood as a verification-informed deployment protocol rather than a universal scheduler. Dense and all-failure boundary experiments delimit the scope of the method. Together, these results suggest that reward generation and reward deployment should be studied as coupled problems: generated rewards must be verified and deployed under changing policy competence.

强化学习大模型奖励设计动态部署

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。