揭秘代码验证器强化学习训练的高效秘诀,助力低成本高精度部署。
Aletheia: What Makes RLVR For Code Verifiers Tick?
- 通过可控测试平台分析三种训练策略的权衡关系
- 大模型中思维预算最关键,小模型则依赖在线学习
- 舍弃思维痕迹可显著降本,适合低算力场景
基于可验证奖励的强化学习(RLVR)训练的多领域推理验证器是现代后训练的核心。然而,由于完整RLVR流程成本过高,其在代码生成中的应用滞后于执行反馈。本文对三个关键决策——中间思维轨迹、从负样本学习、在线学习——在性能与成本间的权衡进行消融分析。我们提出Aletheia,一个执行基准的受控测试平台,支持在不同模型规模和两种常见验证场景下进行去污染分析。结果表明:小模型中在线学习是主要性能驱动因素,而大模型中思维预算最为关键;负样本对训练稳定性至关重要,在大模型中对排序性能的影响随规模增加而提升。帕累托最优分析显示,大模型中移除在线学习仍能获得接近完整RLVR的效果。此外,舍弃思维轨迹在低预算下具有极佳的性价比。我们在Best-of-N部署及两个外部奖励模型基准上验证了结论的泛化性。本工作为训练低成本高效的代码验证器提供实证指导,推动其在后训练流水线中的广泛应用。
原文摘要 · Abstract (English)
Multi-domain thinking verifiers trained via Reinforcement Learning with Verifiable Rewards (RLVR) are a cornerstone of modern post-training. However, their adoption in code generation has lagged behind that of execution feedback due to the prohibitive costs of the full RLVR pipeline. In this work, we ablate three primary choices along the performance-cost trade-off in RLVR: intermediate thinking traces, learning from negative samples, and on-policy training. We introduce Aletheia, a controlled, execution-grounded testbed to facilitate a decontaminated analysis of code verifier training recipes across disparate model sizes and covariate shifts across two common verifier application scenarios. Our analysis reveals that the optimal training recipe is scale-dependent: on-policy learning is the primary performance driver for small verifiers, whereas the thinking budget becomes the most vital factor at larger scales. Negative samples play a key role in stabilizing training at large sizes. They have a constant impact on top-1 selection, but are increasingly important for ranking performance as size increases. Our Pareto optimality analysis demonstrates that eliminating on-policy training at larger model scales could yield a verifier that performs comparably to the full RLVR recipe. Furthermore, we find that eschewing thinking traces is a compute-efficient strategy at lower budgets, offering a strong trade-off between training cost and verifier accuracy. We validate our findings across a Best-of-N deployment setting and two external reward model benchmarks, demonstrating that our findings generalize beyond the controlled testbed. Ultimately, our work offers empirical guidance toward training cost-efficient code verifiers and takes a step toward their wider adoption in post-training pipelines for code.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。