提升大模型推理能力，让强化学习更适配批量生成结果筛选。
The Best of N Worlds: Aligning Reinforcement Learning with Best-of-N Sampling via max@k Optimisation
- 直接优化max@k指标，改进强化学习的采样策略
- 在off-policy场景下实现更高样本效率，性能优于传统方法
- 适合需要批量生成高质量解的数学与代码任务
将基于可验证奖励的强化学习（RLVR）应用于数学和编程领域，显著提升了大语言模型的推理与问题解决能力。尽管在单次生成任务中表现优异，但强化学习微调过程会削弱模型的探索能力，导致生成多样性下降，从而在大规模Best-of-N采样时性能退化。本文聚焦于优化max@k指标——pass@k的连续泛化形式。我们推导出该指标的无偏on-policy梯度估计，并进一步扩展至off-policy更新，后者是现代RLVR算法中的常见机制，能提升样本效率。实验表明，所提目标在off-policy场景中有效优化了max@k，使模型与Best-of-N推理策略更好对齐。
原文摘要 · Abstract (English)
The application of Reinforcement Learning with Verifiable Rewards (RLVR) to mathematical and coding domains has demonstrated significant improvements in the reasoning and problem-solving abilities of Large Language Models. Despite its success in single generation problem solving, the reinforcement learning fine-tuning process may harm the model's exploration ability, as reflected in decreased diversity of generations and a resulting degradation of performance during Best-of-N sampling for large N values. In this work, we focus on optimizing the max@k metric, a continuous generalization of pass@k. We derive an unbiased on-policy gradient estimate for direct optimization of this metric. Furthermore, we extend our derivations to the off-policy updates, a common element in modern RLVR algorithms, that allows better sample efficiency. Empirically, we show that our objective effectively optimizes max@k metric in off-policy scenarios, aligning the model with the Best-of-N inference strategy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。