提出更稳定的max@K策略梯度方法，提升大模型推理训练效果。
On Advantage Estimates for Max@K Policy Gradients
- 设计留两出基线，使优势估计无偏且均值为零
- 新方法MaxPO在批量样本上显著降低梯度方差
- 适用于大模型推理阶段的后训练优化
基于可验证奖励的强化学习广泛用于大模型推理阶段的后训练，但稀疏结果奖励导致探索困难。另一种思路是直接优化推理时目标如pass@K和max@K，但现有策略梯度估计器使用不同信号、基线和归一化方式，关系不清晰。本文通过基线设计与优势中心化研究该问题：从领域领先方法的优势估计器出发，发现其虽无偏但优势不中心化。为此提出留两出（Leave-Two-Out）基线，在保持无偏性的同时使批内真实优势恰好中心化。由此得到的MaxPO方法具有高效的二次时间复杂度实现，并能自然融入基于组的强化学习框架用于大模型后训练。进一步推导出max@K的规范有限批次优势估计，统一了现有方法。实验表明，L2O基线有效降低梯度方差，优于非中心化方案。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards is widely used for post-training reasoning models, but sparse outcome rewards make exploration difficult. A complementary approach is to optimize inference-time objectives such as pass@K and max@K directly, yet existing policy-gradient estimators for these objectives use different signals, baselines, and normalizations, making their relationships unclear. We study this issue through baseline design and advantage centering. Starting from the advantage estimator of a leading method in the field, we show that it is policy-gradient unbiased but yields a non-centered advantage. We then introduce a Leave-Two-Out baseline that preserves policy-gradient unbiasedness while making realized batch advantages exactly centered. The resulting method, MaxPO, has an efficient quadratic-time implementation and integrates naturally into group-based RL for LLM post-training. We further derive the canonical finite-batch advantage for max@K, providing a unified view of existing advantage estimators. Empirically, we verify that the L2O baseline reduces gradient variance and outperforms non-centered alternatives.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。