给罕见正确解加奖励，让强化学习模型突破原有能力边界
Rewarding the Unlikely: Lifting GRPO Beyond Distribution Sharpening
- 用罕见解奖励机制纠正GRPO的排名偏差
- 在合成与真实定理证明中提升pass@N性能
- 适合关注推理模型优化的研究者
强化学习正成为提升语言模型推理能力的主要驱动力。一个核心问题是：当前算法（如主流的组相对策略优化GRPO）是否只是让模型在已擅长的问题上分布更集中？我们在可完美验证的定理证明场景下研究发现，GRPO存在退化排名偏差——高概率轨迹被强化，而稀有但正确的解被忽略，导致分布尖锐化：虽减少采样次数即可通过某些问题，但整体性能不如直接从原模型多采样。为此，我们提出“罕见性奖励”机制，显式提升稀有但正确解的权重。实验表明，该方法缓解了排名偏差，在多种合成与真实定理证明任务中显著提升pass@$N$性能。同时发现排名偏差与一个看似平凡的超参数——每批次更新次数——存在意外关联，据此提出第二项互补缓解策略。结合两者，我们构建了改进版GRPO训练方案，形成开源管道，在miniF2F-test基准上表现媲美DeepSeek-Prover-V1.5-RL。
原文摘要 · Abstract (English)
Reinforcement learning is emerging as a primary driver for improving language model reasoning capabilities. A fundamental question is whether current reinforcement learning algorithms -- such as Group Relative Policy Optimization (GRPO), the de facto standard algorithm used to improve language model reasoning -- merely sharpen the base model's distribution around problems it can already solve. We investigate this question in the context of formal theorem proving, which has access to a perfect verifier. We identify a degenerate rank bias in GRPO in which highly probable trajectories are reinforced and rare ones are neglected. This results in distribution sharpening: the model can solve some problems with fewer samples, but underperforms simply sampling more solutions from the original model. To overcome GRPO's rank bias we introduce unlikeliness reward, a simple method for explicitly up-weighting rare but correct solutions. We show that unlikeliness reward mitigates rank bias and improves pass@$N$ across a large range of $N$ in both synthetic and real theorem proving settings. We also uncover an unexpected link between rank bias and a seemingly mundane hyperparameter -- the number of updates per batch -- that leads to a second, complementary mitigation. We combine our insights into a revised GRPO training recipe for formal theorem proving, yielding an open pipeline that achieves competitive performance to DeepSeek-Prover-V1.5-RL on the miniF2F-test benchmark. We release our implementation at https://github.com/AndreHe02/rewarding-unlikely-release
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。