arXiv:2510.14807cs.AI2025-10被引 12

提出新方法提升强化学习中多候选探索能力,避免只选一个答案。

Beyond the Sampled Token: Preserving Candidate Support in RLVR

  • 通过保持前N个候选概率分布,缓解采样后答案单一问题。
  • 在6个数学、2个逻辑推理、2个编码任务上均提升准确率。
  • 适用于大模型与高采样数场景,不牺牲单次回答质量。

我们从下一个词预测的候选分布视角重新审视可验证奖励强化学习(RLVR)中的探索崩溃问题。理论上证明:当概率集中于最优候选时,无论采样预算K如何,期望不同回复数量都会坍缩至1。实证追踪训练过程发现,最优候选逐渐主导,合理替代方案被压制。因此,有效探索的关键在于保持前N个候选的显著概率质量。为此,我们提出候选感知支持保留(CaSP),包含两项设计:对正确响应,将正梯度在前N个候选间重新分配;对错误响应,对最优候选施加更强惩罚。相比多数仅提升pass@K而牺牲pass@1的方法,CaSP在全范围K值下均提升pass@K。该方法在6个数学、2个逻辑推理、2个编程基准上表现优异,可扩展至320亿参数模型和K=1024的采样预算,是一种基于候选级别的系统性探索改进方案。

原文摘要 · Abstract (English)

We revisit exploration collapse in reinforcement learning with verifiable rewards (RLVR), from the perspective of the \emph{candidate distribution} for next-token prediction. We formally show that as probability concentrates on the top-$1$ candidate, the expected number of distinct responses collapses to one regardless of the sampling budget $K$. This theoretical implication is further verified by our empirical tracking of top-$N$ candidate probabilities during training, where the top-$1$ candidate progressively dominates while plausible alternatives are suppressed. These findings suggest a key desideratum for effective exploration: \emph{preserving non-negligible probability mass on the top-$N$ candidates}. To this end, we propose Candidate-aware Support Preservation (CaSP), with two complementary designs. Specifically, CaSP redistributes positive gradients among top-$N$ candidates for correct responses, and applies a stronger penalty to the top-$1$ candidate for incorrect responses. Unlike many exploration-oriented methods that improve pass@$K$ at the cost of pass@1, CaSP improves pass@$K$ across the full $K$ spectrum. These gains generalize to 6 math, 2 logical-reasoning, and 2 coding benchmarks, and scales to 32B-parameter models and sampling budgets up to $K=1024$, positioning it as a principled, candidate-level approach for RLVR exploration.

强化学习探索策略生成模型候选保留

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。