用上下文强化学习优化语言模型推理，提升采样效率与效果
Contextual Rollout Bandits for Reinforcement Learning with Verifiable Rewards
- 将采样过程建模为上下文相关多臂赌博机，动态选择高价值回复
- 在6个数学推理基准上实现性能提升，训练效率显著提高
- 支持历史采样复用与噪声感知筛选，适用于各类强化学习优化方法
基于可验证奖励的强化学习（RLVR）能有效提升大语言模型的推理能力。然而，现有方法对采样结果处理粗放且仅短期使用：同一提示下不同质量回复被一视同仁，历史采样一旦使用即丢弃。这导致监督信号噪声大、样本效率低、策略更新不佳。本文将采样调度建模为上下文多臂赌博机问题，提出统一神经调度框架，自适应选择高价值采样结果。每个采样视为一个臂，其奖励定义为连续优化步骤间的性能增益。该调度器在单一统一框架内支持噪声感知组内选择和历史采样的自适应全局复用。理论证明中给出次线性遗憾界，并表明扩大采样缓冲区可提升性能上限。六项数学推理基准实验显示，该方法在多种RLVR优化策略下均实现性能与训练效率的持续提升。
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Rewards (RLVR) is an effective paradigm for improving the reasoning capabilities of large language models. However, existing RLVR methods utilize rollouts in an indiscriminate and short-horizon manner: responses of heterogeneous quality within each prompt are treated uniformly, and historical rollouts are discarded after a single use. This leads to noisy supervision, poor sample efficiency, and suboptimal policy updates. We address these issues by formulating rollout scheduling in RLVR as a contextual bandit problem and proposing a unified neural scheduling framework that adaptively selects high-value rollouts throughout training. Each rollout is treated as an arm whose reward is defined by the induced performance gain between consecutive optimization steps. The resulting scheduler supports both noise-aware intra-group selection and adaptive global reuse of historical rollouts within a single principled framework. We provide theoretical justification by deriving sublinear regret bounds and showing that enlarging the rollout buffer improves the achievable performance upper bound. Experiments on six mathematical reasoning benchmarks demonstrate consistent gains in performance and training efficiency across multiple RLVR optimization methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。