arXiv:2510.01180cs.LGcs.CL2025-10被引 18

通过大幅增加每例采样数,突破强化学习性能瓶颈。

BroRL: Scaling Reinforcement Learning via Broadened Exploration

  • 扩大每次训练的采样数量,实现更全面探索
  • 在3000步后仍持续提升模型性能,超越传统方法
  • 适合追求模型上限、需深度优化的开发者

基于可验证奖励的强化学习(RLVR)已成为激发大语言模型复杂推理能力的关键。近期工作ProRL通过增加训练步数实现强化学习的扩展,但在数千步后性能趋于饱和,进一步投入计算资源回报递减。本文提出一种互补策略BroRL:通过将每例采样数提升至数百,充分拓展探索空间,从而在训练步数达到饱和点后仍实现持续性能提升。该方法基于质量平衡方程分析,揭示了正确与错误词元概率质量的变化机制。在一阶强化学习假设下,采样词元始终促进正确质量增长,未采样词元则根据分布和净奖励可能增或减。随着每例采样数N增大,未采样项影响减弱,整体确保正确质量持续扩张。模拟实验验证,在宽松条件下,足够大的采样规模N(即充分探索)能保证所有正确词元概率质量上升。实证显示,BroRL使在3000步后已饱和的模型重新获得提升,在多种基准测试中达1.5B模型最优表现。

原文摘要 · Abstract (English)

Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a key ingredient for unlocking complex reasoning capabilities in large language models. Recent work ProRL has shown promise in scaling RL by increasing the number of training steps. However, performance plateaus after thousands of steps, with clear diminishing returns from allocating more computation to additional training. In this work, we investigate a complementary paradigm for scaling RL, BroR-Lincreasing the number of rollouts per example to hundreds to exhaustively Broaden exploration, which yields continuous performance gains beyond the saturation point observed in ProRL when scaling the number of training steps. Our approach is motivated by a mass balance equation analysis allowing us to characterize the rate of change in probability mass for correct and incorrect tokens during the reinforcement process. We show that under a one-step RL assumption, sampled rollout tokens always contribute to correct-mass expansion, while unsampled tokens outside rollouts may lead to gains or losses depending on their distribution and the net reward balance. Importantly, as the number of rollouts per example N increases, the effect of unsampled terms diminishes, ensuring overall correct-mass expansion. To validate our theoretical analysis, we conduct simulations under more relaxed conditions and find that a sufficiently large rollout size N-corresponding to ample exploration-guarantees an increase in the probability mass of all correct tokens. Empirically, BroRL revives models saturated after 3K ProRL training steps and demonstrates robust, continuous improvement, achieving state-of-the-art results for the 1.5B model across diverse benchmarks.

强化学习模型优化探索策略大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。