通过自适应探索提升大模型推理能力,兼顾难题深度与数据广度。
Depth-Breadth Synergy in RLVR: Unlocking LLM Reasoning Gains with Adaptive Exploration
- 用多阶段采样动态重加权难题,增强困难问题的训练覆盖。
- 扩大批量规模显著提升Pass@1,但单纯增加采样次数反而有害。
- 深度与广度协同优化,适合需要强推理能力的研究者使用。
强化学习结合可验证奖励(RLVR)是提升大语言模型推理能力的有效方法,但其潜力受限于对深度(问题难度)和广度(训练实例数量)的探索不足。我们分析发现,主流的GRPO算法会降低低准确率、高难度问题的权重,而这类问题恰恰对推理能力提升至关重要。为此,提出难度自适应滚动采样(DARS),通过分阶段、目标导向的多轮采样,按重新平衡策略增加难问题的滚动结果数量,从而持续提升Pass@K表现。研究发现,单纯增大滚动规模无法提升性能,甚至可能损害效果;而通过全批量更新扩大训练实例数量,能显著提升Pass@1指标,原因在于更高的词元级熵,确保充分探索并减少梯度噪声。进一步提出DARS-Breadth,将DARS与大规模训练数据结合,实现Pass@K与Pass@1的同步提升,证实深度(自适应探索)与广度(迭代实例扩展)是互补且正交的维度,共同释放RLVR全部潜力。
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Reward (RLVR) is a powerful method for enhancing the reasoning abilities of Large Language Models, but its full potential is limited by a lack of exploration in two key areas: Depth (the difficulty of problems) and Breadth (the number of training instances). Our analysis of the popular GRPO algorithm reveals a bias that down-weights difficult, low-accuracy problems, which are crucial for improving reasoning skills. To address this, we introduce Difficulty Adaptive Rollout Sampling (DARS), a method that re-weights difficult problems by using targeted, multi-stage rollouts. DARS increases the number of rollout outcomes for these harder problems according to our proposed re-balancing schedules and leads to consistent gains in Pass@K. We discovered that increasing rollout size alone does not improve performance and may actually impair it. In contrast, scaling the batch size to increase breadth via full-batch updates significantly boosted Pass@1 metrics. This improvement stems from higher token-level entropy, ensuring robust exploration and minimized gradient noise. We further present DARS-Breadth, a combined approach that uses DARS with a large breadth of training data. This method demonstrates simultaneous gains in both Pass@K and Pass@1, confirming that depth (adaptive exploration) and breadth (scaling iteration instances) are orthogonal and complementary dimensions for unlocking the full power of RLVR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。