让训练中‘失效’的题目重获学习信号，提升大模型推理能力
Explore Data Left Behind in Reinforcement Learning for Reasoning Language Models
- 通过动态调高无反馈题目的采样温度，激发模型生成更多不同答案
- 在多个数学推理基准上超越现有强基线模型
- 适合研究强化学习中训练数据利用率与模型泛化性的读者
基于可验证奖励的强化学习（RLVR）已成为提升大语言模型推理能力的有效方法。GRPO系列在训练过程中表现优异，但随着训练时长增加和模型规模扩大，越来越多的提示变为残余提示——即奖励方差为零、无法提供训练信号的提示。这导致有效参与训练的提示减少，降低多样性并削弱训练效果。为此，我们提出探索残余提示的策略优化框架ERPO，通过为每个提示维护历史记录，并对那些过去全部生成正确答案的残余提示自适应提高采样温度，促使模型生成更丰富的推理路径，引入错误响应以重新激活训练信号。在Qwen2.5系列上的实验表明，ERPO在多个数学推理基准上均持续优于现有强基线。
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as an effective approach for improving the reasoning abilities of large language models (LLMs). The Group Relative Policy Optimization (GRPO) family has demonstrated strong performance in training LLMs with RLVR. However, as models train longer and scale larger, more training prompts become residual prompts, those with zero variance rewards that provide no training signal. Consequently, fewer prompts contribute to training, reducing diversity and hindering effectiveness. To fully exploit these residual prompts, we propose the Explore Residual Prompts in Policy Optimization (ERPO) framework, which encourages exploration on residual prompts and reactivates their training signals. ERPO maintains a history tracker for each prompt and adaptively increases the sampling temperature for residual prompts that previously produced all correct responses. This encourages the model to generate more diverse reasoning traces, introducing incorrect responses that revive training signals. Empirical results on the Qwen2.5 series demonstrate that ERPO consistently surpasses strong baselines across multiple mathematical reasoning benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。