零奖励时，加简单样本能突破强化学习困局
What Can You Do When You Have Zero Rewards During RL?
- 在无正确解情况下，通过加入更易样本辅助训练
- 即使初始奖励为零，模型仍能最终解决难题
- 无需改动算法，仅调整数据集即可生效
基于结果的强化学习在提升大语言模型复杂推理能力方面表现良好，但其成功依赖于基础模型偶尔采样到正确解。当完全无法采样到正确解时，训练会陷入零奖励障碍，因梯度为零而停滞。本文通过Bachmann等人（2024）提出的图搜索任务研究该场景，评估了包含密集奖励、多样性激励和改进信用分配等组件的近期方法。实验表明，若基础模型从不产生正确答案，这些方法均无法突破零奖励障碍。相比之下，我们发现仅通过向训练集添加更简单的样本这一数据中心干预，即可使模型在零奖励起点下最终解决原始难题。重要的是，该策略无需修改强化学习算法本身。由于多个基线方法官方实现不可用，我们自行复现并深入分析其失败原因，现已开源以支持后续研究：https://github.com/rl4reasoning/rl-baselines
原文摘要 · Abstract (English)
Reinforcement learning (RL) with outcome-based rewards has proven effective for improving large language models (LLMs) on complex reasoning tasks. However, its success often depends on the base model occasionally sampling correct solutions. When no correct solutions are sampled, training encounters a zero-reward barrier where learning stalls due to zero gradients. We study this scenario through the graph search task introduced in Bachmann et al. (2024) and evaluate recent methods that incorporate desirable components such as dense rewards, diversity incentives, and improved credit assignment. Our experiments show that none of these approaches overcome the zero-reward barrier if the base model never produces a correct answer. In contrast, we find that a simple data-centric intervention of adding easier samples to the training set enables the model to eventually solve the original hard task despite starting from zero reward. Importantly, this succeeds without modifying the RL algorithm itself. Because official implementations of several baselines were unavailable, we developed our own, which allowed us to conduct a detailed analysis of their failure modes. We release these implementations to support further research at: https://github.com/rl4reasoning/rl-baselines
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。