简单拒绝采样竟比复杂强化学习更有效，为大模型推理训练提供新思路。
A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce
- 用拒绝采样只训练正向奖励样本，方法极简却效果不俗。
- 在相同数据下，该方法性能接近GRPO和PPO，且更稳定高效。
- 适合关注可解释性与轻量级训练的模型优化研究者。
强化学习（RL）已成为微调大语言模型（LLM）处理复杂推理任务的主流方法。其中，GRPO因在DeepSeek-R1等模型上的成功表现而引人注目，但其有效性来源仍不明确。本文从REINFORCE类算法视角重新审视GRPO，分析其核心组件。令人意外的是，一个仅在正向奖励样本上训练的简单拒绝采样基线——RAFT，性能与GRPO和PPO相当。消融实验表明，GRPO的主要优势源于剔除完全错误响应的提示，而非奖励归一化。基于此洞察，我们提出Reinforce-Rej，一种对策略梯度的极简扩展，同时过滤完全错误与完全正确的样本。该方法提升了KL效率与训练稳定性，成为复杂强化学习算法的轻量级替代方案。我们主张将RAFT作为稳健且可解释的基线，并建议未来工作应聚焦于更合理的负样本引入机制，而非盲目使用。本研究为基于奖励的大模型后训练提供了新方向。
原文摘要 · Abstract (English)
Reinforcement learning (RL) has become a prevailing approach for fine-tuning large language models (LLMs) on complex reasoning tasks. Among recent methods, GRPO stands out for its empirical success in training models such as DeepSeek-R1, yet the sources of its effectiveness remain poorly understood. In this work, we revisit GRPO from a reinforce-like algorithm perspective and analyze its core components. Surprisingly, we find that a simple rejection sampling baseline, RAFT, which trains only on positively rewarded samples, yields competitive performance than GRPO and PPO. Our ablation studies reveal that GRPO's main advantage arises from discarding prompts with entirely incorrect responses, rather than from its reward normalization. Motivated by this insight, we propose Reinforce-Rej, a minimal extension of policy gradient that filters both entirely incorrect and entirely correct samples. Reinforce-Rej improves KL efficiency and stability, serving as a lightweight yet effective alternative to more complex RL algorithms. We advocate RAFT as a robust and interpretable baseline, and suggest that future advances should focus on more principled designs for incorporating negative samples, rather than relying on them indiscriminately. Our findings provide guidance for future work in reward-based LLM post-training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。