arXiv:2508.08221cs.LGcs.CL2025-08被引 56

揭秘大模型推理中强化学习的有效用法,给出可复现的实战指南。

Part I: Tricks or Traps? A Deep Dive into RL for LLM Reasoning

  • 在统一框架下重现实用强化学习方法,逐项验证其机制。
  • 仅用基础PPO损失与两项技术组合,性能超越复杂策略如GRPO。
  • 提供按数据、模型规模适配的选型建议,适合实操者参考。

大模型推理中的强化学习已成为热门研究方向,相关算法创新与应用迅速发展。然而,仍存在缺乏标准化使用指南、机制理解碎片化等问题。实验设置不一致、训练数据差异及模型初始化不同导致结论冲突,使从业者难以抉择。本文通过在统一开源框架中严格复现并孤立评估主流RL技术,基于多难度数据集、不同模型规模与架构的细粒度实验,分析各技术的内部机制、适用场景与核心原理。据此提出针对特定设置的清晰选择指南,并提供可靠实践路线图。最终发现,仅用基础PPO损失搭配两项简单技术的最小组合,即可激活无评判器策略的学习能力。结果表明,该组合持续提升性能,优于GRPO、DAPO等复杂策略。

原文摘要 · Abstract (English)

Reinforcement learning for LLM reasoning has rapidly emerged as a prominent research area, marked by a significant surge in related studies on both algorithmic innovations and practical applications. Despite this progress, several critical challenges remain, including the absence of standardized guidelines for employing RL techniques and a fragmented understanding of their underlying mechanisms. Additionally, inconsistent experimental settings, variations in training data, and differences in model initialization have led to conflicting conclusions, obscuring the key characteristics of these techniques and creating confusion among practitioners when selecting appropriate techniques. This paper systematically reviews widely adopted RL techniques through rigorous reproductions and isolated evaluations within a unified open-source framework. We analyze the internal mechanisms, applicable scenarios, and core principles of each technique through fine-grained experiments, including datasets of varying difficulty, model sizes, and architectures. Based on these insights, we present clear guidelines for selecting RL techniques tailored to specific setups, and provide a reliable roadmap for practitioners navigating the RL for the LLM domain. Finally, we reveal that a minimalist combination of two techniques can unlock the learning capability of critic-free policies using vanilla PPO loss. The results demonstrate that our simple combination consistently improves performance, surpassing strategies like GRPO and DAPO.

强化学习大模型推理PPO实践指南

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。