用真实工具使用数据提升大模型推理能力，让小模型也能超越大模型。
Demystifying Reinforcement Learning in Agentic Reasoning
- 用真实端到端工具调用轨迹替代合成数据，初始化效果更强。
- 保持策略熵和探索友好机制，训练效率显著提升。
- 少而精的思考策略优于频繁调用工具或冗长自省。
近期，代理型强化学习（agentic RL）展现出强化学习可有效提升大语言模型的代理推理能力，但其关键设计原则与最佳实践仍不清晰。本文从数据、算法和推理模式三个关键角度开展系统性研究。核心发现：(i) 以真实端到端工具使用轨迹替换拼接的合成轨迹，能显著增强SFT初始化；高多样性、模型感知的数据集可维持探索并大幅提高强化学习性能。(ii) 探索友好的技术至关重要，如裁剪奖励上限、过长奖励塑造及保持足够策略熵，均能提升训练效率。(iii) 采用较少工具调用的深思策略优于频繁调用或冗长自我推理，提升工具使用效率与最终准确率。这些简单实践持续增强代理推理能力与训练效率，在多个挑战性基准上表现优异，包括AIME2024/AIME2025、GPQA-Diamond和LiveCodeBench-v6。使用本方法，4B规模模型在代理推理上可超越32B模型。我们还公开了一个高质量的真实端到端代理SFT数据集和高质强化学习数据集。代码与模型见：https://github.com/Gen-Verse/Open-AgentRL
原文摘要 · Abstract (English)
Recently, the emergence of agentic RL has showcased that RL could also effectively improve the agentic reasoning ability of LLMs, yet the key design principles and optimal practices remain unclear. In this work, we conduct a comprehensive and systematic investigation to demystify reinforcement learning in agentic reasoning from three key perspectives: data, algorithm, and reasoning mode. We highlight our key insights: (i) Replacing stitched synthetic trajectories with real end-to-end tool-use trajectories yields a far stronger SFT initialization; high-diversity, model-aware datasets sustain exploration and markedly improve RL performance. (ii) Exploration-friendly techniques are crucial for agentic RL, such as clip higher, overlong reward shaping, and maintaining adequate policy entropy could improve the training efficiency. (iii) A deliberative strategy with fewer tool calls outperforms frequent tool calls or verbose self-reasoning, improving tool efficiency and final accuracy. Together, these simple practices consistently enhance agentic reasoning and training efficiency, achieving strong results on challenging benchmarks with smaller models, and establishing a practical baseline for future agentic RL research. Beyond these empirical insights, we further contribute a high-quality, real end-to-end agentic SFT dataset along with a high-quality RL dataset, and demonstrate the effectiveness of our insights in boosting the agentic reasoning ability of LLMs across four challenging benchmarks, including AIME2024/AIME2025, GPQA-Diamond, and LiveCodeBench-v6. With our recipes, 4B-sized models could also achieve superior agentic reasoning performance compared to 32B-sized models. Code and models: https://github.com/Gen-Verse/Open-AgentRL
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。