给大模型训练长程工具使用智能体，提供可复现的强化学习配方。
Demystifying Reinforcement Learning for Long-Horizon Tool-Using Agents: A Comprehensive Recipe
- 按5个维度系统测试强化学习设计，找出关键影响因素。
- 约1000条均衡难度数据样本效果最佳，兼顾域内域外表现。
- 环境稳定至关重要，否则策略会退化，适合做智能体研发的团队参考。
强化学习对将大语言模型演化为具备长程规划能力的自主智能体至关重要，但在复杂多轮环境中实现强化学习的规模化实践仍缺乏清晰路径。本文以需协调多种工具、满足多重约束的旅行规划任务（TravelPlanner）为挑战性测试平台，从奖励设计、模型规模、数据构成、算法选择和环境稳定性五个维度系统展开实证研究。通过受控实验得出7项关键结论：(1) 奖励机制与算法选择具有规模依赖性——小模型受益于分阶段奖励与增强探索，大模型则在简单密集奖励下高效收敛；(2) 约1000条训练样本且难度分布均衡时达到性能拐点，显著提升域内与域外表现；(3) 环境稳定性对防止策略退化至关重要。基于此提炼出可复用的强化学习训练配方，所训练模型在TravelPlanner上达到当前最优水平，显著超越主流大模型。
原文摘要 · Abstract (English)
Reinforcement Learning (RL) is essential for evolving Large Language Models (LLMs) into autonomous agents capable of long-horizon planning, yet a practical recipe for scaling RL in complex, multi-turn environments remains elusive. This paper presents a systematic empirical study using TravelPlanner, a challenging testbed requiring tool orchestration to satisfy multifaceted constraints. We decompose the agentic RL design space along 5 axes: reward shaping, model scaling, data composition, algorithm selection, and environmental stability. Our controlled experiments yield 7 key takeaways, e.g., (1) reward and algorithm choices are scale-dependent as smaller models benefit from staged rewards and enhanced exploration, whereas larger models converge efficiently with simpler dense rewards, (2) ~ 1K training samples with a balanced difficulty mixture mark a sweet spot for both in-domain and out-of-domain performance, and (3) environmental stability is critical to prevent policy degradation. Based on our distilled recipe, our RL-trained models achieve state-of-the-art performance on TravelPlanner, significantly outperforming leading LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。