用工具调用完整度做奖励，让大模型更会规划
Encouraging Good Processes Without the Need for Good Answers: Reinforcement Learning for LLM Agent Planning
- 分离规划与回答训练，专注优化任务规划能力
- 规划性能提升8%-12%，最终答案质量提高5%-6%
- 无需正确答案数据，适合强化学习训练复杂决策
大型语言模型（LLM）代理的功能主要依赖于行动规划和答案总结两项能力。其中，行动规划是决定代理性能的核心能力。然而，现有训练范式采用端到端、多目标优化，联合训练这两项能力，面临目标分配失衡和可验证数据稀缺两大挑战，难以提升规划能力。为此，我们提出基于工具使用奖励的强化学习框架（RLTR），通过解耦训练过程，实现对规划模块的聚焦式单目标优化。关键在于，RLTR引入基于工具使用完整度的奖励信号，直接评估工具调用序列的质量，相比评估最终回答内容，提供更直接可靠的训练信号，从而无需依赖可验证数据。实验表明，相比端到端基线，RLTR在规划性能上提升8%-12%；该增强的规划能力进一步带来整体代理系统最终响应质量5%-6%的提升。
原文摘要 · Abstract (English)
The functionality of Large Language Model (LLM) agents is primarily determined by two capabilities: action planning and answer summarization. The former, action planning, is the core capability that dictates an agent's performance. However, prevailing training paradigms employ end-to-end, multi-objective optimization that jointly trains both capabilities. This paradigm faces two critical challenges: imbalanced optimization objective allocation and scarcity of verifiable data, making it difficult to enhance the agent's planning capability. To address these challenges, we propose Reinforcement Learning with Tool-use Rewards (RLTR), a novel framework that decouples the training process to enable a focused, single-objective optimization of the planning module. Crucially, RLTR introduces a reward signal based on tool-use completeness to directly evaluate the quality of tool invocation sequences. This method offers a more direct and reliable training signal than assessing the final response content, thereby obviating the need for verifiable data. Our experiments demonstrate that RLTR achieves an 8%-12% improvement in planning performance compared to end-to-end baselines. Moreover, this enhanced planning capability, in turn, translates to a 5%-6% increase in the final response quality of the overall agent system.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。