TIER让大模型用工具时能自动获得每一步的反馈,支持多种解法。
TIER: Trajectory-Invariant Execution Rewards for Multi-Step Tool Composition

- 基于函数接口和运行时结果生成奖励,不依赖人工标注路径。
- 在6步任务中准确率超90%,而传统方法4步后崩溃。
- 适合需要多步骤工具调用的复杂任务,如自动化工作流设计。
工具使用使大语言模型通过一系列API调用解决复杂任务,但现有强化学习方法难以扩展到多步组合场景。基于结果的奖励提供稀疏反馈,而轨迹监督奖励依赖标注的参考解法,惩罚有效替代方案,限制可扩展性。我们提出TIER:轨迹无关执行奖励,一种直接从函数模式和运行时执行中提取监督的奖励框架。该奖励分解为格式正确性、模式符合性、执行成功性和答案正确性,提供来自工具使用各步骤细粒度验证的密集、可解释的序列级反馈。此设计使任何有效执行路径都能获得奖励,自然支持多种解题策略,并适应不断演进的工具接口。在按深度分层(1至6步)的DepthBench基准上,TIER在所有步骤中均实现超过90%的准确率,而轨迹监督奖励在第4步后迅速下降。我们在BFCL v3和NestFUL等基准上也展示了稳定提升。消融实验表明所有奖励组件均必要,强调多层级监督对组合推理的重要性。
原文摘要 · Abstract (English)
Tool use enables large language models to solve complex tasks through sequences of API calls, yet existing reinforcement learning approaches fail to scale to multi-step composition settings. Outcome-based rewards provide only sparse feedback, while trajectory-supervised rewards depend on annotated reference solutions, penalizing valid alternatives and limiting scalability. We propose TIER: Trajectory-Invariant Execution Rewards, a reward framework that derives supervision directly from function schemas and runtime execution, rather than from reference trajectories. The reward decomposes into format validity, schema adherence, execution success, and answer correctness, providing dense, interpretable sequence-level feedback derived from fine-grained verification of individual steps of tool use. This design allows any valid execution path to receive credit, naturally supporting multiple solution strategies and adapting to evolving tool interfaces. On DepthBench, a compositional benchmark stratified by depth (1 to 6 steps), TIER achieves >90% accuracy across steps, where trajectory-supervised rewards collapse beyond step-4. We further demonstrate consistent gains on benchmarks like BFCL v3 and NestFUL. Ablation studies confirm that all reward components are necessary, highlighting the importance of multi-level supervision for compositional reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。