通过双向匹配实现细粒度奖励,提升大模型工具调用的精准性。
MatchTIR: Fine-Grained Supervision for Tool-Integrated Reasoning via Bipartite Matching
- 用双向匹配计算每步工具调用的奖励,区分有效与无效操作。
- 4B模型在长序列任务上超越多数8B模型,准确率显著提升。
- 适合需要精细推理与多轮工具交互的复杂任务研究者。
工具集成推理(TIR)使大语言模型通过穿插推理步骤与外部工具调用,解决复杂任务。然而,现有强化学习方法通常依赖结果或轨迹级奖励,对轨迹内所有步骤赋予相同优势,导致粗粒度信用分配无法区分有效工具调用与冗余或错误操作,尤其在长时序多轮场景中表现不佳。为此,我们提出MatchTIR框架,通过基于双向匹配的逐轮奖励分配与双层优势估计实现细粒度监督。具体而言,将信用分配建模为预测轨迹与真实轨迹间的双向匹配问题,采用两种分配策略生成密集的逐轮奖励。同时,为平衡局部步骤精度与全局任务成功率,引入双层优势估计机制,融合逐轮与轨迹级信号,为每个交互回合分配不同优势值。在三个基准上的大量实验表明MatchTIR性能优越。值得注意的是,我们的4B模型在长时序与多轮任务中超越多数8B模型。代码已开源:https://github.com/quchangle1/MatchTIR。
原文摘要 · Abstract (English)
Tool-Integrated Reasoning (TIR) empowers large language models (LLMs) to tackle complex tasks by interleaving reasoning steps with external tool interactions. However, existing reinforcement learning methods typically rely on outcome- or trajectory-level rewards, assigning uniform advantages to all steps within a trajectory. This coarse-grained credit assignment fails to distinguish effective tool calls from redundant or erroneous ones, particularly in long-horizon multi-turn scenarios. To address this, we propose MatchTIR, a framework that introduces fine-grained supervision via bipartite matching-based turn-level reward assignment and dual-level advantage estimation. Specifically, we formulate credit assignment as a bipartite matching problem between predicted and ground-truth traces, utilizing two assignment strategies to derive dense turn-level rewards. Furthermore, to balance local step precision with global task success, we introduce a dual-level advantage estimation scheme that integrates turn-level and trajectory-level signals, assigning distinct advantage values to individual interaction turns. Extensive experiments on three benchmarks demonstrate the superiority of MatchTIR. Notably, our 4B model surpasses the majority of 8B competitors, particularly in long-horizon and multi-turn tasks. Our codes are available at https://github.com/quchangle1/MatchTIR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。