arXiv:2606.16215cs.CLcs.AI2026-06

用专家轨迹辅助训练,让多轮工具调用模型更灵活高效。

PACT: Privileged Trace Co-Training for Multi-Turn Tool-Use Agents

论文配图:PACT: Privileged Trace Co-Training for Multi-Turn Tool-Use Agents
图 1 · 摘自论文原文
  • 训练时用专家轨迹做优化信号,推理时仍保持纯提示驱动。
  • 在多个基准上超越强基线,提升多轮任务成功率。
  • 适合需要灵活决策的复杂工具调用场景。

多轮工具调用智能体需在多步交互中推理、调用工具并适应观察结果。现有方法面临奖励稀疏和信用分配困难(强化学习),或过度依赖固定轨迹(监督微调)的问题。本文提出PACT框架,将专家轨迹仅作为训练时的优化信号而非推理时的指导。该方法通过两个互补信号引导优化:基于轨迹的强化学习代理评估纯提示生成的轨迹,以及分组件的监督微调损失,对推理前缀和工具调用进行渐弱监督。为减少对训练轨迹的依赖,引入纯提示锚定机制。此外,通过隐式轨迹视角揭示了专家轨迹如何在不参与推理的情况下指导优化。在FTRL、BFCL和ToolHop数据集上的实验表明,PACT持续优于主流的SFT与RL基线,验证了特权轨迹协同训练的有效性。

原文摘要 · Abstract (English)

Multi-turn tool-use agents must reason, call tools, and adapt to observations across several interaction turns. Post-training such agents is challenging, as reinforcement learning often suffers from sparse rewards and weak credit assignment despite matching the prompt-only inference setting, while supervised fine-tuning on expert traces provides dense process supervision but can over-constrain the model to fixed trajectories. To tackle this, we propose PACT, a Privileged trAce Co-Training framework for multi-turn tool-use agents. The key idea is to use expert traces only as training-time optimization signals rather than rollout-time hints. PACT keeps rollout generation prompt-only, then uses expert traces to guide optimization through two complementary signals: a trace-conditioned RL surrogate that evaluates prompt-only rollouts under expert-trace context, and a component-aware SFT loss that supervises reasoning prefixes and tool-calls with annealed strength. To reduce over-reliance on the training-only trace context, PACT further introduces a prompt-only anchoring. We also provide a latent-trace view that connects the two trace-based objectives and explains how expert traces can guide optimization without being used during rollout generation. Experiments on FTRL, BFCL, and ToolHop show that PACT consistently improves over strong SFT- and RL-based baselines, highlighting the value of privileged trace co-training for multi-turn tool-use learning.

工具调用多轮推理强化学习监督微调

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。