用兄弟回溯对比引导信用分配，让强化学习更稳地完成长序列工具任务。
Keep Policy Gradient in Charge: Sibling-Guided Credit Distillation for Long-Horizon Tool-Use Agents
- 通过对比成功与失败的回溯轨迹，生成训练专用的信用参考。
- 在AppWorld和tau^3-airline上，测试指标分别提升至45.6和0.602。
- 适合追求高可靠性长序列决策的AI代理开发人员。
长时序工具使用强化学习依赖结果验证，但轨迹级优势信号被广播到推理、API调用和答案词元中。直接自蒸馏虽能提供更密集信号，但在实验中可能因重复教师行为而破坏工具使用能力。本文提出兄弟引导信用蒸馏（SGCD），将蒸馏用于有界信用加权而非替代策略损失。动态采样生成混合成功与失败的兄弟回溯轨迹；外部大模型总结其差异作为仅训练用的信用参考；解耦的师生偏差重塑GRPO词元优势。部署的学生仅接收清洁任务提示。在AppWorld和tau^3-airline上，SGCD的保留点估计优于GRPO系列基线：AppWorld测试正常集从42.9升至45.6，挑战集从24.7升至27.0；tau^3-airline保留评估器得分从0.583提升至0.602。结果支持一个设计准则：长时序工具使用智能体应以蒸馏引导信用分配，同时保持策略梯度主导动作更新。
原文摘要 · Abstract (English)
Long-horizon tool-use reinforcement learning learns from outcome verification, but trajectory-level advantages are broadcast over reasoning, API, and answer tokens. Direct self-distillation can supply a denser signal, but in our experiments it can also destroy tool use by rehearsing teacher behavior without identifying which actions the verifier rewards. We introduce Sibling-Guided Credit Distillation (SGCD), which uses distillation for bounded credit weighting rather than as a competing actor loss. Dynamic sampling produces mixed successful and failed sibling rollouts; an external LLM summarizes their contrast into a training-only credit reference; and detached teacher/student divergence reshapes GRPO token advantages. The deployed student receives only the clean task prompt. Across AppWorld and tau^3-airline, SGCD reports higher held-out point estimates than GRPO-family comparators: AppWorld TGC improves from 42.9 to 45.6 on test_normal and from 24.7 to 27.0 on test_challenge, and tau^3-airline held-out evaluator score improves from 0.583 to 0.602. These results support a narrow design rule for long-horizon tool-use agents: use distillation to guide credit assignment while keeping policy gradient in charge of the actor update.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。