arXiv:2605.17873cs.LGcs.AI2026-05被引 4

只在关键失败动作上做强化训练,提升长程智能体效率

HINT-SD: Targeted Hindsight Self-Distillation for Long-Horizon Agents

论文配图:HINT-SD: Targeted Hindsight Self-Distillation for Long-Horizon Agents
图 1 · 摘自论文原文
  • 用全程回溯分析定位失败相关动作,精准选择训练区间
  • 相比每步反馈,平均性能提升13.60个百分点,训练速度加快2.26倍
  • 适合需要高效长序列决策的AI助手、自动化工具等场景

使用强化学习训练长时序大模型智能体面临挑战:稀疏的最终奖励只能判断任务成败,无法定位是哪个中间动作导致结果。现有方法通过逐轮生成奖励或文本提示,或采用反馈条件下的自蒸馏来缓解。但每步都生成反馈效率低,且固定或错位的反馈难以指导真正影响失败的动作。为此,我们提出HINT-SD,一种基于全轨迹回溯的靶向自蒸馏框架,仅对与失败相关的动作段落应用反馈蒸馏。在BFCL v3和AppWorld上的实验表明,该方法相比密集逐轮反馈基线平均提升13.60个百分点,同时每训练步骤耗时减少2.26倍,证明选择蒸馏位置比盲目训练更有效。

原文摘要 · Abstract (English)

Training long-horizon LLM agents with reinforcement learning is challenging because sparse outcome rewards reveal whether a task succeeds, but not which intermediate actions caused the outcome or how they should be corrected. Recent methods alleviate this issue by generating rewards or textual hints from turn-level action-output signals, or by using feedback-conditioned self-distillation. However, generating feedback at every turn is inefficient when many intermediate turns are already successful or neutral, and applying feedback at a fixed or misaligned turn often fails to supervise the actions that contributed to the failure. To bridge this gap, we propose HINT-SD, a targeted self-distillation framework that uses full-trajectory hindsight to select failure-relevant actions and applies feedback-conditioned distillation only to targeted action spans. Experiments on BFCL v3 and AppWorld show that our method outperforms the dense per-turn feedback baseline by up to 13.60 percentage points on average while achieving a 2.26$\times$ reduction in time per training step, suggesting that selecting where to distill is key to effective and efficient long-horizon agent training.

强化学习长程决策自蒸馏智能体

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。