只在关键失败动作上做强化训练,提升长程智能体效率
HINT-SD: Targeted Hindsight Self-Distillation for Long-Horizon Agents

- 用全程回溯分析定位失败相关动作,精准选择训练区间
- 相比每步反馈,平均性能提升13.60个百分点,训练速度加快2.26倍
- 适合需要高效长序列决策的AI助手、自动化工具等场景
使用强化学习训练长时序大模型智能体面临挑战:稀疏的最终奖励只能判断任务成败,无法定位是哪个中间动作导致结果。现有方法通过逐轮生成奖励或文本提示,或采用反馈条件下的自蒸馏来缓解。但每步都生成反馈效率低,且固定或错位的反馈难以指导真正影响失败的动作。为此,我们提出HINT-SD,一种基于全轨迹回溯的靶向自蒸馏框架,仅对与失败相关的动作段落应用反馈蒸馏。在BFCL v3和AppWorld上的实验表明,该方法相比密集逐轮反馈基线平均提升13.60个百分点,同时每训练步骤耗时减少2.26倍,证明选择蒸馏位置比盲目训练更有效。
原文摘要 · Abstract (English)
Training long-horizon LLM agents with reinforcement learning is challenging because sparse outcome rewards reveal whether a task succeeds, but not which intermediate actions caused the outcome or how they should be corrected. Recent methods alleviate this issue by generating rewards or textual hints from turn-level action-output signals, or by using feedback-conditioned self-distillation. However, generating feedback at every turn is inefficient when many intermediate turns are already successful or neutral, and applying feedback at a fixed or misaligned turn often fails to supervise the actions that contributed to the failure. To bridge this gap, we propose HINT-SD, a targeted self-distillation framework that uses full-trajectory hindsight to select failure-relevant actions and applies feedback-conditioned distillation only to targeted action spans. Experiments on BFCL v3 and AppWorld show that our method outperforms the dense per-turn feedback baseline by up to 13.60 percentage points on average while achieving a 2.26$\times$ reduction in time per training step, suggesting that selecting where to distill is key to effective and efficient long-horizon agent training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。