arXiv:2608.07371cs.LGcs.CL2026-08

通过轨迹相对回溯蒸馏,让智能体决策更精准。

Trajectory-Relative Hindsight Distillation for Agentic Reinforcement Learning

论文配图:Trajectory-Relative Hindsight Distillation for Agentic Reinforcement Learning
图 1 · 摘自论文原文
  • 基于轨迹内决策点的回溯对比,动态分配监督信号
  • 在WebShop上将成功率从56.4%提升至75.2%
  • 适用于各类大模型与环境,提升决策一致性

近期的智能体强化学习方法利用回溯机制弥补稀疏结果奖励。然而,一次完整轨迹可产生大量回溯信号,其在各决策回合中的合理分配尚不明确。本文提出TRIAL,一种轨迹相对回溯蒸馏框架,采用统一的回合对齐评分协议。对于每个决策回合,TRIAL提取该决策实际后果的成果视图,并评估同一响应在常规与回溯条件下的表现。符号化对数概率差决定词级监督的方向与局部强度,回合级强度在实际轨迹上联合归一化。所得分配乘子具有合格词加权均值为1的特性,实现密集监督在回合间的再分配且保持平均乘子不变。在WebShop和ALFWorld上的实验表明,TRIAL在所有八个骨干模型、环境与评估指标组合中均优于GRPO,六种方法中有六次取得最佳或并列最佳表现。在WebShop上使用Qwen3-1.7B时,成功率达75.2%(原56.4%),任务得分提升至85.7%(原78.7%)。受控消融实验进一步表明,轨迹相对回合分配带来的增益显著超过仅用密集回溯蒸馏的效果。

原文摘要 · Abstract (English)

Recent agentic reinforcement learning methods use hindsight to complement sparse outcome rewards. However, a completed rollout can yield many such signals, leaving their appropriate allocation across turns unclear. We introduce TRIAL, a trajectory-relative hindsight distillation framework with a unified turn-aligned scoring protocol. For each decision turn, TRIAL extracts an outcome view of that decision's realized consequence and evaluates the same response under ordinary and hindsight-conditioned contexts. The signed log-probability gap determines the direction and local strength of token-level supervision, while turn-level magnitudes are normalized jointly over the realized trajectory. The resulting allocation multipliers have an eligible-token-weighted mean of one, redistributing dense supervision across turns while fixing its average multiplier. Experiments on WebShop and ALFWorld with different backbones show that TRIAL outperforms GRPO across all eight combinations of backbone, environment, and evaluation metric, while achieving the best or tied-best performance among six methods on six of them. On WebShop with Qwen3-1.7B, TRIAL improves the success rate from 56.4% to 75.2% and the task score from 78.7% to 85.7%. Controlled ablations further show that trajectory-relative turn allocation provides substantial gains beyond those of dense hindsight distillation alone.

强化学习智能体回溯蒸馏

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。