让大模型在用工具解题时，学会从每一步的回望中自我改进。
TurnSight: Turn-Level Hindsight Self-Distillation for Tool-Integrated Reasoning
- 基于每轮交互的回溯视角生成监督信号，更贴近真实决策过程。
- 通过多前瞻视角一致性筛选可靠信号，提升长期任务优化效果。
- 适合需要精细策略学习的复杂工具调用任务，如代码生成与推理。
工具集成推理（TIR）使大语言模型通过迭代使用工具解决复杂任务。然而，现有强化学习方法通常依赖轨迹级监督，在长程TIR场景中难以实现细粒度的信用分配。在线自蒸馏可通过具备特权上下文的教师分支提供更密集的信号，但现有方法的上下文通常来自真实答案或检索到的技能，未必反映智能体实际访问的状态。此外，词元级监督无法捕捉工具交互的回合结构。为此，我们提出TurnSight，一种基于回合级回溯自蒸馏的框架，直接从执行条件化的回溯中提取监督信号。该方法构建多个具有不同前瞻视野的回溯视图，并通过跨视野方向一致性筛选可靠监督信号。最终，将选定的回溯信号在兄弟回放序列间归一化，并自适应地调节强化学习优势，同时保留其原始优化方向。在三个基准上的大量实验验证了TurnSight的有效性。代码已开源：https://github.com/quchangle1/TurnSight。
原文摘要 · Abstract (English)
Tool-Integrated Reasoning (TIR) enables LLMs to solve complex tasks through iterative tool interactions. However, existing reinforcement learning methods often rely on trajectory-level supervision, limiting fine-grained credit assignment in long-horizon TIR scenarios. On-policy self-distillation offers denser signals through teacher branches with privileged context, but existing approaches typically derive such context from ground-truth answers or retrieved skills, which may not reflect the states actually visited by the agent. Moreover, token-level supervision fails to capture the turn-level structure of tool interactions. To address this, we propose TurnSight, a turn-level hindsight self-distillation framework that derives supervision directly from execution-conditioned hindsight. It then constructs multiple hindsight views with different lookahead horizons and selects reliable supervision through cross-horizon directional agreement. Finally, the selected hindsight signal is normalized across sibling rollouts and used to adaptively modulate RL advantages while preserving their original optimization direction. Extensive experiments on three benchmarks demonstrate the effectiveness of TurnSight. Our codes are available at https://github.com/quchangle1/TurnSight.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。