让智能体通过全局视角学习反思,提升长程任务规划能力。
LoongReflect: Boosting Long-Horizon Reflection in Search Agents via Global Perspective Distillation

- 将反思视为记忆控制策略,结合显式回溯动作构建可逆轨迹树。
- 在多跳检索生成与数学推理任务上,显著优于仅依赖结果反馈的基线方法。
- 适合研究长程推理、自主决策或强化学习中的反思机制的读者。
大型语言模型代理在解决涉及规划、工具使用和记忆的复杂任务时,越来越依赖长程推理。其中关键能力是反思:评估路径进展、识别缺失证据和不可靠中间状态,并决定是否继续、修正或放弃当前分支。然而,有效学习反思极具挑战性,因为反思在当前分支内局部执行,而其价值只能通过最终路径结果来判断。这种局部-全局不匹配导致基于结果的强化学习仅提供局部、稀疏且延迟的监督信号。为此,我们提出 LoongReflect,一种将反思建模为记忆控制策略的训练框架。代理在可逆轨迹树上运行,使用显式的反思与回溯动作。反思将已验证事实、缺失证据和分支特异性风险存入工作内存,回溯则移除不可靠分支并保留简洁纠错经验。为学习该策略,LoongReflect 通过前瞻式、外梯度风格的协调机制融合两种互补信号:快速通道从特权教师中蒸馏全局信息驱动的反思行为,监督限于反思与回溯标记;慢速通道使用基于结果的 GRPO 优化完整轨迹,使局部控制决策与最终任务成功对齐。在多跳检索增强生成与数学推理基准上的实验表明,相比仅依赖结果的强化学习与自蒸馏基线,该方法实现持续性能提升。
原文摘要 · Abstract (English)
Large language model agents increasingly rely on long-horizon reasoning to solve complex tasks involving planning, tool use, and memory. A critical capability in such settings is reflection: assessing trajectory progress, identifying missing evidence and unreliable intermediate states, and deciding whether to continue, revise, or abandon the current branch. Learning effective reflection, however, is challenging because reflection is performed locally within the current branch, whereas its utility can only be determined by its contribution to the final trajectory outcome. This local-global mismatch makes outcome-based reinforcement learning provide only local, sparse and delayed supervision for reflective decisions. To solve these, we propose LoongReflect, a training framework that formulates reflection as a memory-control policy. The agent operates over a reversible trajectory tree using explicit reflect and backtrack actions. Reflection consolidates verified facts, missing evidence, and branch-specific risks into working memory, while backtracking removes an unreliable branch from the active context and preserves a concise corrective lesson. To learn this policy, LoongReflect combines two complementary signals through a look-ahead, extragradient-style coordination mechanism. A fast channel distills globally informed reflective behavior from a privileged teacher, with supervision restricted to reflection and backtracking tokens. A slow channel optimizes complete trajectories using outcome-based GRPO, aligning local control decisions with final task success. Experiments on multi-hop retrieval-augmented generation and mathematical reasoning benchmarks demonstrate consistent improvements over outcome-only reinforcement learning and self-distillation baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。