arXiv:2608.24794cs.AI2026-08

让搜索智能体与反馈机制共同进化,提升纠错能力。

CAFE: Self-Improving Search Agents Need Co-Evolving Feedback

论文配图:CAFE: Self-Improving Search Agents Need Co-Evolving Feedback
图 1 · 摘自论文原文
  • 共享参数模型交替扮演搜索代理和批评者角色
  • 在7个基准上平均表现优于现有方法,降低幻觉率
  • 适合需要持续优化的自主搜索系统研究者

结果监督的搜索智能体能学习何时何地检索证据,但最终奖励无法定位中间错误,也无法在错误累积前引导当前路径。将纠正反馈视为一种轨迹中的干预,将两个角色耦合:智能体需决定何时请求并使用反馈,而批评者需从受结果混淆的回溯中推断有效修正,其失败模式随智能体改进而变化。我们提出CAFE(耦合智能体-反馈进化)框架,其中共享参数模型交替承担搜索代理和批评者角色。CAFE从基础智能体自身失败轨迹中初始化反馈条件下的恢复机制,随后耦合在线与离线优化。在线强化学习中,比较反馈估计利用提示级“请求-跳过成功差距”塑造请求回报;反馈感知的优势重塑则在反馈前后重新加权词元优势。离线阶段,基于回溯的偏好优化从匹配的成功与失败轨迹中学习反馈。在七个智能体搜索基准上,CAFE平均表现优于评估的基于RL的搜索智能体,且在全部六个跨领域基准上保持优势,同时减少答案级幻觉。单方面消融实验显示,仅优化智能体或仅优化批评者最终会停滞,而交替更新两者则持续提升性能。这些发现表明,自进化的搜索智能体需要与其策略协同演化的反馈机制。

原文摘要 · Abstract (English)

Outcome-supervised search agents learn when and how to retrieve evidence, but terminal rewards neither localize intermediate errors nor redirect an ongoing trajectory before those errors compound. Treating corrective feedback as a learned in-trajectory intervention couples the two roles: the agent must decide when to request and use feedback, while the critic must infer useful corrections from outcome-confounded rollouts whose failure patterns shift as the agent improves. We introduce CAFE (Coupled Agent--Feedback Evolution), a framework in which a shared-parameter model alternates between search-agent and critic roles. CAFE initializes feedback-conditioned recovery from trajectories built around the base agent's own failures, then couples online and offline optimization. During online RL, a comparative feedback estimate uses a prompt-level call--skip success gap to shape request returns, while feedback-aware advantage shaping reweights token advantages before and after feedback. Offline, rollout-derived preference optimization learns feedback from matched successful and unsuccessful trajectories. On seven agentic search benchmarks, CAFE outperforms the evaluated RL-based search agents on average, retains its gains across all six out-of-domain benchmarks, and reduces answer-level hallucinations. One-sided ablations show that improving only the agent or only the critic eventually plateaus, whereas alternating the two updates continues to improve performance. These findings suggest that a self-improving search agent needs feedback that co-evolves with the policy it guides.

智能体强化学习反馈机制自我改进

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。