arXiv:2608.06128cs.AI2026-08

让搜索代理真正用好外部信息,避免仅凭记忆猜答案。

Contextual Information Policy Optimization for Search Agents

论文配图:Contextual Information Policy Optimization for Search Agents
图 1 · 摘自论文原文
  • 用外部信息时给奖励,引导推理紧跟检索内容
  • 在7个任务上显著减少先入为主式推理
  • 无需人工标注或额外模型,通用性强

搜索代理通过获取和使用外部证据,将大语言模型从静态参数记忆中解放出来,用于多步推理。对于涉及复杂或动态信息的知识密集型任务,其可靠性不仅取决于是否检索到相关证据,还在于能否用证据指导后续推理。然而现有方法主要奖励最终答案正确性或中间进展,未直接评估检索后行为是否基于所获证据。这种错配导致代理倾向于依赖内部知识形成结论,并仅用检索来验证已有判断,产生确认偏见并低效利用证据。为此,我们提出上下文信息策略优化(CIPO),一种以证据为核心的强化学习框架,显式对齐策略优化与外部证据使用。CIPO为受检索信息影响的每一步推理动作分配密集的轮次级奖励,同时结合全局结果奖励以保障答案正确性。该机制抑制脱离证据的猜测,促进由检索事实引导或修正的推理路径。重要的是,CIPO无需人类过程标注或额外奖励模型。在七个领域内和领域外基准上的实验表明,CIPO显著降低了先入为主推理的出现频率,并在多数任务上取得优异表现。

原文摘要 · Abstract (English)

Search agents extend large language models beyond static parametric memory by enabling them to acquire and use external evidence during multi-step reasoning. For knowledge-intensive tasks involving complex or evolving information, their reliability depends not only on retrieving relevant evidence but also on using it to guide subsequent reasoning. However, existing methods primarily reward final-answer correctness or intermediate progress, without directly assessing whether post-retrieval actions are grounded in the retrieved evidence. This misalignment encourages prior-driven reasoning: agents form conclusions based on internal knowledge and use retrieval mainly to confirm them, resulting in confirmation bias and inefficient evidence use. To address this issue, we propose Contextual Information Policy Optimization (CIPO), an evidence-oriented reinforcement learning framework that explicitly aligns policy optimization with external evidence use. CIPO assigns dense, turn-level credit to reasoning actions influenced by retrieved information, while combining this evidence-use signal with a global outcome reward to preserve answer correctness. With this manner, CIPO discourages evidence-detached guesses and promotes reasoning trajectories in which retrieved facts can guide or revise subsequent reasoning. Importantly, CIPO requires neither human process annotations nor an additional reward model. Extensive experiments on seven in-domain and out-of-domain benchmarks show that CIPO reduces the prevalence of prior-driven reasoning and achieves excellent performance on most tasks.

搜索代理强化学习证据使用推理优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。