让搜索模型每一步都朝着正确答案走,提升复杂问答准确率。
OASES: Outcome-Aligned Search-Evaluation Co-Training for Agentic Search

- 用问题答案反推每步搜索的合理性,生成精准过程奖励。
- 搜索策略与评估器联合训练,动态适应行为变化,避免奖励失效。
- 在五个多跳问答数据集上显著超越现有强化学习方法,适合复杂推理任务。
代理式搜索使语言模型通过多步动态获取外部证据来解决知识密集型任务。当前广泛采用的可验证奖励强化学习(RLVR)依赖最终结果奖励,但该奖励稀疏且难以对中间搜索动作进行有效信用分配。现有过程奖励方法试图通过代理信号、外部评估器或基于似然的信息增益来增强监督,但代理奖励可能偏离最终目标,固定评估器随策略演化而过时,导致过程监督不可靠。为此,我们提出OASES框架,通过评估每个中间搜索状态对原始问题回答的支持程度,生成与最终结果对齐的过程奖励。同时,该框架联合训练搜索策略与状态评估器,使评估器能随策略演进而自适应调整,提供更可靠的监督信号。在五个多跳问答基准上的实验表明,OASES持续优于强基线方法,分析进一步验证了对齐结果的过程奖励和搜索-评估联合训练的有效性。
原文摘要 · Abstract (English)
Agentic search enables language models to solve knowledge-intensive tasks by adaptively acquiring external evidence over multiple steps. Reinforcement learning with verifiable rewards (RLVR) has emerged as a widely adopted training paradigm for search agents, yet outcome-only rewards are sparse and provide limited credit assignment for intermediate search actions. Existing process-reward methods therefore seek to densify supervision through proxy signals, external evaluators, or likelihood-based information gain. However, proxy rewards can deviate from the final outcome objective, while fixed evaluators can become stale as the search policy evolves, leading to unreliable process supervision. To address these challenges, we propose OASES, an Outcome-Aligned Search-Evaluation Supervision framework for agentic search. OASES derives outcome-aligned process rewards by evaluating how well each intermediate search state supports answering the original question. It further co-trains the search policy and the state evaluator on policy, allowing the evaluator to adapt to evolving search behavior and provide more reliable process rewards. Experiments on five multi-hop QA benchmarks show that OASES consistently outperforms strong RL baselines, with further analyses confirming the benefits of outcome-aligned process rewards and search-evaluation co-training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。