让推理和检索联合优化,提升智能搜索问答准确率
CoSearch: Joint Training of Reasoning and Document Ranking via Reinforcement Learning for Agentic Search
- 用强化学习同时训练推理代理和生成式文档排序模型
- 在7个基准上相对基线提升最高达26.8%的F1分数
- 适合关注智能搜索系统性能提升的研究者
智能搜索任务要求代理通过多步推理、发起查询并整合检索信息来回答复杂问题,现有方法如Search-R1将检索系统视为固定工具,仅优化推理代理。初步实验显示,理想检索与固定检索之间存在高达26.8%的相对F1差距,表明检索是性能瓶颈。为此,我们提出CoSearch框架,通过组相对策略优化(GRPO)联合训练多步推理代理与生成式文档排序模型。为解决排序器因推理轨迹差异导致的训练难题,我们引入基于词级相似性的语义分组策略,在不增加采样的前提下构建有效优化组。同时设计复合奖励函数,融合排序质量信号与轨迹级反馈,提供即时与长期学习信号。在7个单跳与多跳QA基准上的实验表明,该方法持续优于强基线,消融实验证明各设计有效性。结果表明,推理与检索系统的联合训练既可行又高效,是未来搜索代理的关键要素。
原文摘要 · Abstract (English)
Agentic search -- the task of training agents that iteratively reason, issue queries, and synthesize retrieved information to answer complex questions -- has achieved remarkable progress through reinforcement learning (RL). However, existing approaches such as Search-R1, treat the retrieval system as a fixed tool, optimizing only the reasoning agent while the retrieval component remains unchanged. A preliminary experiment reveals that the gap between an oracle and a fixed retrieval system reaches up to +26.8% relative F1 improvement across seven QA benchmarks, suggesting that the retrieval system is a key bottleneck in scaling agentic search performance. Motivated by this finding, we propose CoSearch, a framework that jointly trains a multi-step reasoning agent and a generative document ranking model via Group Relative Policy Optimization (GRPO). To enable effective GRPO training for the ranker -- whose inputs vary across reasoning trajectories -- we introduce a semantic grouping strategy that clusters sub-queries by token-level similarity, forming valid optimization groups without additional rollouts. We further design a composite reward combining ranking quality signals with trajectory-level outcome feedback, providing the ranker with both immediate and long-term learning signals. Experiments on seven single-hop and multi-hop QA benchmarks demonstrate consistent improvements over strong baselines, with ablation studies validating each design choice. Our results show that joint training of the reasoning agent and retrieval system is both feasible and strongly performant, pointing to a key ingredient for future search agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。