arXiv:2510.04695cs.AI2025-10被引 5

分离搜索与回答训练,提升大模型工具调用准确性

Beyond Outcome Reward: Decoupling Search and Answering Improves LLM Agents

  • 分两阶段训练:先优化搜索召回率,再优化答案准确率
  • 在7个问答数据集上,搜索召回率和答案准确率均显著提升
  • 适合需要高可靠搜索能力的智能代理系统

让大语言模型使用搜索工具是克服知识截止和幻觉问题的有效途径。现有方法多采用强化学习,通过最终答案的奖励(如精确匹配)来联合训练推理与检索。我们发现,仅依赖结果奖励会导致搜索行为出现系统性缺陷,包括不调用工具、生成无效查询、重复搜索等,进而损害最终答案质量。为此,提出DeSA(Decoupling Search-and-Answering)框架:第一阶段用检索召回率作为奖励,专门优化搜索效果;第二阶段用最终答案奖励优化回答生成。在七个QA基准上,DeSA训练的代理在搜索行为和答案准确率上均优于单一阶段的基线,且超越同时优化召回与结果的方案,证明显式解耦的必要性。

原文摘要 · Abstract (English)

Enabling large language models (LLMs) to utilize search tools offers a promising path to overcoming fundamental limitations such as knowledge cutoffs and hallucinations. Recent work has explored reinforcement learning (RL) for training search-augmented agents that interleave reasoning and retrieval before answering. These approaches usually rely on outcome-based rewards (e.g., exact match), implicitly assuming that optimizing for final answers will also yield effective intermediate search behaviors. Our analysis challenges this assumption: we uncover multiple systematic deficiencies in search that arise under outcome-only training and ultimately degrade final answer quality, including failure to invoke tools, invalid queries, and redundant searches. To address these shortcomings, we introduce DeSA (Decoupling Search-and-Answering), a simple two-stage training framework that explicitly separates search optimization from answer generation. In Stage 1, agents are trained to improve search effectiveness with retrieval recall-based rewards. In Stage 2, outcome rewards are employed to optimize final answer generation. Across seven QA benchmarks, DeSA-trained agents consistently improve search behaviors, delivering substantially higher search recall and answer accuracy than outcome-only baselines. Notably, DeSA outperforms single-stage training approaches that simultaneously optimize recall and outcome rewards, underscoring the necessity of explicitly decoupling the two objectives.

大模型搜索增强强化学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。