用真实答案引导检索,让大模型搜得准、答得对。
Search-GRT: Guided Retrieval Training of Search Agents to Optimize for Complex Question Answering

- 训练时用正确文档限制检索范围,增强学习信号
- 多跳问答任务性能提升超40%,且训练步数更少
- 适合需要精准检索的复杂问答场景
大语言模型有效使用搜索引擎仍是挑战,尤其在复杂的多跳问答(MHQA)任务中。这类任务要求模型将问题拆解为子查询,检索相关信息,并从多个来源合成答案,常因早期检索不佳导致错误累积。强化学习虽有潜力提升搜索能力,但训练中奖励稀疏,难以有效学习。为此,我们提出引导检索训练(GRT),在强化学习训练过程中利用真实答案信息限制检索过程,聚焦于精选的相关文档,提供更强的学习信号,缓解奖励稀疏问题,提升生成准确子查询和合成正确答案的能力。实验表明,GRT在多种问答任务上持续优于现有方法(如Search-R1),在多跳问答任务中性能提升超过40%;同时,训练效率更高,用更少步骤达到更好效果。
原文摘要 · Abstract (English)
The effective use of search engines by large language models (LLMs) remains a significant challenge, particularly in complex, multi-hop question-answering (MHQA) tasks. These tasks require the model to decompose questions into subqueries, retrieve relevant information, and synthesize answers from multiple sources, often leading to cascading errors due to poor retrieval in early stages. Reinforcement learning (RL) has shown promise in improving LLMs' search capabilities, but it often suffers from sparse rewards during training, hindering the model's ability to learn effectively. To address these challenges, we introduce Guided Retrieval Training (GRT), a novel method that improves the performance of a search agent by restricting the retrieval process during RL training using ground truth information. By focusing on a curated set of relevant documents, GRT provides the model with a stronger learning signal, mitigating the problem of sparse rewards and improving its ability to generate accurate subqueries and synthesize correct answers. Our experimental results demonstrate that GRT achieves consistent performance improvements over existing methods, such as Search-R1, across a wide range of question-answering (QA) tasks. Notably, GRT excels in MHQA tasks, achieving over 40% improvements in performance. Additionally, GRT enhances training efficiency by achieving better QA performance with fewer training steps.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。