让大模型学会智能调用搜索，提升复杂任务推理能力
R-Search: Empowering LLM Reasoning with Search via Multi-Reward Reinforcement Learning
- 用多奖励强化学习指导模型决定何时搜索、何时推理
- 在7个数据集上比先进检索增强方法提升最高32.2%
- 适合需要深度知识交互的复杂逻辑推理场景
大语言模型在多步推理方面已取得显著进展，但将其推理能力扩展至与搜索深度融合仍面临挑战，因模型常无法识别最优的推理-搜索交互路径，导致回答质量不佳。我们提出R-Search，一种新型强化学习框架，用于推理-搜索融合，使大模型能够自主执行多步推理并深度交互搜索，通过多奖励信号学习最优推理-搜索轨迹，从而提升复杂逻辑与知识密集型任务的回答质量。R-Search动态指导模型决定何时检索或推理，并全局整合关键证据以增强推理与搜索间的深度知识交互。在强化学习训练中，R-Search提供多阶段、多类型奖励，联合优化推理-搜索轨迹。在七个数据集上的实验表明，R-Search相比先进RAG基线，在域内任务上性能提升最高达32.2%，域外任务提升25.1%。代码与数据已在https://github.com/QingFei1/R-Search公开。
原文摘要 · Abstract (English)
Large language models (LLMs) have notably progressed in multi-step and long-chain reasoning. However, extending their reasoning capabilities to encompass deep interactions with search remains a non-trivial challenge, as models often fail to identify optimal reasoning-search interaction trajectories, resulting in suboptimal responses. We propose R-Search, a novel reinforcement learning framework for Reasoning-Search integration, designed to enable LLMs to autonomously execute multi-step reasoning with deep search interaction, and learn optimal reasoning search interaction trajectories via multi-reward signals, improving response quality in complex logic- and knowledge-intensive tasks. R-Search guides the LLM to dynamically decide when to retrieve or reason, while globally integrating key evidence to enhance deep knowledge interaction between reasoning and search. During RL training, R-Search provides multi-stage, multi-type rewards to jointly optimize the reasoning-search trajectory. Experiments on seven datasets show that R-Search outperforms advanced RAG baselines by up to 32.2% (in-domain) and 25.1% (out-of-domain). The code and data are available at https://github.com/QingFei1/R-Search.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。