让大模型学会主动上网搜索，提升推理准确性。
R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning
- 用强化学习教会大模型在推理时自主调用外部搜索。
- 在数学和编码任务上超越现有检索增强方法，接近GPT-4o-mini表现。
- 无需过程奖励或知识蒸馏，可冷启动且适配基础与指令模型。
现有大型推理模型虽已展示强化学习（RL）在提升大语言模型复杂推理能力上的潜力，但在数学、编程等挑战性任务中仍依赖内部知识，面对时效性强或知识密集型问题时易出错甚至幻觉。为此，我们提出R1-Searcher，一种基于结果的两阶段强化学习方法，旨在增强大模型的外部搜索能力。该方法使模型在推理过程中能自主调用外部搜索系统获取额外信息。我们的框架仅依赖强化学习，无需过程奖励或知识蒸馏即可实现冷启动。实验表明，该方法显著优于现有强基准的RAG方法，甚至在性能上超过闭源的GPT-4o-mini。
原文摘要 · Abstract (English)
Existing Large Reasoning Models (LRMs) have shown the potential of reinforcement learning (RL) to enhance the complex reasoning capabilities of Large Language Models~(LLMs). While they achieve remarkable performance on challenging tasks such as mathematics and coding, they often rely on their internal knowledge to solve problems, which can be inadequate for time-sensitive or knowledge-intensive questions, leading to inaccuracies and hallucinations. To address this, we propose \textbf{R1-Searcher}, a novel two-stage outcome-based RL approach designed to enhance the search capabilities of LLMs. This method allows LLMs to autonomously invoke external search systems to access additional knowledge during the reasoning process. Our framework relies exclusively on RL, without requiring process rewards or distillation for a cold start. % effectively generalizing to out-of-domain datasets and supporting both Base and Instruct models. Our experiments demonstrate that our method significantly outperforms previous strong RAG methods, even when compared to the closed-source GPT-4o-mini.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。