让大模型学会用搜索增强推理，提升问答准确率
Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning
- 用强化学习训练模型自主生成搜索关键词，边思考边查资料
- 在7个数据集上，性能比现有方法提升41%（7B模型）和20%（3B模型）
- 适合需要实时知识更新的AI系统开发者参考
高效获取外部知识和最新信息对大语言模型（LLMs）的有效推理与文本生成至关重要。在推理阶段通过提示让具备推理能力的LLM使用搜索引擎往往效果不佳，因为模型可能不具备与搜索引擎最优交互的能力。本文提出Search-R1，一种基于强化学习（RL）的推理框架扩展，使LLM在逐步推理过程中能自主生成（多个）搜索查询并实时获取检索结果。Search-R1通过多轮搜索交互优化推理路径，采用检索到的词元掩码实现稳定的强化学习训练，并使用简单的基于结果的奖励函数。在七个问答数据集上的实验表明，Search-R1在相同设置下相较于多种RAG基线方法，性能提升41%（Qwen2.5-7B）和20%（Qwen2.5-3B）。本文还提供了关于强化学习优化方法、LLM选择及响应长度动态的实证洞察。代码与模型检查点见https://github.com/PeterGriffinJin/Search-R1。
原文摘要 · Abstract (English)
Efficiently acquiring external knowledge and up-to-date information is essential for effective reasoning and text generation in large language models (LLMs). Prompting advanced LLMs with reasoning capabilities to use search engines during inference is often suboptimal, as the LLM might not fully possess the capability on how to interact optimally with the search engine. This paper introduces Search-R1, an extension of reinforcement learning (RL) for reasoning frameworks where the LLM learns to autonomously generate (multiple) search queries during step-by-step reasoning with real-time retrieval. Search-R1 optimizes LLM reasoning trajectories with multi-turn search interactions, leveraging retrieved token masking for stable RL training and a simple outcome-based reward function. Experiments on seven question-answering datasets show that Search-R1 improves performance by 41% (Qwen2.5-7B) and 20% (Qwen2.5-3B) over various RAG baselines under the same setting. This paper further provides empirical insights into RL optimization methods, LLM choices, and response length dynamics in retrieval-augmented reasoning. The code and model checkpoints are available at https://github.com/PeterGriffinJin/Search-R1.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。