让大模型学会动态调用内外知识，减少幻觉并提升推理效率。
R1-Searcher++: Incentivizing the Dynamic Knowledge Acquisition of LLMs via Reinforcement Learning
- 分两阶段训练：先格式学习，再用强化学习驱动知识获取。
- 在多个基准上超越现有方法，实现高效检索增强推理。
- 适合需要精准推理和持续学习的AI应用开发者。
大型语言模型（LLMs）虽强大，但因知识静态易产生幻觉。检索增强生成（RAG）通过引入外部信息缓解此问题，但现有方法成本高、泛化差或忽视模型内部知识。本文提出R1-Searcher++，一种新框架，训练模型自适应融合内、外部知识。采用两阶段策略：初始SFT冷启动阶段完成格式学习，随后通过强化学习实现动态知识获取。强化学习阶段使用结果监督鼓励探索，设计奖励机制促进内部知识利用，并引入记忆机制持续吸收检索内容，丰富模型内部知识。结合内部知识与外部搜索引擎，模型持续提升能力，实现高效检索增强推理。实验表明，R1-Searcher++优于以往RAG与推理方法，在多个数据集上表现更佳。代码已开源：https://github.com/RUCAIBox/R1-Searcher-plus。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are powerful but prone to hallucinations due to static knowledge. Retrieval-Augmented Generation (RAG) helps by injecting external information, but current methods often are costly, generalize poorly, or ignore the internal knowledge of the model. In this paper, we introduce R1-Searcher++, a novel framework designed to train LLMs to adaptively leverage both internal and external knowledge sources. R1-Searcher++ employs a two-stage training strategy: an initial SFT Cold-start phase for preliminary format learning, followed by RL for Dynamic Knowledge Acquisition. The RL stage uses outcome-supervision to encourage exploration, incorporates a reward mechanism for internal knowledge utilization, and integrates a memorization mechanism to continuously assimilate retrieved information, thereby enriching the model's internal knowledge. By leveraging internal knowledge and external search engine, the model continuously improves its capabilities, enabling efficient retrieval-augmented reasoning. Our experiments demonstrate that R1-Searcher++ outperforms previous RAG and reasoning methods and achieves efficient retrieval. The code is available at https://github.com/RUCAIBox/R1-Searcher-plus.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。