用强化学习让大模型分步搜索、提炼、推理，更准更快答复杂问题。
OThink-SRR1: Search, Refine and Reasoning with Reinforced Learning for Large Language Models
- 三步迭代：搜、精炼、推理，用强化学习优化每一步
- 在4个复杂问答数据集上准确率超基线，检索步骤和字数更少
- 适合需要精准信息获取的智能助手、科研工具等场景
检索增强生成（RAG）扩展了大语言模型的知识边界，但现有静态检索方法难以应对复杂的多跳问题。尽管近期动态检索策略有所改进，仍面临两大挑战：一是无关检索内容会误导推理过程，二是处理全文带来高昂的计算与延迟成本。为此，我们提出OThink-SRR1框架，通过强化学习训练大模型执行迭代式的搜索-精炼-推理流程。其核心的精炼阶段将检索到的文档压缩为简洁且相关的事实，再进行推理。我们引入GRPO-IR，一种端到端的强化学习算法，奖励精确证据识别，惩罚过度检索，使模型兼具专注性与高效性。在四个多跳问答基准测试中，该方法在准确率上优于强基线，同时减少检索步骤和使用token数量。这使OThink-SRR1成为信息获取型智能体的有力基础模型。
原文摘要 · Abstract (English)
Retrieval-Augmented Generation (RAG) expands the knowledge of Large Language Models (LLMs), yet current static retrieval methods struggle with complex, multi-hop problems. While recent dynamic retrieval strategies offer improvements, they face two key challenges: 1) irrelevant retrieved noise can misdirect the reasoning process, and 2) processing full documents incurs prohibitive computational and latency costs. To address these issues, we propose OThink-SRR1, a framework that enhances large models with an iterative Search-Refine-Reason process trained via reinforcement learning. Its core Refine stage distills retrieved documents into concise, relevant facts before reasoning. We introduce GRPO-IR, an end-to-end reinforcement learning algorithm that rewards accurate evidence identification while penalizing excessive retrievals, thus training the model to be both focused and efficient. Experiments on four multi-hop QA benchmarks show our approach achieves superior accuracy over strong baselines while using fewer retrieval steps and tokens. This positions OThink-SRR1 as a potent foundational model for information-seeking agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。