让大模型在思考时不断搜索并精炼知识，提升复杂推理准确率。
Search and Refine During Think: Facilitating Knowledge Refinement for Improved Retrieval-Augmented Reasoning
- 思考过程中插入多次检索与知识精炼步骤，逐步过滤和组织证据。
- 在多跳问答任务中准确率显著提升，尤其在复杂推理场景下优势明显。
- 适合需要深度知识推理的应用，如智能客服、学术研究辅助。
大型语言模型虽具强大推理能力，但受限于自身知识库。检索增强推理通过调用外部资源缓解此问题，但现有方法常检索到无关或噪声信息，影响推理准确性。本文提出 AutoRefine，一种基于强化学习的后训练框架，采用全新的“思考中搜索与精炼”范式。该框架在连续检索之间引入显式的知识精炼步骤，使模型可迭代地过滤、提炼和组织证据后再生成答案。此外，结合特定检索奖励与答案正确性奖励，使用组相对策略优化进行训练。在单跳与多跳问答基准上的实验表明，AutoRefine 显著优于现有方法，尤其在复杂多跳推理任务中表现突出。详细分析显示，AutoRefine 能发起更频繁且更高质的检索，并有效整合证据。
原文摘要 · Abstract (English)
Large language models have demonstrated impressive reasoning capabilities but are inherently limited by their knowledge reservoir. Retrieval-augmented reasoning mitigates this limitation by allowing LLMs to query external resources, but existing methods often retrieve irrelevant or noisy information, hindering accurate reasoning. In this paper, we propose AutoRefine, a reinforcement learning post-training framework that adopts a new "search-and-refine-during-think" paradigm. AutoRefine introduces explicit knowledge refinement steps between successive search calls, enabling the model to iteratively filter, distill, and organize evidence before generating an answer. Furthermore, we incorporate tailored retrieval-specific rewards alongside answer correctness rewards using group relative policy optimization. Experiments on single-hop and multi-hop QA benchmarks demonstrate that AutoRefine significantly outperforms existing approaches, particularly in complex, multi-hop reasoning scenarios. Detailed analysis shows that AutoRefine issues frequent, higher-quality searches and synthesizes evidence effectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。