让大模型通过强化学习学会边搜索边推理，无需人工标注步骤。
ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning
- 用强化学习训练模型在推理中自主决定何时搜索、如何搜索。
- 在多跳问答任务上超越基线模型，32B版本在HotpotQA上达87.3%准确率。
- 能自发产生反思与自我修正，适合需要深度推理的复杂问答场景。
大型语言模型（LLMs）在推理方面表现出色，如OpenAI-o1和DeepSeek-R1的成功所示。然而，将推理与外部搜索过程结合仍具挑战性，尤其针对需多步检索的复杂多跳问题。我们提出ReSearch，一种新颖框架，通过强化学习训练LLMs进行搜索式推理，无需任何关于推理步骤的监督数据。该方法将搜索操作视为推理链的核心组成部分：何时及如何执行搜索由文本形式的思考引导，而搜索结果又影响后续推理。我们在Qwen2.5-7B(-Instruct)和Qwen2.5-32B(-Instruct)模型上训练ReSearch，并进行了广泛实验。尽管仅在一个数据集上训练，模型在多个基准测试中展现出强大泛化能力。分析表明，ReSearch在强化学习过程中自然激发了反思与自我修正等高级推理能力。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have shown remarkable capabilities in reasoning, exemplified by the success of OpenAI-o1 and DeepSeek-R1. However, integrating reasoning with external search processes remains challenging, especially for complex multi-hop questions requiring multiple retrieval steps. We propose ReSearch, a novel framework that trains LLMs to Reason with Search via reinforcement learning without using any supervised data on reasoning steps. Our approach treats search operations as integral components of the reasoning chain, where when and how to perform searches is guided by text-based thinking, and search results subsequently influence further reasoning. We train ReSearch on Qwen2.5-7B(-Instruct) and Qwen2.5-32B(-Instruct) models and conduct extensive experiments. Despite being trained on only one dataset, our models demonstrate strong generalizability across various benchmarks. Analysis reveals that ReSearch naturally elicits advanced reasoning capabilities such as reflection and self-correction during the reinforcement learning process.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。