让大模型一次完成多步多源搜索,效率提升50%以上
Reinforcement Fine-Tuning for Reasoning towards Multi-Step Multi-Source Search in Large Language Models
- 单个大模型内整合规划、搜索与回答,避免多代理开销
- 通过强化学习优化输出结构,正确率显著高于现有方法
- 适合需要快速准确检索新事件的实时问答场景
大语言模型在回答训练数据之后的时效性问题时存在事实局限。现有搜索增强方法分为两类:多代理框架因跨多个模型分离规划与生成而计算开销大;单模型工具调用方法仅支持顺序单源搜索。本文提出R-Search,一种统一多步规划、多源搜索执行与答案合成的单模型框架。创新性地将输出结构化为四部分:指导搜索的推理步骤(<think>)、表示多源计划的自然语言有向无环图(<search>)、执行搜索得到的结果(<result>)和最终答案(<answer>)。为此,我们设计基于GRPO的专门强化微调(ReFT)方法及多组件奖励函数,优化答案正确性、图结构有效性与格式一致性。在FinSearchBench-24、SearchExpertBench-25和七个QA基准上的实验表明,R-Search优于当前最优方法,同时实现70%上下文令牌减少和约50%执行延迟降低。代码已公开于https://github.com/wentao0429/Reasoning-search。
原文摘要 · Abstract (English)
Large language models (LLMs) can face factual limitations when responding to time-sensitive queries about recent events that arise after their knowledge thresholds in the training corpus. Existing search-augmented approaches fall into two categories, each with distinct limitations: multi-agent search frameworks incur substantial computational overhead by separating search planning and response synthesis across multiple LLMs, while single-LLM tool-calling methods restrict themselves to sequential planned, single-query searches from sole search sources. We present Reasoning-Search (R-Search), a single-LLM search framework that unifies multi-step planning, multi-source search execution, and answer synthesis within one coherent inference process. Innovatively, it structure the output into four explicitly defined components, including reasoning steps that guide the search process (<think>), a natural-language directed acyclic graph that represents the search plans with respect to diverse sources (<search>), retrieved results from executing the search plans (<result>), and synthesized final answers (<answer>). To enable effective generation of these structured outputs, we propose a specialized Reinforcement Fine-Tuning (ReFT) method based on GRPO, together with a multi-component reward function that optimizes LLM's answer correctness, structural validity of the generated DAG, and adherence to the defined output format. Experimental evaluation on FinSearchBench-24, SearchExpertBench-25, and seven Q and A benchmarks demonstrates that R-Search outperforms state-of-the-art methods, while achieving substantial efficiency gains through 70% reduction in context token usage and approximately 50% decrease in execution latency. Code is available at https://github.com/wentao0429/Reasoning-search.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。